A Variance Principle Explains why Dropout Finds Flatter Minima

1 INTRODUCTION . Dropout is used with gradient-descent-based algorithms for training DNNs ( Hinton et al. , 2012 ; Srivastava et al. , 2014 ) . During training , the output of each neuron is multiplied with a random variable with probability p as one and 1−p as zero . Note that p is called dropout rate , and every time for computing concerned quantity , the variable is randomly sampled at each feedforward operation . Dropout has been an indispensable trick in the training of deep neural networks ( DNNs ) , however , with very little understanding . Similar to SGD , training with dropout is equivalent to that with some specific noise . To understand what kind of noise benefits the generalization of training , we proposes a variance principle of a noise , that is , Variance Principle : the variance of a noise is larger at the sharper direction of the loss landscape . If a noise satisfies the variance principle , it can help the training select flatter minima and leads the training to better generalization . As shown in Zhu et al . ( 2018 ) ; Feng & Tu ( 2021 ) , the noise in SGD satisfies the variance principle and SGD can find flatter minima and obtain better generalization ( Keskar et al. , 2016 ; Neyshabur et al. , 2017 ) . In this work , we study the characteristic of minima learned with dropout . We show that compared with the standard gradient descent ( GD ) , the GD with dropout selects flatter minima . As suggested by many existing works ( Keskar et al. , 2016 ; Neyshabur et al. , 2017 ; Zhu et al. , 2018 ) , flatter minima are more likely to have better generalization and stability . We then put efforts to show that the noise induced by the dropout satisfies the variance principle , which explains why dropout finds flatter minima . To examine the variance principle , we explore the relation between the flatness of the loss landscape and the noise structure induced by dropout at minima through three methods and obtain a consistent result that the noise is larger at the sharper direction of the loss landscape to help the training select flatter minima . Our experiments are conducted over synthetic data and fully-connected neural networks as well as modern datasets and models such as , MNIST , CIFAR10 and CIFAR-100 and ResNet-20 , thus our conclusion is a rather general result . First , we examine the inverse variance-flatness relation , similar to Feng & Tu ( 2021 ) . We define the flatness of a minimum at one direction by the length of the largest interval in the considered direction , which covers the minimum and no point in the interval has loss larger than twice of the loss of the minimum , denoted by Fp for the direction p. We then consider two definitions of the noise structure , i.e. , random trajectory covariance Σt and random gradient covariance Σg . For the random trajectory covariance , we train the network to an “ exploration phase ” ( Shwartz-Ziv & Tishby , 2017 ) , where the loss decreases with a very slow speed and then sample parameter sets { θi } Ni=1 from N consecutive training steps to compute the covariance , where θi is the network parameter set at step i . For the gradient covariance , we train the network until the loss is very small and then freeze the training . We sample N gradients { gi } Ni=1 with different dropout variables to compute the covariance . In each sample , the dropout rate is fixed . For both random trajectory covariance and random gradient covariance , we perform principal component analysis ( PCA ) and obtain similar results . We find that at the direction of larger variance ( larger eigen-value ) , the loss landscape of the minimum is sharper , i.e. , inverse variance-flatness relation . Second , we study the relation between the Hessian and the noise structure induced by dropout . The eigenvalues of the Hessian of the loss at a minimum are also often used to indicate the flatness . The landscape at an eigen-direction is claimed sharper if the corresponding eigen-value is larger . For each eigen-direction vj , we project the parameter trajectory { θi } Ni=1 or gradients { gi } Ni=1 to the direction of vj and compute the variance . We find that the noise variance is larger at the direction with a larger eigen-value . Third , we show that the Hessian matrix aligns well with the random gradient covariance of gradients { gi } Ni=1 , i.e. , their eigen directions of large eigen-values are close , similar to Zhu et al . ( 2018 ) . These empirical works show that the noise structure induced by the dropout tends to have larger variance in order to escape the sharper direction , i.e. , variance principle , thus , leading to flatter minima . These characteristics of dropout are very similar to SGD ( Keskar et al. , 2016 ; Zhu et al. , 2018 ; Feng & Tu , 2021 ) . The similarity between dropout and SGD suggests that modelling their similarity may be a key to understanding how and what stochasticity benefits the training . 2 RELATED WORKS . Dropout was proposed as a simple way to prevent neural networks from overfitting , and thus improving the generalization of the network , to a certain extent ( Hinton et al. , 2012 ; Srivastava et al. , 2014 ) . Many works aim to find an explicit regularization form of dropout . Wager et al . ( 2013 ) studies the explicit form of dropout on linear regression and logistic problem , but for studying non-linear neural network , it is still unclear how to characterize the effect of dropout by an explicit regularization term . McAllester ( 2013 ) presents PAC-Bayesian bounds , and Wan et al . ( 2013 ) , Mou et al . ( 2018 ) derives Rademacher generalization bounds . These results show that the reduction of complexity brought by dropout is O ( p ) , where p is the probability of keeping an element in dropout . Mianjy & Arora ( 2020 ) show that dropout training with logistic loss achieves -suboptimality in test error in O ( 1/ ) iterations . All of the above works need specific settings , such as norm assumptions and logistic loss , and they only give a rough estimate of the generalization error bound , which usually consider the worst case . However , it is not clear what is the characteristic of the dropout training process and how to bridge the training with the generalization . In this work , we show that dropout noise has a special structure , which closely relates with the loss landscape . The structure of the effective noise induced by the dropout may be a key reason why dropout can find solutions with better generalization . Many researches have empirically shown that SGD can improve the generalization performance of neural networks through finding a flatter solution ( Li et al. , 2017 ; Jastrzebski et al. , 2017 ; 2018 ) . This work utilizes the current understanding of SGD to study dropout and shows that much similarity is shared between SGD and dropout . The flatness of the solution is an important aspect of understanding the generalization of neural networks ( Keskar et al. , 2016 ; Neyshabur et al. , 2017 ; Zhu et al. , 2018 ) . A number of works suggested that the learning rate and batch size determine the flatness of the solutions ( Jastrzebski et al. , 2017 ; 2018 ; Wu et al. , 2018 ) . Li et al . ( 2017 ) propose a visualization method of the loss landscape at 1-d cross-section to visualize the flatness . Feng & Tu ( 2021 ) investigate the connection between SGD learning dynamics and the loss landscape through the principal component analysis ( PCA ) , and show that SGD dynamics follow a lowdimensional drift-diffusion motion in the weight space . Through characterizing the loss landscape by its flatness in each PCA direction around the solution found by SGD , they also reveal a robust inverse relation between the weight variance and the landscape flatness in PCA directions , thus finding that SGD serves as a landscape dependent annealing algorithm to search for flat minima . Zhu et al . ( 2018 ) study a general form of gradient based optimization dynamics with unbiased noise to analyze the behavior of SGD on escaping from minima and its regularization effects . They also introduce an indicator to characterize the efficiency of escaping from minima through measuring the alignment of noise covariance and the curvature of loss function and thus revealing the anisotropic noise of SGD . 3 PRELIMINARY . 3.1 DROPOUT . Consider an L-layer neural network fθ ( x ) . With dropout ( Srivastava et al. , 2014 ) , the feedforward operation in a network fθ ( x ) is f [ 0 ] θ ( x ) = x , ( 1 ) rj ∼ Bernoulli ( p ) , ( 2 ) f [ l ] θ ( x ) = r [ l ] ◦ σ ◦ ( W [ l−1 ] f [ l−1 ] θ ( x ) + b [ l−1 ] ) 1 ≤ l ≤ L− 1 , ( 3 ) fθ ( x ) = f [ L ] θ ( x ) = W [ L−1 ] f [ L−1 ] θ ( x ) + b [ L−1 ] , ( 4 ) where p is the dropout rate , W [ l ] ∈ Rml+1×ml , b [ l ] = Rml+1 , m0 = din = d , mL = do , σ is a scalar function and “ ◦ ” means entry-wise operation . We denote the set of parameters by θ = ( W [ 0 ] , W [ 1 ] , . . . , W [ L−1 ] , b [ 0 ] , b [ 1 ] , . . . , b [ L−1 ] ) , 3.2 INTERVAL FLATNESS . We use the definition of flatness in Feng & Tu ( 2021 ) . For convenience , we call it interval flatness Around a specific solution θ∗0 , we compute the loss function profile Lv along the direction v : Lv ( δθ ) ≡ L ( θ∗0 + δθv ) . The interval flatness Fv is defined as the width of the region within which Lv ( δθ ) ≤ 2Lv ( 0 ) . We determine Fv by finding two closest points θlv < 0 and θ r v > 0 on each side of the minimum that satisfy Lv ( θlv ) = Lv ( θ r v ) = 2Lv ( 0 ) . The scale factor 2 is used in Feng & Tu ( 2021 ) , and after our test , the result is not sensitive to the selection of this factor . In this work , we follow their experimental scheme to show the similarity between dropout and SGD . The interval flatness is defined as : Fv ≡ θrv − θlv . ( 5 ) A larger value of Fv means a flatter landscape in the direction v . 3.3 RANDOMNESS INDUCED BY DROPOUT . 3.3.1 RANDOM TRAJECTORY DATA . The training process of neural networks are usually divided into two phases , fast convergence and exploration phase ( Shwartz-Ziv & Tishby , 2017 ) . Feng & Tu ( 2021 ) ’ s work focuses on the behavior of networks in the exploration phase . In this work , we follow the experimental scheme in Feng & Tu ( 2021 ) to show the similarity between dropout and SGD . This can be understood by frequency principle ( Xu et al. , 2019 ; 2020 ) , which states that DNNs fast learn low-frequency components but slowly learn high-frequency ones . We collect parameter sets Spara = { θi } Ni=1 from N consecutive training steps in the exploration phase , where θi is the network parameter set at step i .

Summary: This paper provides an empirical study on how the dropout can lead to minima within a flatter landscape (hence a better generalization performance). Based on the definition of minimizer flatness (i.e., eq. 5) and random trajectory data (parameters from a optimization path and gradients when loss is stably small), the authors first show that dropout can help to find flatter minima, and also find a inverse relation between the flatness (the length $F_v$) and the algorithmic variance (i.e., in area with sharper direction, algorithm with dropout has a larger variance), and further find that for areas with larger eigenvalues (i.e., with sharper landscape), the variances of the algorithm along the directions of their eigenvectors are also large. Such empirical results together demonstrate that the dropout can play a good role in improving the generalization of existing algorithms by introducing certain level of noise.