A VARIANCE PRINCIPLE EXPLAINS WHY DROPOUT FINDS FLATTER MINIMA
Year: 2021

Abstract:
Although dropout has achieved great success in deep learning, little is known about how it helps the training find a good generalization solution in the highdimensional parameter space. In this work, we show that the training with dropout finds the neural network with a flatter minimum compared with standard gradient descent training. We further study the underlying mechanism of why dropout finds flatter minima through experiments. We propose a Variance Principle that the variance of a noise is larger at the sharper direction of the loss landscape. Existing works show that SGD satisfies the variance principle, which leads the training to flatter minima. Our work show that the noise induced by the dropout also satisfies the variance principle that explains why dropout finds flatter minima by experiments on various datasets, i.e., CIFAR100, CIFAR10, MNIST and synthetic data, and various structures, i.e., fully-connected networks and large residual convolutional networks. In general, our work points out that the variance principle is an important similarity between dropout and SGD that lead the training to find flatter minima and obtain good generalization.

heading: 1 INTRODUCTION
text: Dropout is used with gradient-descent-based algorithms for training DNNs (Hinton et al., 2012; Srivastava et al., 2014). During training, the output of each neuron is multiplied with a random variable with probability p as one and 1−p as zero. Note that p is called dropout rate, and every time for computing concerned quantity, the variable is randomly sampled at each feedforward operation. Dropout has been an indispensable trick in the training of deep neural networks (DNNs), however, with very little understanding.
Similar to SGD, training with dropout is equivalent to that with some specific noise. To understand what kind of noise benefits the generalization of training, we proposes a variance principle of a noise, that is,
Variance Principle: the variance of a noise is larger at the sharper direction of the loss landscape.
If a noise satisfies the variance principle, it can help the training select flatter minima and leads the training to better generalization. As shown in Zhu et al. (2018); Feng & Tu (2021), the noise in SGD satisfies the variance principle and SGD can find flatter minima and obtain better generalization (Keskar et al., 2016; Neyshabur et al., 2017).
In this work, we study the characteristic of minima learned with dropout. We show that compared with the standard gradient descent (GD), the GD with dropout selects flatter minima. As suggested by many existing works (Keskar et al., 2016; Neyshabur et al., 2017; Zhu et al., 2018), flatter minima are more likely to have better generalization and stability. We then put efforts to show that the noise induced by the dropout satisfies the variance principle, which explains why dropout finds flatter minima.
To examine the variance principle, we explore the relation between the flatness of the loss landscape and the noise structure induced by dropout at minima through three methods and obtain a consistent result that the noise is larger at the sharper direction of the loss landscape to help the training select flatter minima. Our experiments are conducted over synthetic data and fully-connected neural networks as well as modern datasets and models such as, MNIST, CIFAR10 and CIFAR-100 and ResNet-20, thus our conclusion is a rather general result.
First, we examine the inverse variance-flatness relation, similar to Feng & Tu (2021). We define the flatness of a minimum at one direction by the length of the largest interval in the considered direction, which covers the minimum and no point in the interval has loss larger than twice of the loss of the minimum, denoted by Fp for the direction p. We then consider two definitions of the noise structure, i.e., random trajectory covariance Σt and random gradient covariance Σg . For the random trajectory covariance, we train the network to an “exploration phase” (Shwartz-Ziv & Tishby, 2017), where the loss decreases with a very slow speed and then sample parameter sets {θi}Ni=1 from N consecutive training steps to compute the covariance, where θi is the network parameter set at step i. For the gradient covariance, we train the network until the loss is very small and then freeze the training. We sample N gradients {gi}Ni=1 with different dropout variables to compute the covariance. In each sample, the dropout rate is fixed. For both random trajectory covariance and random gradient covariance, we perform principal component analysis (PCA) and obtain similar results. We find that at the direction of larger variance (larger eigen-value), the loss landscape of the minimum is sharper, i.e., inverse variance-flatness relation.
Second, we study the relation between the Hessian and the noise structure induced by dropout. The eigenvalues of the Hessian of the loss at a minimum are also often used to indicate the flatness. The landscape at an eigen-direction is claimed sharper if the corresponding eigen-value is larger. For each eigen-direction vj , we project the parameter trajectory {θi}Ni=1 or gradients {gi}Ni=1 to the direction of vj and compute the variance. We find that the noise variance is larger at the direction with a larger eigen-value.
Third, we show that the Hessian matrix aligns well with the random gradient covariance of gradients {gi}Ni=1, i.e., their eigen directions of large eigen-values are close, similar to Zhu et al. (2018). These empirical works show that the noise structure induced by the dropout tends to have larger variance in order to escape the sharper direction, i.e., variance principle, thus, leading to flatter minima. These characteristics of dropout are very similar to SGD (Keskar et al., 2016; Zhu et al., 2018; Feng & Tu, 2021). The similarity between dropout and SGD suggests that modelling their similarity may be a key to understanding how and what stochasticity benefits the training.

heading: 2 RELATED WORKS
text: Dropout was proposed as a simple way to prevent neural networks from overfitting, and thus improving the generalization of the network, to a certain extent (Hinton et al., 2012; Srivastava et al., 2014). Many works aim to find an explicit regularization form of dropout. Wager et al. (2013) studies the explicit form of dropout on linear regression and logistic problem, but for studying non-linear neural network, it is still unclear how to characterize the effect of dropout by an explicit regularization term. McAllester (2013) presents PAC-Bayesian bounds, and Wan et al. (2013), Mou et al. (2018) derives Rademacher generalization bounds. These results show that the reduction of complexity brought by dropout is O(p), where p is the probability of keeping an element in dropout. Mianjy & Arora (2020) show that dropout training with logistic loss achieves -suboptimality in test error in O(1/ ) iterations. All of the above works need specific settings, such as norm assumptions and logistic loss, and they only give a rough estimate of the generalization error bound, which usually consider the worst case. However, it is not clear what is the characteristic of the dropout training process and how to bridge the training with the generalization. In this work, we show that dropout noise has a special structure, which closely relates with the loss landscape. The structure of the effective noise induced by the dropout may be a key reason why dropout can find solutions with better generalization.
Many researches have empirically shown that SGD can improve the generalization performance of neural networks through finding a flatter solution (Li et al., 2017; Jastrzebski et al., 2017; 2018). This work utilizes the current understanding of SGD to study dropout and shows that much similarity is shared between SGD and dropout.
The flatness of the solution is an important aspect of understanding the generalization of neural networks (Keskar et al., 2016; Neyshabur et al., 2017; Zhu et al., 2018). A number of works suggested that the learning rate and batch size determine the flatness of the solutions (Jastrzebski et al., 2017; 2018; Wu et al., 2018). Li et al. (2017) propose a visualization method of the loss landscape at 1-d cross-section to visualize the flatness.
Feng & Tu (2021) investigate the connection between SGD learning dynamics and the loss landscape through the principal component analysis (PCA), and show that SGD dynamics follow a lowdimensional drift-diffusion motion in the weight space. Through characterizing the loss landscape by its flatness in each PCA direction around the solution found by SGD, they also reveal a robust inverse relation between the weight variance and the landscape flatness in PCA directions, thus finding that SGD serves as a landscape dependent annealing algorithm to search for flat minima.
Zhu et al. (2018) study a general form of gradient based optimization dynamics with unbiased noise to analyze the behavior of SGD on escaping from minima and its regularization effects. They also introduce an indicator to characterize the efficiency of escaping from minima through measuring the alignment of noise covariance and the curvature of loss function and thus revealing the anisotropic noise of SGD.

heading: 3 PRELIMINARY

heading: 3.1 DROPOUT
text: Consider an L-layer neural network fθ(x). With dropout (Srivastava et al., 2014), the feedforward operation in a network fθ(x) is
f [0] θ (x) = x, (1) rj ∼ Bernoulli(p), (2)
f [l] θ (x) = r [l] ◦ σ ◦ (W [l−1]f [l−1]θ (x) + b [l−1]) 1 ≤ l ≤ L− 1, (3)
fθ(x) = f [L] θ (x) = W [L−1]f [L−1] θ (x) + b [L−1], (4)
where p is the dropout rate, W [l] ∈ Rml+1×ml , b[l] = Rml+1 , m0 = din = d, mL = do, σ is a scalar function and “◦” means entry-wise operation. We denote the set of parameters by
θ = (W [0],W [1], . . . ,W [L−1], b[0], b[1], . . . , b[L−1]),

heading: 3.2 INTERVAL FLATNESS
text: We use the definition of flatness in Feng & Tu (2021). For convenience, we call it interval flatness Around a specific solution θ∗0 , we compute the loss function profile Lv along the direction v:
Lv(δθ) ≡ L(θ∗0 + δθv).
The interval flatness Fv is defined as the width of the region within which Lv(δθ) ≤ 2Lv(0). We determine Fv by finding two closest points θlv < 0 and θ r v > 0 on each side of the minimum that satisfy Lv(θlv) = Lv(θ r v) = 2Lv(0). The scale factor 2 is used in Feng & Tu (2021), and after our test, the result is not sensitive to the selection of this factor. In this work, we follow their experimental scheme to show the similarity between dropout and SGD. The interval flatness is defined as:
Fv ≡ θrv − θlv. (5) A larger value of Fv means a flatter landscape in the direction v.

heading: 3.3 RANDOMNESS INDUCED BY DROPOUT

heading: 3.3.1 RANDOM TRAJECTORY DATA
text: The training process of neural networks are usually divided into two phases, fast convergence and exploration phase (Shwartz-Ziv & Tishby, 2017). Feng & Tu (2021)’s work focuses on the behavior of networks in the exploration phase. In this work, we follow the experimental scheme in Feng & Tu (2021) to show the similarity between dropout and SGD. This can be understood by frequency principle (Xu et al., 2019; 2020), which states that DNNs fast learn low-frequency components but slowly learn high-frequency ones.
We collect parameter sets Spara = {θi}Ni=1 from N consecutive training steps in the exploration phase, where θi is the network parameter set at step i.

heading: 3.3.2 RANDOM GRADIENT DATA
text: However, due to the limitation of the number of sampling points, we often need larger time interval for sampling. Although the network loss is small, compared with the initial sampling parameters, the network parameters could have large changes during the long-time sampling. Therefore, much extra noise may be induced. Meanwhile, for dropout, it is difficult to get a small loss value on large networks and datasets, therefore, model parameters often have large fluctuations. Based on this problem, we propose a more appropriate sampling method to avoid additional noise caused by sampling parameters in a large time interval. We train the network until the loss is very small and then freeze the training. We sample N gradients of the loss function w.r.t. the parameters with different dropout variables, i.e., Sgrad = {gi}Ni=1. In each sample, the dropout rate is fixed. In this way, we can get the noise structure of dropout without being affected by parameter changes caused by long-term training.

heading: 3.4 INVERSE VARIANCE-FLATNESS RELATION
text: We study the inverse variance flatness relation for both random trajectory data and random gradient data. For convenience, we denote data as S and its covariance as Σ.

heading: 3.4.1 INTERVAL FLATNESS VS. VARIANCE
text: Following Feng & Tu (2021), we perform principal component analysis (PCA) for Σ. Denote λi(Σ) as the i-th eigenvalue of Σ, and denote its corresponding eigen-vector as vi(Σ). We then compute the flatness of the loss landscape in the direction vi(Σ), and denote it as Fvi(Σ). Finally, we display the scatter plot of {λi(Σ), Fvi(Σ)}.

heading: 3.4.2 PROJECTED VARIANCE VS. HESSIAN FLATNESS
text: The method in Feng & Tu (2021), requires many training steps to obtain the covariance. To obtain the variance induced by the dropout at a fixed position θ, we propose another way to characterize the inverse variance-flatness relation. We use the the eigen-value λi(H) of the Hessian H of the loss landscape at θ to denote the sharpness. Hessian matrix is the matrix obtained by the second derivative of the loss function of the neural network with respect to the parameter vector of neural network. Here, the parameter vector is a vector consisting of all the parameters of network. The variance induced by dropout at the corresponding eigen-direction vi(H) is computed by the following procedure. For each eigen-direction vi of Hessian H , we project the sampled parameters or the gradients of sampled parameter S to direction vi by inner product, denoted by Projvi(S). Then, the variance at direction vi(H) is the variance of dataset Projvi(S), denoted by Var(Projvi(S)). Finally, we display the scatter plot of {Var(Projvi(S)), λi(H)}.

heading: 3.5 ALIGNMENT BETWEEN HESSIAN AND GRADIENT COVARIANCE
text: We use the method in Zhu et al. (2018) to quantify the alignment between the noise structure and the curvature of loss surface. For each training step i, we calculate the alignment parameter Ti:
Ti = Tr(HiΣi),
where Σi is the ith-step covariance matrix generated by dropout layers andHi is the ith-step Hessian matrix of network parameters.

heading: 4 EXPERIMENTAL SETUP
text: To understand the effects of dropout, we trained a number of networks with different structures. We consider the following types of neural networks: 1) Fully-connected neural networks (FNNs) trained by MNIST. For FNN, all parameters are initialized by Xavier initialization(Glorot & Bengio, 2010). 2) Convolutional neural networks (CNNs) trained by CIFAR-10. For CNN, all parameters are initialized by He initialization (He et al., 2015). The loss of all our experiments is cross entropy loss. 3) Deep residual neural networks (ResNets) (He et al., 2016) trained by CIFAR-100. For ResNet, all parameters are initialized by He initialization (He et al., 2015). The loss of all our experiments is cross entropy loss.
For Fig. 1 (a), we use the FNN with size 784 − 1024 − 1024 − 10. We add dropout layers behind the first and the second layers with dropout rate of 0.8 and 0.5, respectively. We train the network using default Adam optimizer (Kingma & Ba, 2015) with a learning rate of 0.0001.
For Fig. 1 (b), we use vgg-9 (Simonyan & Zisserman, 2014) to compare the loss landscape flatness w/o dropout layers. Models are trained using GD with Nesterov momentum, training-size 2048 for 300 epochs. The learning rate was initialized at 0.1, and divided by a factor of 10 at epochs 150, 225 and 275. We only use 2048 examples for training to compromise with the computational burden.
For Fig. 1 (c), 3, 5, we use ResNet-20 (He et al., 2016) to compare the loss landscape flatness w/o dropout layers. Models are trained using GD, training-size 50000 for 1200 epochs. The learning rate was initialized at 0.01. Since the Hessian calculation of ResNet takes much time, for the ResNet experiment, we only perform it at a specific dropout rate and learning rate.
For Fig. 2, 4, 6, we use the FNN with size 784− 50− 50− 10. We train the network using GD with the first 10,000 training data as the training set. We add a dropout layer behind the second layer. The dropout rate and learning rate are specified and unchanged in each experiment.
It is worth noting that, in order to avoid the influence of the noise induced by SGD in our experiments, all our networks are trained using GD, so it is difficult for us to verify on larger datasets such as ImageNet.

heading: 5 DROPOUT FINDS FLATTER MINIMA
text: Dropout is almost ubiquitous in training deep networks. It is interesting and important to understand what makes dropout improve the generalization of training neural networks. Inspired by the study of SGD (Keskar et al., 2016), we explore the flatness of the minima found by dropout.
The loss landscape of DNNs is highly non-convex and complicate (Skorokhodov & Burtsev, 2019), but with certain characteristics, such as embedding principle (Zhang et al., 2021) shows that the loss landscape of any network “contains” all critical points of all narrower networks, in the sense that, any critical point of any narrower network can be embedded to a critical point of the target network preserving its output function. It is impractical to visualize the loss landscape in the highdimensional space. To compare two minima, θ and θ′, in a loss landscape, one simple way (Keskar et al., 2016) is to visualize 1-d cross-section of the interpolation between θ and θ′. However, empirical studies (Li et al., 2017) show that this simple way could be misleading when θ and θ′ have difference of orders of magnitude. Li et al. (2017) visualizes loss functions using filter-wise normalized directions to remove the scaling effect. We adopt this method (Li et al., 2017) in this work as follows. To obtain a direction for a network with parameters θ, we begin by producing a random Gaussian direction vector d with dimensions compatible with θ. Then, we normalize each filter in d to have the same norm of the corresponding filter in θ. In other words, we make the replacement di,j ← di,j‖di,j‖ ‖θi,j‖, where di,j represents the jth filter (not the jth weight, for FNN, one layer is one filter.) of the ith layer of d, and ‖ · ‖ denotes the Frobenius norm. We use f(α) = L (θ + αd) to characterize the loss landscape around the minima obtained with dropout layers θ∗d and without dropout layer θ∗. For all network structures shown in Fig. 1, dropout can improve the generalization of the network and find a flatter minimum. In Fig. 1(a), (b), for both networks trained with and
without dropout layers, the training loss values are all closed to zero, but their flatness and generalization are still quite different. In Fig. 1(c), due to the complexity of the dataset and network, and the large number of dropout layers, the loss value of network with dropout layers is larger than the one without dropout layer. However, the network with dropout layers still finds a flatter minimum with better generalization.

heading: 6 INVERSE VARIANCE-FLATNESS RELATION
text: Similar to SGD, the effect of dropout can be equivalent to imposing a specific noise on the gradient. A random noise, such as isotropic noise, can help the training escape local minima, but can not robustly improve generalization (An, 1996; Zhu et al., 2018). The noise induced by the dropout should have certain properties that can lead the training to good minima.
In this section, we show that the noise induced by the dropout satisfies the variance principle, that is, the noise variance is larger along the sharper direction of the loss landscape at a minimum. The landscape-dependent structure helps the training escape sharp minima. We utilize three methods to examine the variance principle for dropout, as summarized in Table 1.

heading: 6.1 INTERVAL FLATNESS VS. VARIANCE
text: We use the principal component analysis (PCA) to study the weight variations when the accuracy is nearly 100%. For FNNs, networks are trained on MNIST with the first 10000 examples as the training set for computational efficiency. For ResNets, networks are trained on CIFAR-100 with 50000 examples as the training set to make a more convincing conclusion on modern datasets and models. The networks are trained with full batch for different learning rates and dropout rates under the same random seed (that is, with the same initialization parameters). After the loss is small enough, we sample the parameters or gradients of parameters N times (N = 3000 in this experiment) and use the method introduced in Sec. 3.3 to construct covariance matrix Σ by the weights Spara or gradients Sgrad between two hidden layers. The PCA was done for the covariance matrix Σ. We then compute the interval flatness of the loss function landscape at eigen-directions, i.e., {Fvi(Σ)}Ni=1. Note that the PCA spectrum {λi(Σ)}Ni=1 indicate the variance of weights Spara or gradients Sgrad at corresponding eigen-directions.
As shown in Fig. 2, 3, for different learning rates and dropout rates, there is an inverse relationship between the interval flatness of the loss function landscape {Fvi(Σ)}Ni=1 and the dropout variance,
i.e., the PCA spectrum {λi(Σ)}Ni=1. We can approximately see a power-law relationship between {Fvi(Σ)}Ni=1 and {λi(Σ)}Ni=1 for different dropout rates and learning rates. More detailed, for small flatness part, the variance of noise induced by dropout is generally large, which indicates that the noise induced by dropout has larger variance in sharp directions, for large flatness part, as the loss landscape flatter, the linear relationship more obvious, we can see a clearer asymptotic behavior in the results. Overall, we can observe the negative correlation between the eigenvalues and flatness in Fig. 2, 3.

heading: 6.2 PROJECTED VARIANCE VS. HESSIAN FLATNESS
text: The eigenvalues of the Hessian of the loss at a minimum are also often used to indicate the flatness. A large eigenvalue correspond to a sharper direction. In this section, we study the relationship between eigen-values of the HessianH of parameters θ at the end point of training and the variances of dropout at corresponding eigen-directions. As mentioned in the Preliminary section, we sample the parameters or gradients of parameters 1000 times, that isN = 1000. For each eigen-direction vi of Hessian H , we project the sampled parameters or the gradients of sampled parameter to direction vi by inner product, denoted by Projvi(S). Then, we compute the variance of the projected data, i.e., Var(Projvi(S)).
As shown in Fig. 4, 5, we find that there is also a power-law relationship between {λi(H)}Di=1 and {Var(Projvi(S))} D i=1 for different dropout rates and learning rates, no matter if S is sampled
from parameters or gradients of parameters. The positive correlation between the eigenvalue and the projection variance show the structure of the dropout noise, which helps the network escape the bad minima. At the same time, as shown in Fig. 2, 3, 4, 5, we can see that gradient sampling has a more obvious linear structure than parameter sampling, which shows that gradient sampling can better avoid the interference of irrelevant noise.

heading: 6.3 ALIGNMENT BETWEEN HESSIAN AND GRADIENT COVARIANCE
text: Zhu et al. (2018) show that the alignment indicator Tr(HΣ) plays an crucial role for stochastic processes escaping from minima, where H is the Hessian and Σ is the noise covariance. In this subsection, we study the alignment between the Hessian and the random gradient covariance at each training step. Note that the training is performed by GD without dropout. At step i, we sample the gradients of parameters {gji }Nj=1 by tentatively adding a dropout layer between the hidden layers. For each step i, we the compute Tr(HiΣi), where Hi is the Hessian of the loss at the parameter set at step i and Σi is the covariance of {gji }Nj=1. In order to show the anisotropic structure, we construct the isotropic noise for comparison, i.e., Σ̄i = Tr Σi D I of the covariance matrix Σi, where D is the number of parameters. In our experiments, D = 2500. As shown in Fig. 6 , in the whole training process under different learning rates
and dropout rates, Tr(HiΣi) is much larger than Tr(HiΣ̄i), indicating the anisotropic structure of dropout noise and its high alignment with the Hessian matrix.

heading: 7 CONCLUSION AND DISCUSSION
text: In this work, we propose a variance principle that noise is larger at the sharper direction of the loss landscape. If a noise satisfies the variance principle, it helps the training select flatter minima and leads the training to good generalization. We empirical show that dropout finds flat minima during the training and examine that the dropout satisfies the variance principle to explain why dropout finds flat minima over various datasets and neural network structures by the following three perspectives: interval flatness vs. variance, projected variance vs. Hessian flatness and alignment between Hessian and gradient covariance.
The dropout and the SGD are common in sharing the variance principle. Based on this, there could be a general theory framework that can model both dropout and SGD to understand why they have better generalization. The research on understanding the good generalization of dropout and SGD is far from complete. As a starting point, the variance principle shows a promising and reasonable direction for understanding the stochastic training of neural networks.

Review summaries:
This paper provides an empirical study on how the dropout can lead to minima within a flatter landscape (hence a better generalization performance). Based on the definition of minimizer flatness (i.e., eq. 5) and random trajectory data (parameters from a optimization path and gradients when loss is stably small), the authors first show that dropout can help to find flatter minima, and also find a inverse relation between the flatness (the length $F_v$) and the algorithmic variance (i.e., in area with sharper direction, algorithm with dropout has a larger variance), and further find that for areas with larger eigenvalues (i.e., with sharper landscape), the variances of the algorithm along the directions of their eigenvectors are also large. Such empirical results together demonstrate that the dropout can play a good role in improving the generalization of existing algorithms by introducing certain level of noise.