A Variance Principle Explains why Dropout Finds Flatter Minima
Authors: Zhongwang Zhang
Hanxu Zhou
Zhiqin Xu
Venue: ICLR 2022 Submitted (2022)
Keywords: dropout
stochastic gradient descent
loss landscape
flatness
neural network

Abstract: Although dropout has achieved great success in deep learning, little is known about how it helps the training find a good generalization solution in the high-dimensional parameter space. In this work, we show that the training with dropout finds the neural network with a flatter minimum compared with standard gradient descent training. We further study the underlying mechanism of why dropout finds flatter minima through experiments. We propose a Variance Principle that the variance of a noise is larger at the sharper direction of the loss landscape. Existing works show that SGD satisfies the variance principle, which leads the training to flatter minima. Our work show that the noise induced by the dropout also satisfies the variance principle that explains why dropout finds flatter minima. In general, our work points out that the variance principle is an important similarity between dropout and SGD that lead the training to find flatter minima and obtain good generalization. 

Under review as a conference paper at ICLR 2022
A VARIANCE PRINCIPLE EXPLAINS WHY DROPOUT FINDS FLATTER MINIMA
Anonymous authors Paper under double-blind review
ABSTRACT
Although dropout has achieved great success in deep learning, little is known about how it helps the training find a good generalization solution the high- dimensional parameter space. In this work, we show that the training with dropout finds the neural network with a flatter minimum compared with standard gradient descent training. We further study the underlying mechanism of why dropout finds flatter minima through experiments. We propose a Variance Principle that the variance of noise is larger at the sharper direction of the loss landscape. Existing works show that SGD satisfies the variance principle, which leads the training to flatter minima. Our work show that the noise induced by the dropout also satisfies the variance principle that explains why dropout finds flatter minima by experi- ments on various datasets, i.e., CIFARIO0, CIFARIO, 10, MNIST and synthetic data, and various structures, i.e., fully-connected networks and large residual convolu- tional networks. In general, our work points out that the variance principle is an important similarity between dropout and SGD that lead the training find flatter minima and obtain good generalization.
INTRODUCTION
Dropout is used with gradient-desseettbaase algorithms for training DNNs (Hinton al., 2012 Srivastava et al. 2014). During training, the output of each neuron multiplied with a random variable with probability as and 1 1 p zero. Note that p is called dropout rate, and every time for computing concerned quantity, the variable randomly sampled at each feedforward operation. Dropout has been an indispensable trick in the training of deep neural networks (DNNs), however, with very little understanding.
Similar to SGD, training with dropout is equivalent to that with specific noise. To understand what kind of noise benefits the generalization of training, we proposes variance principle of a noise, that is,
Variance Principle: the variance of noise is larger at the sharper direction of the loss landscape.
If noise satisfies the variance principle, it can help the training select flatter minima and leads the training to better generalization. As shown in Zhu et al. (018);Feng & Tu (202T). the noise in SGD satisfies the variance principle and SGD can find flatter minima and obtain better generalization
(Keskar et al. 2016 Neyshabur et al. 2017).
In this work, we study the characteristic of minima learned with dropout. We show that compared with the standard gradient descent (GD), the GD with dropout selects flatter minima. As suggested by many existing works (Keskar al. ( are more likely to have better generalization and stability. We then put efforts to show that the noise induced by the dropout satisfies the variance principle, which explains why dropout finds flatter minima.
al. 2017] Zhu et al.) 2018)) flatter minima
To examine the variance principle, we explore the relation between the flatness of the loss landscape and the noise structure induced by dropout at minima through three methods and obtain a consistent result that the noise is larger at the sharper direction the loss landscape to help the training se- lect flatter minima. Our experiments are conducted over synthetic data and fully-connected neural networks as well as modern datasets and models such as, MNIST, CIFARIO and CIFAR- 1-0 and ResNet-20, thus our conclusion is rather general result.

---

Under review as a conference paper at ICLR 2022
First, we examine the inverse srriance-fllaness relation, similar to Feng & Tu (021) We define the flatness of a minimum at one direction by the length of the largest interval in the considered direction, which covers the minimum and no point in the interval has loss larger than twice of the loss of the minimum, denoted by Pp for the direction p. We then consider two definitions of the noise structure, i.e,, random trajectory covariance E, and random gradient covariance Eg- For the random trajectory covariance, we train the network to an "exploration phase" (Shwartz-Ziv & Tishby| (2017). where the loss decreases with a very slow speed and then sample parameter sets (0. from N consecutive training steps to compute the covariance, where 0; is the network parameter set at step i. For the gradient covariance, we train the network until the loss is very small and then freeze the training- We sample N gradients {g: )=1 with different dropout variables to compute the covariance. In each sample, the dropout rate is fixed. For both random trajectory covariance and random gradient covariance, we perform principal component analysis (PCA) and obtain similar results. We find that at the direction of larger variance (larger cigen-value), the loss landscape of the minimum is sharper, i... inverse variance- flattesss relation.
Second, we study the relation between the Hessian and the noise structure induced by dropout. The eigenvalues of the Hessian of the loss at minimum are also often used to indicate the flatness. The landscape at an eigen-ding digen-direction is claimed sharper if the corresponding eigen-value is larger. For each eigen-direction Vj. we project the parameter trajectory {0.) or gradients ( to the direction of vj and compute the variance. We find that the noise variance is larger at the direction with a larger eigen-value.
Third, we show that the Hessian matrix aligns well with the random gradient covariance of gradients (g: J.v i.e., their eigen directions of large eigen-values are close, similar Zhu et al. (2018).
These empirical works show that the noise structure induced by the dropout tends to have larger variance in order to escape the sharper direction, i.e., variance principle. thus, leading to flatter minima. These characteristics of dropout are very similar to SGD (Keskar et al. 2016 Zhu et al.) 2018 Feng & Tu, (02I). The similarity between dropout and SGD suggests that modelling their similarity may be key to understanding how and what stochasticity benefits the training.
2 RELATED WORKS
Dropout was proposed as a simple way to prevent neural networks from overfitting, and thus improv- ing the generalization of the network, to a certain extent (Hinton et al. 2012, Srivastava et al., 2014). Many works aim to find an explicit regularization form of dropout. Wager et al. (2013) studies the explicit form of dropout on linear regression and logistic problem, but for studying non- linelr neural network, it is still unclear how to characterize the effect of dropout by an explicit regularization term. McAllester 2013) presents PAC-Bayesiie bounds, and Wan et al. (2013). Mou et al. (2018) derives Rademacher generalization bounds. These results show that the reduction of complexity brought by dropout is Op)). where is the probability of keeping an element in dropout. Mianjy & Arora (2020) show that dropout training with logistic loss achieves e-suboptimality in test error in O(1/e) iterations. All of the above works need specific settings, such as norm assumptions and logistic loss, and they only give rough estimate of the generalization error bound, which usually consider the worst case. However, it is not clear what is the characteristic of the dropout training process and how to bridge the training with the generalization. In this work, we show that dropout noise has a special structure, which closely relates with the loss landscape. The structure of the effective noise induced by the dropout may be a key reason why dropout can find solutions with better generalization.
Many researches have empirically shown that SGD can improve the generalization performance of neural networks through finding a flatter solution (Li et al. 2017 7 Jastrzebski et al. 2017] 2018). This work utilizes the current understanding of SGD to study dropout and shows that much similarity is shared between SGD and dropout.
The flatness of the solution is important aspect of understanding the generalization of neural net- works (Keskar et al.) 2016 Neyshabur et al., 2017 Zhu et al. 2018). A number of works suggested that the learning rate and batch size determine the flatness of the solutions (Jastrzebski et al. 2017 2018 Wu et al.) 2018). Li et al. (2017) propose a visualization method of the loss landscape at I-d cross-section visualize the flatness.
2

---

Under review as a conference paper at ICLR 2022
Three types of experiments explain why dropout finds flat minima. Var(Proja, is the variance of the network parameters or gradients projected in the characteristic direction of the Hessian matrix.
Table 1:
Dropout covariance Σ
Trajectory variance Σg
gradient variance E
Interval flatness F
X vs. F.. Fig.
{ [Var(Proje,, (S)),, (.)))), Fig.
Hessian flatness X(H)
Alignment: Tr(HE, Fig. 6
Feng & Tu (202T) investigate the connection between SGD learning dynamics and the loss land- scape through the principal component analysis (PCA), and show that SGD dynamics follow a low- dimensional trift-diffusion motion the weight space. Through characterizing the loss landscape by its flatness in each PCA direction around the solution found by SGD, they also reveal robust in- verse relation between the weight variance and the landscape flatness in PCA directions, thus finding that SGD serves landscape dependent annealing algorithm to search for flat minima.
Zhu et al. (2018) study a general form of gradient based optimization dynamics with unbiased noise to analyze the behavior of SGD on escaping from minima and its regularization effects. They also introduce an indicator to characterize the efficiency of escaping from minima through measuring the alignment of noise covariance and the curvature of loss function and thus revealing the anisotropic noise of SGD.
3 PRELIMINARY
3.1 DROPOUT
Consider an L-layer neural network far)).. With dropout (Srivastava et al. 2014), the feedforward operation in network fo()) is
10 (a=) x
rj ~ Bernoullipp)
(2)
10 (a)) = pIl a (Wil-1) 1/0--111--111 111 (x)
(xx)
+ 6 b"-11)
1515
(3)
fo(x) (x) wi p - (x) b16-11
where p is the dropout rate, Will m xmr blll R mi+1, mg = dn = d, my do. a is
scalar function and "o" means entry-wise operation. We denote the set of parameters by
0 (Wlo) w ..... W 00 D
B1--1)),
3.2
INTERVAL FLATNESS
We use the definition of flatness in Feng & Tu (202)). For convenience, we call it interval flatness Around specific solution 0% we compute the loss function profile L, along the direction v:
(80) = L(0ī + 80v).
The interval flatness F. defined as the width of the region within which (.( < 2L. (0). We determine F, by finding two closest points 0¹, < 0 and 0" > 0 on each side of the minimum that satisfy (e,)
L. ( 2L.,()). The scale factor 2 is used in Feng Tu 2021). and after our test, the result is not sensitive to the selection of this factor. In this work, we follow their experimental scheme to show the similarity between dropout and SGD. The interval flatness defined as:
F, = 0, -' O'. A larger value of F, means flatter landscape in the direction
(5)
3

---

Under review as a conference paper at ICLR 2022
3.3 RANDOMNESS INDUCED BY DROPOUT
3.3.1 RANDOM TRAJECTORY DATA
The training process of neural networks are usually divided into two phases, fast convergence and exploration phase (Shwartz-Ziv & Tishby 2017). Feng & Tu (202I)'s work focuses on the behavior of networks in the exploration phase. In this work, we follow the experimental scheme in Feng & Tu (2021) to show the similarity between dropout and SGD. This can be understood by frequency principle (Xu et al. 2019 2020), which states that DNNs fast learn low-frequency components but slowly learn high-frequency ones.
We collect parameter sets Spara (0.) from N phase, where 8, is the network parameter set at step i.
consecutive training steps in the exploration
3.3.2 RANDOM GRADIENT DATA
However, due to the limitation of the number of sampling points, we often need larger time interval for sampling. Although the network loss is small, compared with the initial sampling parameters, the network parameters could have large changes during the long-time sampling. Therefore, much extra noise may be induced. Meanwhile, for dropout, is difficult get a small loss value on large networks and datasets, therefore, model parameters often have large fluctuations. Based on this problem, we propose a more appropriate sampling method to avoid additional noise caused by sampling parameters in large time interval. We train the network until the loss is very small and then freeze the training. We sample N gradients of the loss function w.r.t. the parameters with different dropout variables, i.e., S grodd = { In each sample, the dropout rate is fixed. In this way, we can get the noise structure of dropout without being affected by parameter changes caused by long-term term training.
3.4 INVERSE VARIANCE- AARIANCE-PAATNESS RELATION
We study the inverse variance flatness relation for both random trajectory data and random gradient data. For convenience, we denote data as S and its covariance as Σ.
3.4.1 INTERVAL FLATNESS VS. VARIANCE
Following Feng & Tu 2021) we perform principal component analysis (PCA) for Σ. Denote (x as the i-th eigenvalue of Σ, and denote its corresponding eigen-vector v,(2). We then compute the flatness of the loss landscape in the direction ii (E), and denote as (a)()))) Finally, we display the scatter plot of (Mc(E), F (.)).
3.4.2 PROJECTED VARIANCE VS. HESSIAN FLATNESS
The method in Feng& & Tu (202T), requires many training steps to obtaín the covariance. To obtain the variance induced by the dropout fixed position 0, propose another way to characterize the inverse variance-flatness relation. We use the the eigen-value (,(I) of the Hessian H of the loss landscape 0 to denote the sharpness. Hessian matrix is the matrix obtained by the second derivative of the loss function the neural network with respect to the parameter vector of neural network. Here, the parameter vector is vector consisting of all the parameters of network. The variance induced by dropout at the corresponding eigen-direction uy (H) is computed by the follow- ing procedure. For each eigen-direction v; of Hessian H, we project the sampled parameters or the gradients of sampled parameter S to direction v, by inner product, denoted by Proje, Then, the variance direction v; (H) is the variance of dataset Proje, (S), denoted by Var(Proje, (S)). Finally, we display the scatter plot of (Var(Proj,,, (S)), >; (HH)
3.5 ALIGNMENT BETWEEN HESSIAN AND GRADIENT COVARIANCE
We use the method in Zhu et al. 2018 to quantify the alignment between the noise structure and the curvature of loss surface. For each training step i, we calculate the alignment parameter T.:
T Tr E.)
4

---

Under review as a conference paper at ICLR 2022
where Σ, is the sth- covariance matrix generated by dropout layers and H, is the ith-step Hessian matrix of network parameters.
4 EXPERIMENTAL SETUP
To understand the effects of dropout, we trained number of networks with different structures. We consider the following types of neural networks: 1) Fully-connected neural networks (FNNs) trained by MNIST. For FNN, all parameters are initialized by Xavier initialization
Bengio, 2010). 2) Convolutional neural networks (CNNs) trained by CIFAR-10. For CNN, all parameters are initialized by He initialization (He et al. 2015). The loss of all our experiments is cross entropy loss. For ResNet, all parameters are initialized by He initialization (He et al. 2013). The loss of all our experiments is cross entropy loss.
3) Deep residual neural networks (ResNets) (He et al. (2066) trained by CIFAR-100
For Fig. (a), we use the FNN with size 784 1024 - 1024 - 10. We add dropout layers behind the first and the second layers with dropout rate of 0.8 and 0.5, respectively. We train the network using default Adam optimizer (Kingma & Ba 2013) with a learning rate of 0.0001.
For Fig. (b). we use vgg-9 (Simonyan & Zisserman 2014) compare the loss landscape flatness w/o dropout layers. Models are trained using GD with Nesterov momentum, training-size 2048 for 300 epochs. The learning rate was initialized at O. 1, and divided by factor of 10 epochs 150, 225 and 275. We only use 2048 examples for training to compromise with the computational burden.
we use ResNet-20 (He et al. 2016) compare the loss landscape flatness w/o dropout layers. Models are trained using GD. training-size 50000 for 1200 epochs. The learning rate was initialized at 0.01. Since the Hessian calculation of ResNet takes much time, for the ResNet experiment, we only perform it at a specific dropout rate and learning rate.
For Fig. (c). 3 5
For Fig. 2 6 we use the FNN with size 784 50 50 10. We train the network using GD with the first 10,000 training data the training set. We add dropout layer behind the second layer. The dropout rate and learning rate are specified and unchanged in each experiment.
It is worth noting that, in order to avoid the influence of the noise induced by SGD in our experi- ments, all our networks are trained using GD, so it is difficult for us to verify on larger datasets such as ImageNet.
5 DROPOUT FINDS FLATTER MINIMA
Dropout is almost ubiquitous in training deep networks. It is interesting and important to understand what makes dropout improve the generalization of training neural networks. Inspired by the study of SGD (Keskar et al. 2016), we explore the flatness of the minima found by dropout.
The loss landscape of DNNs is highly non-convex and complicate (Skorokhodov Burtsey 2019 but with certain characteristics, such as embedding principle (Zhang et al.) 2021) shows that the loss landscape of any network "contains" all critical points of all narrower networks, in the sense that, any critical point of any narrower network can be embedded to a critical point of the target network preserving its output function. It is impractical to visualize the loss landscape in the high- dimensional space. To compare two minima, 0 and 0', in loss landscape, et al. (016) is to visualize 1-d cross-section of the interpolation between and 0'. However, em- pirical studies (Li et al. 2017) show that this simple way could be misleading when O and O' have difference of orders of magnitude. et et al. (2017) visualizes loss functions using filter-wise normal- ized directions remove the scaling effect. We adopt this method (Li et al. 2017) in this work as follows. To obtain direction for a network with parameters O, we begin by producing a random Gaussian direction vector with dimensions compatible with 0. Then, we normalize each filter in d to have the same norm of the corresponding filter in O. In other words, we make the replacement d.,;
simple way (Keskar
d [d., ||O,,]l, ||. where d,,, represents the jth filter (not the jth weight, for FNN, one layer is one filter.) of the ith layer of d. and || denotes the Frobenius norm. We use f(a) = (9 + ad) to characterize the loss landscape around the minima obtained with dropout layers 0 and without dropout layer 0- For all network structures shown in Fig. [ dropout can improve the generaliza- tion of the network and find flatter minimum. In Fig. I [). (b), for both networks trained with and
5

---

Under review as a conference paper at ICLR 2022
without dropout layers, the training loss values are all closed to zero, but their flatness and general- ization are still quite different. In Fig. [T,, due to the complexity of the dataset and network, and the large number of dropout layers, the loss value of network with dropout layers is larger than the one without dropout layer. However, the network with dropout layers still finds a flatter minimum with better generalization.
whhoot
without dropout dropost
without arapost
40
f 0.05
0.00
-1
1166 -0.5
0.0
0.8
1.6
-1.
-6.5
0.0
0.5
1.0
(a) flatness of FNN
(b) flatness of vgg-9
(c) flatness of ResNet-20
Figure I: The ID visualization of solutions of different network structures obtained with without dropout layers. (a) The FNN is trained on MNIST dataset. For experiment with dropout layers, we add dropout layer after the first and the second layers. the dropout rates of the two dropout layers are 0.8 and 0.5, respectively. The test accuracy for model with dropout layers is 98.7% while 98. % for model without dropout layers. (b) The vgg-9 network is trained on CIFAR- 10 dataset using the first 2048 examples as training dataset. For experiment with dropout layers, we add dropout layers after the pooling layers, the dropout rates of dropout layers are 0.8. The test accuracy for model with dropout layers is 60.6% while 59.2% for model without dropout layers. (c) The ResNet-20 network trained on CIFAR- - dataset using the 50000 examples as training dataset. For experiment with dropout layers, we add dropout layers after the convolutional layers, the dropout rates of dropout layers are 0.8. The test accuracy for model with dropout layers 54.7% while 34. 1 % for model without dropout layers.
INVERSE VARIANCE- RELATION
Similar to SGD, the effect of dropout can be equivalent to imposing specific noise on the gradient. A random noise, such as isotropic noise, can help the training escape local minima, but can not robustly improve generalization (An 1996 Zhu et al., 2018). The noise induced by the dropout should have certain properties that can lead the training to good minima.
In this section, show that the noise induced by the dropout satisfies the variance principle, that is, the noise variance is larger along the sharper direction of the loss landscape at a minimum. The landscape-dependent structure helps the training escape sharp minima. We utilize three methods to examine the variance principle for dropout, as summarized in Table [
6.1 INTERVAL FLATNESS VS. VARIANCE
We use the principal component analysis (PCA) study the weight variations when the accuracy is nearly 100%. For FNNs, networks are trained on MNIST with the first 10000 examples as the training for computational efficiency. For ResNets, networks are trained on CIFAR-100 with 50000 examples as the training set to make more convincing conclusion on modern datasets and models. The networks are trained with full batch for different learning rates and dropout rates under the same random seed (that is, with the same initialization parameters). After the loss is small enough, we sample the parameters or gradients of parameters times (N = 3000 in this experiment) and use the method introduced in Sec. 3.3 to construct covariance matrix Σ by the weights Spara or gradients Syrod between two hidden layers. The PCA was done for the covariance matrix E. We then compute the interval flatness of the loss function landscape at sigen-directions, i.e., { [ LN Note that the PCA spectrum {><(E) indicate the variance of weights Spare or gradients S grad at corresponding digen-directions..
As shown in Fig. for different learning rates and dropout rates, there is an inverse relationship between the interval flatness of the loss function landscape (D) and the dropout variance,

---

Under review as a conference paper at ICLR 2022
Le., the PCA spectrum (x, We approximately (F), h= and {M{()) for different dropout rates and learning rates. More detailed, for small flatness part, the variance of noise induced by dropout is generally large, which indicates that the noise induced by dropout has larger variance in sharp directions, for large flatness part, the loss landscape flatter, the linear relationship more obvious, we can see clearer asymptotic behavior in the results. Overall, Fig.
power-law relationship between
observe the negative correlation between the eigenvalues and flatness in
10
p:o. 0, 00:005 05
9, In: 1
10
pro. 9, In:0 05
slape -13
6, In:01
10-1
10-
p:0.9, In:005
10-
slope
10
p:0.9, In:0.1
p.0.8, 1n:.05
p.0.8, In:01
10a
10
FuD
Fxte
(a) datasets S sampled from parame- (b) datasets S sampled from gradi- ents of parameters
ters
Figure 2: The inverse relation between the variance {}:()) and the flatness (F for different choices of dropout rate p and learning rate Ir. The FNN trained on MNIST dataset using the first 10000 examples training dataset. The PCA is done for different datasets S sampled from parameters for (a) and sampled from gradients of parameters for (b). The dash lines give the approximate slope of the scatter.
10
10-4
p.O.8, In0.01
p:o.8, In0.02
10-1
10-
10-
10-1
slope 3378
slope
10-t
10a
Fwo 10-
10
10
Fum
(a) datasets S sampled from parame- (b) datasets S sampled from gradi- ents of parameters
Figure 3: The inverse relation between the variance (A₃ (E)) and the flatness (F) NN for dif- ferent choices of dropout rate and learning rate Ir. The ResNer is trained on CIFAR- 100 dataset. The PCA done for different datasets S sampled from parameters for (a) and sampled from gradi- ents of parameters for (b). The dash lines give the approximate slope of the scatter.
6.2 PROJECTED VARIANCE VS. HESSIAN FLATNESS
The eigenvalues the Hessian of the loss a minimum are also often used to indicate the flatness. A large eigenvalue correspond to sharper direction. In this section, study the relationship between eigen-values of the Hessian H of parameters at the end point of training and the variances of dropout at corresponding eigen- direediirecsooss As mentioned in the Preliminary section, sample the parameters or gradients of parameters 1000 times, that is = 1000. For each eigen-direction v; of Hessian H, we project the sampled parameters or the gradients of sampled parameter to direction v; by inner product, denoted by Proje, (S). Then, we compute the variance of the projected data, i.e. Var(Projæ, (S))-
As shown in Fig- we find that there is also power-law relationship between (A, H for different dropout rates and learning rates, matter if is sampled
and (Var(Projo,
7

---

Under review as a conference paper at ICLR 2022
from parameters gradients of parameters. The positive correlation between the eigenvalue and the projection variance show the structure the dropout noise, which helps the network escape the bad minima. At the same time, shown in Fig. 2 5 see that gradient sampling has a more obvious linear structure than parameter sampling, which shows that gradient sampling can better avoid the interference of irrelevant noise.
10
10
0.33
0.O.9. In 0.05
10-1
VariPrl,.15))
10-11
Var(Proi (5)
p: 0.9.9 In
10-23
0.9,
0.05
10-12
slope
10-17
0.05
p: 0.8. 0.1
10-20
10-14
10
10-11
10-1:
10
10
A,00
AdHI
(a) datasets S sampled from parame- (h) datasets S sampled from gradi- ters
ents parameters
Figure 4: The relation between the variance { [Var(Proj,,, (S)) and the eigenvalue { for different choices of dropout rate p and learning rate (r. The FNN is trained on MNIST dataset using the first 10000 examples as training dataset. The projection is done for different datasets $ sampled from parameters for (a) and sampled from gradients of parameters for (b). The dash lines give the approximate slope of the scatter.
10-2
10
p:0.8, Ir:0.05
p:0.8, 1r:0.05
VariProl, (5))
Var,Proj, LSM
10-2.
10-4
10-7
stope
10-10
slopp 1.33
10-12
10-2
10
10
10
A000
A(H)
(a) datasets S sampled from parame- datasets S sampled from gradi- ters
ents parameters
Figure 5: The relation between the variance (Var(Proj,,, (S))), and the eigenvalue { for different choices of dropout rate p and learning rate Ir. The ResNet is trained CIFAR- 100 dataset using all the examples as training dataset. The projection done for different datasets S sampled from parameters for (a) and sampled from gradients of parameters for (b). The dash lines give the approximate slope of the scatter.
6.3 ALIGNMENT BETWEEN HESSIAN AND GRADIENT COVARIANCE
Zhu et al. (2018) show that the alignment indicator Tr(HE) plays an crucial role for stochastic processes escaping from minima, where H is the Hessian and Σ is the noise covariance. In this sub- section, we study the alignment between the Hessian and the random gradient covariance at each training step. Note that the training is performed by GD without dropout. At step i, sample the gradients of parameters 1 y by tentatively adding dropout layer between the hidden layers. For each step we the compute Tr(H,E,), where H, is the Hessian of the loss at the parameter set step and is the covariance of {g/
In order to show the anisotropic structure, we construct the isotropic noise for comparison, i.e., E, the covariance matrix Σ, where D is the number parameters. In our experiments, D 2500. As shown in Fig. 6 in the whole training process under different learning rates
Σ, = D
8

---

Under review as a conference paper at ICLR 2022
and dropout rates, Tr, is much larger than Tr(H, indicating the anisotropic structure of dropout noise and its high alignment with the Hessian matrix.
10
10-3
Tr(HZ)
trace
p: 0.8, Ir: 0.1
10-5
Tr(HE)
p: 0.8, Ir: 0.05
p: 0.9, Ir: 0.1
p: 0.9, Ir: 0.05
10-7
50
100
150 epoch
200
250 300
Figure 6: Comparison between Tr,H,E,) and Tr(H, in each training epoch for different choices of dropout rate and learning rate Ir. The FNN trained on MNIST dataset using the first 10000 examples training dataset. The solid and the dotted lines represent the value of Tr(H,E,) and HH), respectively.
CONCLUSION AND DISCUSSION
In this work, propose variance principle that noise is larger the sharper direction of the loss landscape. a noise satisfies the variance principle, helps the training select flatter minima and leads the training to good generalization. We empirical show that dropout finds flat minima during the training and examine that the dropout satisfies the variance principle to explain why dropout finds flat minima various datasets and neural network structures by the following three perspectives: interval flatness vs. variance, projected variance vs. Hessian flatness and alignment between Hessian and gradient covariance.
The dropout and the SGD are common in sharing the variance principle. Based on this, there could be a general theory framework that can model both dropout and SGD understand why they have better generalization. The research on understanding the good generalization of dropout and SGD far from complete. As starting point, the variance principle shows promising and reasonable direction for understanding the stochastic training of neural networks.
REFERENCES
Guozhong An. The effects adding noise during backpropagation training on a generalization performance. Neural computation, 8(3):643-674, 1996.
Yu Feng and Yuhai Tu. The inverse sitiince-flatteee relation in stochastic gradient descent is critical for finding flat minima. Proceedings of the National Academy of Sciences, 118(9), 2021.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference artificial intelligence and statistics, pp. 249-256. 2010.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.

---

Under review as a conference paper at ICLR 2022
Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdi- nov. Improving neural networks by preventing co-adaptation feature detectors. arXiv preprint arXiv:1207.0580. 2012.
Stanislaw Jastrzebski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima in sgd. arXiv preprint arXiv:1711.04623, 2017.
Stanislaw Jastrzebski, Zachary Kenton, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. On the relation between the sharpest directions of dnn loss and the sgd step length. arXiv preprint 1XX.:0007.00031 2018.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv: 04666, 2016.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In ICLR (Poster), 2015.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss land- scape of neural nets. arXiv preprint arXiv:1712.099133 2017.
David McAllester. pac-bayesian tutorial with a dropout bound. arXiv preprint arXiv:1307.211 2013.
Poorya Mianjy and Raman Arora. On convergence and generalization dropout training. Advances in Neural Information Processing Systems, 33, 2020.
Wenlong Mou, Yuchen Zhou, Jun Gao, and Liwei Wang. Dropout training, data-dependent reg-
ularization, and generalization bounds. In International conference machine learning, pp- 3645-3653. PMLR, 2018.
Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nathan Srebro. Exploring gener- alization in deep learning. arXiv preprint arXiv: 1eXXiv:1700... 1706.08947 2017.
Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via informa- tion. arXiv preprint arXiv: 1eXXiv:1003.000 7000000800000 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 140.:v11099.166,,. 2014.
Ivan Skorokhodov and Mikhail Burtsev. Loss landscape sightseeing with multi-point optimization. arXiv preprint arXiv: 2019.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-199 1958, 2014.
Stefan Wager, Sida Wang, and Percy S Liang. Dropout training as adaptive regularization. Advances in neural information processing systems, 26:351-359, 2013.
Li Wan, Matthew Zeiler, Sixin Zhang, Yann Lecun, and Rob Fergus. Regularization of neural net- works using dropconnect. In In Proceedings of the International Conference on Machine learning. Citeseer, 2013.
Lei Wu, Chao Ma, et al. How sgd selects the global minima pver-parameterized learning: A dynamical stability perspective. Advances in Neural Information Processing Systems, 31:8279- 8288, 2018
Zhi-Qin John Xu, Yaoyu Zhang, and Yanyang Xiao. Training behavior of deep neural network in frequency domain. International Conference on Neural Information Processing, pp. 264-274. Springer, 2019.
10

---

Under review as a conference paper at ICLR 2022
Zhi-Qin John Xu, Yaoyu Zhang, Tao Luo, Yanyang Xiao, and Zheng Ma. Frequency principle: Fourier analysis sheds light on deep neural networks. Communications in Computational Physics, 2865):1746-1767, 2020.
Yaoyu Zhang, Zhongwang Zhang, Tao Luo, and Zhi-Qin John Xu. Embedding principle of loss landscape of deep neural networks. arXiv preprint arXiv:2105. 14573, 2021.
Zhanxing Zhu, Jingfeng Wu, Bing Yu, Lei Wu, and Jinwen Ma. The anisotropic noise in stochastic gradient descent: Its behavior of escaping from sharp minima and regularization effects. arXiv preprint aivi 803,000399 2018.


Reviews:
[{"review_id": "itxUS5RkeCu", "reviewer": "Reviewer_mVLJ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I am not quite familiar with this area. But I think the topic is quite interesting. \n\nThe main concerns are:\n\n1. It seems most of the claims are not theoretically supported. I tried to find something in the supplementary material but only code is found.\n2. It is observed that regularization also helps and find flatter minimizer and there is work to bridge the dropout and reguarliation: Dropout Training as Adaptive Regularization. More discussion is encouraged.\n", "summary_of_the_paper": "This paper tries to study the effect of dropout by studying the trajectory of optimization via dropout. It is claimed that dropout can help find flatter minimizer.", "summary_of_the_review": "See main review.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "galbF5aLfw", "reviewer": "Reviewer_HpHx", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In general, the paper is interesting; however, the novelty and contribution are limited. Most of the tools are from the existing work, and neither insights nor empirical results are strong enough to make this paper stands out.\n\nThe paper may miss some important discussions regarding the relationship between dropout, regularization, and generalization bounds, as discussed in [1, 2, 3]. Adding the discussion and extra experiments would increase the value of the current draft.\n\nThe paper divides the training process into two phases, i.e., fast convergence and exploration phase. However, this phase division is relatively rough and has no support (authors are encouraged to cite some prior works to support their justification).\n\nThe computation method for H on neural networks has never been introduced in the main text, making the current empirical results questionable. The computation cost of $H_i$ remains unclear; this comment also applies to the quality of hessian estimation, which may significantly impact the final numerical observation.\n\nIt is a bit hard to identify the (inverse) relation between variance and the flatness in Figure 2 and Figure 3a, unlike the clear pattern in Figure 3b. It would be great if the authors could polish the figures and/or explain the current observation.\n\n## Reference\n1. Dropout Training, Data-dependent Regularization, and Generalization Bounds\n2. On Convergence and Generalization of Dropout Training\n3. A PAC-Bayesian Tutorial with A Dropout Bound", "summary_of_the_paper": "The paper uses the tools from different prior works to study the flatness brought by dropout. The paper shows that the noise induced by the dropout has a similar structured introduced by SGD that leads the training to find flatter minima.", "summary_of_the_review": "In general the paper is interesting; however, the novelty and contribution is limited. Most of tools are from the existing work, and neither insights nor empirical results are strong enough to make this paper stands out.\nAdding the discussion related to the generalization bound (as pointed in the provided literature) and extra experiments would increase the value of the current draft.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "L_lUAdJWhxx", "reviewer": "Reviewer_dtVh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "My detailed comments are given as below. \n\nStrength: \n1. This paper is well written and the motivation is clear to me. Dropout is a widely used technique for improving the generalization performance of various learning algorithms, but there is limited understanding how this scheme helps to achieve this better generalization. This paper provides some empirical justification showing that dropout can lead to flatter minima. \n\nWeakness:\n1. Some parts need to be explained more clearly. For example, I have a hard time understanding Section 3.3., where interval flatness is defined as the width of the region where $L_v(\\delta \\theta)<2L_v(0)$. I am wondering why $2L_v(0)$ is chosen here, and whether the factor $2$ is sufficiently general to characterize the landscapes for all settings. I am not familiar with this part, but I hope the authors should explain it a little bit.  \n\n2. My second concern is the novelty of the results. So far I see that some important tools, e.g., flatness definition, PCA, follow from existing studies, although there is some new treatment in obtain the variance. In addition, the empirical results are not that surprising to me (but I agree that it is not clear why the noise induced by dropout plays some similar role as SGD), and a more interesting part may be to provide some theoretical justification even in some simplified settings. \n\n3. Experiments can be strengthened. I see that all experiments are conducted over small datasets (e.g., CIFAR-10) and small models. To make a more convincing conclusion that dropout can lead to flatter minima, I believe experiments on modern datasets and models (e.g., ResNet on ImageNet) need to be run. \n\n\n\n\n\n", "summary_of_the_paper": "This paper provides an empirical study on how the dropout can lead to minima within a flatter landscape (hence a better generalization performance). Based on the definition of minimizer flatness (i.e., eq. 5) and random trajectory data (parameters from a optimization path and gradients when loss is stably small), the authors first show that dropout can help to find flatter minima, and also find a inverse relation between the flatness (the length $F_v$) and the algorithmic variance (i.e., in area with sharper direction, algorithm with dropout has a larger variance), and further find that for areas with larger eigenvalues (i.e., with sharper landscape), the variances of the algorithm along the directions of their eigenvectors are also large. Such empirical results together demonstrate that the dropout can play a good role in improving the generalization of existing algorithms by introducing certain level of noise. ", "summary_of_the_review": "I think this paper provides some empirical explanation on why dropout is useful to achieve a flatter minima. However,  I feel it is more interesting to rigorously explain this phenomenon given that many works already prove that SGD can reach flatter minima. In addition, the novelty of this paper is not that high based on my understanding. Therefore, I am slightly negative about this paper, but I am open to adjust my score based on the authors' feedback and other reviewers' comments.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8tKUqClzVUW", "reviewer": "Reviewer_Y6pQ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nTo my best knowledge, this is a first work that shows the relationship between Dropout and the flatness of minima. It could provide another perspective for understanding the effectiveness of Dropout. \n\nHowever, my main concern about this manuscript is the complete missing of theoretical analysis, which makes the current version of the paper not strong enough for publication. I strongly suggest the authors conduct theoretical analysis over the covariance introduced by Dropout noise and loss curvature, at least on linear cases. \n", "summary_of_the_paper": "This paper empirically found that Dropout noise could help the neural network search for the flat minima, by showing that the covariance of parameter and gradient align well with the loss curvature and have inverse relationship with interval flatness. ", "summary_of_the_review": "Complete missing of theoretical analysis. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors make an experimental case that dropout aids generalization by
promoting "flatter minima".

The reviewers felt that the work reported in this paper makes a useful step
forward on a question of central interest.  The consensus view was that the
total weight of evidence presented was not sufficient for publication in
ICLR.  The paper could be strengthened was more extensive and varied experiments
and/or theoretical analysis.
Decision: Reject