Weight Expansion: A New Perspective on Dropout and Generalization
Authors: Gaojie Jin
Xinping Yi
Pengfei Yang
Lijun Zhang
Sven Schewe
Xiaowei Huang
Venue: Accepted by TMLR ()

Abstract: While dropout is known to be a successful regularization technique, insights into the mechanisms that lead to this success are still lacking. We introduce the concept of weight expansion, an increase in the signed volume of a parallelotope spanned by the column or row vectors of the weight covariance matrix, and show that weight expansion is an effective means of increasing the generalization in a PAC-Bayesian setting. We provide a theoretical argument that dropout leads to weight expansion and extensive empirical support for the correlation between dropout and weight expansion. To support our hypothesis that weight expansion can be regarded as an indicator of the enhanced generalization capability endowed by dropout, and not just as a mere by-product, we have studied other methods that achieve weight expansion (resp.\ contraction), and found that they generally lead to an increased (resp.\ decreased) generalization ability. This suggests that dropout is an attractive regularizer, because it is a computationally cheap method for obtaining weight expansion. This insight justifies the role of dropout as a regularizer, while paving the way for identifying regularizers that promise improved generalization through weight expansion.

Published in Transactions on Machine Learning Research (09/2022)
Weight Expansion: A New Perspective on Dropout and Generalization
Gaojie Jin', Xinping YI²", Pengfei Yang², Lijun Zhang?, Sven Schewe', Xiaowei Huang!"
Department of Computer Science, University of Liverpool
Department of Electrical Engineering and Electronics, University of Liverpool
State Key Laboratory of Computer Science, Institute of Software Chinese Academy of Sciences Corresponding author g g.jin3 jiin@@iieeppool.oo ac. uk, sven. schewee @ @liverpool.ac. uk, riaowei. huangg divennnlieeppool.coo
rinping-yy wwiiiivrroooll.oooo uc. uk; vungpffios.ac.cm, shanglj@ios.ac.cm aww../...s..cc.oo
Reviewed OpenReview: http:://opaaaeeeeee..ee/////rrrd
Abstract
While dropout is known be a successful regularization technique, insights into the mecha- nisms that lead this success are still lacking. We introduce the concept of weight expansion, an increase in the signed volume of parallelotope spanned by the column of the weight covariance matrix, and show that weight expansion is an effective means of increasing the generalization in PAC-Bayesian setting. We provide a theoretical argument that dropout leads to weight expansion and extensive empirical support for the correlation between dropout and weight expansion. To support our hypothosis that weight expansion can be regarded as an indiculor of the enhanced generalization capability endowed by dropout, and not just as a mere by-product, we have studied other methods that achieve weight expansion (resp. contraction), and found that they generally lead to increased (resp. decreased) generalization ability. This suggests that dropout is an attractive regularizer, because it is computationally cheap method for obtaining weight expansion. This insight justifies the role of dropout as regularizer, while paving the way for identifying regularizers that promise improved generalization through weight expansion.
vectors
1 Introduction
Research on why dropout is effective in improving the generalization ability of neural networks has been intensive. Many intriguing phenomena induced by dropout have also been studied in this research (Gao et al., 2019; Lengerich et al., 2020; Wei et al., 2020). In particular, it has been suggested that dropout optimizes the training process (Baldi & Sadowski, 2013; Wager et al., 2013; Helmbold & Long, 2015; Kingma et al., 2015; Gal & Ghahramani, 2016; Helmbold Long, 2017; Nalisnick et al., 2019) and that dropout regularizes the inductive bias (Cavazza al., 2018; Mianjy et al., 2018; Mianjy Arora, 2019; 2020).
A fundamental understanding of just how dropout achieves its success a regularization technique will not only allow us to understand when (and why) to apply dropout, but also enable the design of new training methods. In this paper, suggest new measure, weight expansion, which both furthers understanding and allows the development of new regularizers. Broadly speaking, the application of dropout leads to non-trivial changes in the weight covariance matrix. As the weight covariance malrix is massively parameterized and hard to comprehend, we abstract to the signed volume of a parallelotope spanned by the vectors of the matrix-weight volume (normalized generalized variance (Kocherlakota Kocherlakota, 2004)) for short. Dropout increases this weight volume. Figure illustrates the weight expansion obtained by dropout and visualizes how it improves the generalization ability.

---

Published in Transactions on Machine Learning Research (09/2022)
(a) Normal CNN waight volume: 0.1592 Normal CNN feature visualization
(c) Dropout CNN weight volume: 0.3239
(d) Dropout CNN feature visualization
Figure 1: Visualization of weight volume and features of the last layer in a CNN on MNIST, with and without dropout during training. Plots (b) and (d) present feature visualization for normal and dropout CNN, respectively. We can see that the dropout network has clearer separation between points of different classes a clear sign of a better generalization ability. As we will argue in this paper, this better generalization is related to the "weight expansion" phenomenon, as exhibited in plots (a) and (c). For (a) and (c), the weight volume of the normal CNN and the dropout CNN is 0.1592 and 0.3239, respectively. That is, the application of dropout during training "expands" " the weight volume. For visualization purpose, we randomly select three dimensions in the weight covariance matrix and display their associated 3-dimensional parallclotope. Both (a) and (c) contains 3 x 3 = 9 such parallelotopes. We can see that those parallelotopes have a visibly larger volume in the dropout network. Details of the networks are provided in Appendix A.
This leads to our hypotheses that (1) weight expansion reduces the generalization error of neural networks (Section 3.1), and that (2) dropout leads to weight expansion (Section 3.2). We hypothesize that weight expansion can be an indicator of the increased generalization ability and dropout is 'merely" an efficient way to achieve it.
The weight volume is defined the normalized determinant of the weight covariance matrir (determinant of weight correlatton matriz), which is the second-order statistics of weight vectors when treated as multivariate random variable. For example, when the weight vector is an isotropic Gaussian, the weight volume is large and the network generalizes well. Likewise, if the weights are highly correlated, then the weight volume is small and the network does not generalize well. Technically, as a determinant, the weight volume serves as a measure of the orthogonality of the rows/columnn of the weight covariance matrix. The more orthogonal the rows/columns are, the larger the weight volume is s--which makes the different classes more distinguishable. Figure 1 visualizes that, with increasing weight volume, the decision boundaries of different classes become clearer. Note that, this is different from the "weight orthogonalityy proposed in Saxc et al. (2013); Mishkin & Matas (2015); Bansal et al. (2018): Weight volume is statistical properly of weight matrices treated as random variables, while orthogonality treats weight matrices as deterministic variables (more discussions are provided in Appendix B).
In Section 3.1, we provide theoretical analysis to support hypothesss (1) through an extension to the PAC- Bayesian theorem, i.e., weight expansion can tighten a generalization bound based on the PAC-Bayesian theorem. Then, in Section 3.2, we support hypothesis (2) through a onc-step theoretical analysis that dropout can expand weight volume within one mini-batch.
In addition to our theoretical arguments, our empirical results also support our hypotheses. To mitigate estimation errors, we consider two independent weight volume estimation methods in Section 4, i.e., sampling method and Laplace approximation. Section 5.1 demonstrates that the two methods lead to consistent results, which show that using dropout brings about weight expansion (coherence to hypothesis (2)) and generalizaation-imprreee
(coherence to hypothesis (1)) across data sets and networks.
Then, embed the weight volume (a measurable quantity) to new complexity measure that is based on our adaptation of the PAC-Bayesian bound (Section 3.1). Our experiments in Section 5.2, conducted on 3 x 288 network configurations, show that, comparing to existing complexity measures which do not consider weight volume, the new complexity measure is most closely correlated with generalization (coherence to hypothesis (1)) and dropout (coherence to hypothesis (2)) for these dropout networks with different dropout rates.
Finally, we use disentanglement noises (Section 4.1) to achieve weight expansion and further to improve generalization (or other noises to achieve contraction and further to destroy generalization, Section 5.3). This
2

---

Published in Transactions on Machine Learning Research (09/2022)
in particular supports hypothesis (1) and indicates that weight expansion is key indicator for a reduced generalization error, while dropout is a computationally inexpensive means to achieve it. (Disentanglement noises, e.g., are more expensive.)
2 Preliminaries
Notation. Let S be a training set with m samples drawn from the input distribution D, and E S a single sample. Loes, expected loss, and empirical loss are defined as {(f((s)) [p(ww) [EaDD[([[w [s)]; and (f(f)) = 1 (aws((((((s))) respectively, where fw(() is learning model. Note that, in the above notation, we omit the label y of the sample s, it is clear from the context. Tet W, W,, and wa be the model weight matrix, the weight matrix of I-th layer, and the element of the i-th row and the j-th column in W, respectively. Note that we omit bias for convenience. To make a clear distinction, let wi be the multivariate random variable of vec(Wi) (vectorization of Wi), and Wej be the random variable of way- W and W denote the weight matrix before and after training. respectively. Further, we use W* to represent the maximum---posterriot (MAP) estimate of W. Note that W can be seen as an approximation of W* after training. We consider feedforward neural network fw(+) that takes sample so E S as input and produces output fu, after L fully connected layers. At the (-th layer (I = 1, ,,...,,,, L), the latent representation before and after the activation function is denoted as h1 = Wiaj-1 and = act; (hi), respectively, where act(-) is the activation function.
Node Dropout. We focus on node dropout (Hinton et al., 2012; Srivastava et al., 2014), a popular version of dropout that randomly sets the output of given activation layer to O. We adopt a formal definition of node dropout from the usual Bernoulli formalism. For the layer I with the output of the activation function ar IR and given probability q E [0,1), 1), we sample scaling vector f E R.N: with independently and identically distributed elements
0 with probability q:
(m))
(1)
with probability 1 - Qi-
for i = I,...,N,, N₁, where N₁ is the number of neurons at the layer 1. With slight abuse of notation, we denote by 'ir a vector of independently distributed random variables, each of which has an one mean, and also a _ when referring to a specific realization. As such, applying node dropout, we compute the updated output after the activation as â = n O at. That is, with probability Øi an element of the output with node dropout â is reset to O, and with probability qi it is rescaled with factor q- The index of the layer I will be omitted when it is clear from the context.
vector with each element being either or I-q
3 Dropout and weight expansion
In this section, we first formally define the weight volume, which has been informally introduced in Figure 1. We aim to answer the following two questions: 1. Why does large weight volume promote generalization? 2. Why does dropout expand the weight volume?
By modeling weights of neural network random variables {wi), 1 we have the following definition.
Definition 3.1 (Weight Volume) Let Σ E((w) - (ww))((/-((wi))) [[w))) be the weight covariance matrix in a neural network. The weight volume (also called as normalized generalized variance or determinant of weight correlation matriz) is defined as
det(Zi) [[AEi
vol(w;)
E [0, 1],
(2)
'The modeling of neural network weights as random variables has been widely adopted in the literature for mutual information estimation (Xu & Raginsky, 2017; Negrea et al., 2019), PAC-Bayesian analysis (McAllester, 1999; Langford & Caruans, 2002; Dziugaite & Roy, 2017), to in Jiang al. (2019), Flatness PAC-Bayes in Dziugaitc al. (2020b)) assume Gaussian random weights scatter around the DNN, with the condition that their gencralization performance is similar the DNN's, to compute the bound in practice. The networks parameterized by these random weights can be considered to in the same local minima as they are close and have similar generalization performance. We also fallow this assumption and estimate weight volume Section 4.
just few. Under DNN PAC-Bayesian framework, previous works (sharpness PAC-Bayes
3

---

Published in Transactions on Machine Learning Research (09/2022)
where the denominator is the product of the diagonal elements in Σg.
Intuitively, the weight volume comes from the geometric interpretation of the determinant of the matrix det(Er), with normalization over the diagonal elements of the matrix. Because En a covariance matrix and thus positive semi-definite, we have vol(w)) E [0. 1] since 0 S deu(E;) S [E]e. We note that vol(w,) implies that the the weight covariance matrix does not have full rank, suggesting that some weights are completely determined by the others, while vol(w;) = suggests that the weights w/ uncorrelated. The larger vol(w;) is, the more dispersed are the weights, which means random weights lie in more spherical (i.e., flattened) local minima.
Before presenting empirical results, we first provide theoretical support for our first hypothesis by adapting the PAC-Bayesian theorem (McAllester, 1999; Dziugaite & Roy, 2017) (Section 3.1): We show that a larger weight volume leads to a tighter bound for the generalization error. In Section 3.2, we argue that dropout can expand weight volume through the theoretical derivation.
3.1 Why does large weight volume promote generalization?
First, we support hypothesis (1) on the relevance of weight expansion through an extension of the PAC- Bayesian theorem, and then verify it empirically in Section 5. The PAC-Bayesian framework, developed in McAllester (1999); Neyshabur et al. (2017); Dziugaite & Roy (2017), connects weights with generalization by establishing an upper bound on the generalization error with respect to the Kullback-Leibler divergence (DRl) between the posterior distribution Q and the prior distribution P of the weights. In the following, we extend this framework to work with vol(w:).
Let P be a prior and Q be a posterior over the weight space. For any 5 > O, with probability 1 6 over the draw of the input space, we have (Dziugaile & Roy, 2017)
Ew-elLo([w)] S [wwol[[((w]] + (Kl(())P) + In m
2(m - 1)
Given a learning setting, Dziugaite & Roy (2017); Neyshabur et al. (2017); Jiang al. (2019) assume P >=N(up,Ep) and Q (uQ Eq). Thus, (Q] can be simplified
DKL (QI 2 (x((Ep Σq) (t)) - + (up (C)--=))TP) - nو) + In det(Ep) det(Eq)
(4)
where & is the dimension of w, tr denotes the trace, and det denotes the determinant. The derivation is given in Appendix D. To simplify the analysis, Neyshabur et al. (2017; 2018a) instantiate the prior P to be a vec(W" (or 0) mean and o² variance Gaussian distribution, and assume that the posterior Q to also be a veo(W#) mean spherical Gaussian with variance o² in each direction.
We relax their assumption by letting Q be non-spherical Gaussian (the off-diagonal correlations for same layer not whereas those for different layers are 0), while retaining the assumption the o² variance in every direction (but allowing arbitrary covariance between them). This retains both ((E))1)) k and [L = (ls. for all 1. Then, we get the following Lemma.
= Ep= I, Q = ". Eq), the diagonal elements of Σq be o² and the covariances for different layers in Eq be 0. Thas, we gel
Lemma 3.1 (Weight expansion reduces (kI(Q)!P)) Let P = N(vec(W"), Ep), Ep
1 Σ DKL(Q]IP) 2
Σ
| wil1
In vol(wi)
(5)
o2
Details of the derivation are provided in Appendix D. Note that Lemma 3.1 is a theoretical analysis from prior to postcrior, in which we regard the whole training procedure as a single step. From Equations (3) and (5), we notice that the PAC-Bayesian upper bound of the generalization error becomes smaller when vol(w;) increases, which is also demonstrated by wide experiments in Section
4

---

Published in Transactions on Machine Learning Research (09/2022)
1.0
Drapout
Weight volue
0.8
0.7-
0.6-
3
0.5-
0.4
Variation of weight Epoch
(a) Normalzed covariance matrix (normal)
(c) Normalized covariance matrix (dropout)
We train two small NNs (64-32-16-10 10 network with with/without dropout) on MNIST. (b) shows the changes in weight, volume along with epochs. There are four sampling points (D234) to track the absolute normalized covariance matrix (correlation matrix) of normal gradient updates (a) and dropout gradient updates (c). The results for VGG16 on CIFAR-10 are provided in Appendix C.
Figure 2:
3.2 Why does dropout expand the weight volume?
In the above PAC-Bayesian framework, we assume the (non-spherical) Gaussian distributed weights and connect weight volume with generalization. In this subsection, connect weight. volume with dropout through correlation (of gradient updates and weights), and further to support hypothesis (2), consider the arbilvurily distribuled (with finite variance) gradient updates of (-th layer, A for normal network and A for the corresponding dropout network (all learning settings are same except for dropout). That is, let vec(Ar) and veo(A() be i.i.d. and follow an arbitrary distribution Ma(ua, ) and dropout noise ti be applied the gradient updates through
vec(Ái) (1 & fj)) O vec(Ai). Ai),
where the one vector E R.-11 and the random vector & in E N- with & being Kronecker product. As we aim to sludy the impact of in on the correlation of gradient updales, we assume vec(Ar) and vec( are ii.dd simplify the corvelation analysis. Thus, with a slight abuse of notation, denote (1 69 rj)) o veo as în to be applied on gradient updates of corresponding neurons for notational convenience, we can now obtain Lemma 3.2.
Lemma 3.2 (Dropout reduces correlation of gradient updates) Let Dig and Agy be the gradient updates for Wij and Way of some layer a normal network, where i + i', Dij and Dyy be the gradient updates for the same setting dropout network. Let vec(A), vec(Â') be i.i.d. and follow an arbitrary distribution Ma(Hs, vec(Â) = (1 @ ii) @ vec(A'). As dropout noises for differrnt nodes (units) are independent, the (absolute) correlation between Âi and Âey in a dropout network can be upper bounded by that in the normally trained network with the same selling, i.e.,
|PSy,Argl
S (1-9))
(6)
Details of the proof are given in Appendix E. The equality holds if, and only if, = 0 or PA.,Ary = 0. Lemma 3.2 shows that the dropout noise decreases correlation among gradient updates, which is also shown by experiments Figure and Appendix C. That is, the correlation among gradient updates in a normal network (Figure 2a) is obviously larger than the correlation in dropout network (Figure 2c). Next, Lemma 3.2 leads to the following lemma.
Lemma 3.3 (Dropout reduces correlation of updated weights) Given the assumptions in Lemma 3.2 and Gaussian weights, we get that the (absolute) correlation of updated weights in a dropout network is upper bounded by that in the normal network. That is,
1 weg S |ww.yta wey +
5

---

Published in Transactions on Machine Learning Research (09/2022)
Algorithm 1 Sampling Method.
Input: Convergent network fw, data set S, iterations n, Convergent network loss L(fw)
W' W
for i ==1 to do
vec(W")) f- vec(W") + 0.000.0
W' is updated by (S) with 1 epoch, 0.0001 learning rate
Collect W as a sample if [f[w) - [(Sw)] S
end for
The equality holds if and only if q 0 or Cov(wi, + Aij, wey Aeyr) 0, and the full proof is given in Appendix F. It is clear that weight volume (equals the determinant of weight correlation matrix) is one of measures of correlation among weights (refer to Figure 1), and small correlation can lead to large weight volume. We also provide more discussions about the relationship between correlation and weight volume in Appendix G. Consistent with Lemma 3.3, we show in addition, that gradient updates with large correlation (Figure 2a, normal network) can lead to large correlation of weights- thus to a small weight volume (Figure 2b), while gradient updates with small correlation (Figure 2c, dropout network) can lead to small correlation of weights and thus to large weight volume.
Note that, as the correlation during the whole training procedure is too complicated to be analyzed, Lemmas 3.2 and 3.3 adopt one-step theoretical argument (with one mini-batch update). With the same pre-condition, we prove that dropout noise can reduce the correlation among weights. Based on the one-step theoretical result, our results in Figure empirically show that this phenomenon is extensible to the whole training procedure.
4 Weight volume approximation
To mitigate estimation errors, we use two methods to estimate the weight volume. One is the sampling method, and the other is Laplace approximation (Botev et al., 2017; Ritter et al., 2018) of neural networks, which models dropout as a noise in the Hessian matrix. Laplace approximation enables us to generalize dropout to other disentanglement noises (Section 4.1). Both sampling method and Laplace approximation follow the assumption Footnote 1. Note that some of other popular regularizers, c.g., weight decay and batch normalization, improve generalization not through weight expansion (more discussions are provided in Appendix H).
First, we use sharpncss-like method (Keskar et al., 2016; Jiang et al., 2019) to get a set of weight samples drawn from (W + U) such that [C(Sw-u) [(ww) S e, where vec(U) ~ NO, () is a multivariate random variable obeys zero mean Gaussian with du = 0.1. To get the samples from the posterior distribution steadily and fastly, we train the convergent network with learning rate 0.0001 and noise U, then collect corresponding samples. We also provide the pseudo-code for sampling method in Algorithm 1. As the samples are stabilized at training loss but with different weights, we can treat. them the samples from same posterior distribution. The covariance between Wiy and way can then be computed by Cov(wij, Wey) = B[(w:j ([wyy)[wey E[wry])] Finally, can estimate vol(w) according to Equation (2).
Sampling Method.
Laplace Approximation. Laplace approximation has been used in posterior estimation in Bayesian inforence (Bishop, 2006; Ritter et al., 2018). It aims to approximate the posterior distribution p( p(w[S) by a Gaussian distribution, based on the second-order Taylor approximation of the in posterior around its MAP estimate. Specifically, for layer and given weights with an MAP estimate W on S, we have
1 Inp(w:]S) ~ Inp (In(ve)(W3)))) - { (ok-vvec())) ' "w: veecvv))))
(8)
where Σ, = "P'=EE,[AA [H.] is the expectation of the Hessian matrix over input data sample s, such that the Hessian (e)(fw(e))
matrix H, is given by Hi 3vve(W))dvee(W.)
6

---

Published in Transactions on Machine Learning Research (09/2022)
Algorithm Gradient update with disentanglement noise.
Input: minibatch {s:) activation noise Ta, node loss noise r):; noise strengths X1, A2-
Forward Propagation: apply noise A1 Te to activations.
Back Propagation: apply noise A2 1h to node loss.
Calculate g 1, Σi= Vw (((((((s)))) by considering both noises as indicated above.
Use for the optimization algorithm.
It is worth noting that the gradient is zero around the MAP estimate W so the first-order Taylor polynomial is inexistent. Taking a closer look at Equation 8, one can find that its right hand side is exactly the logarithm of the probability density function of a Gaussian distributed multivariate random variable with mean vec(W]") and covariance Σn, i.e., Wi (V(ec(W;)) Ex), where Mn can be viewed as the covariance matrix of wi.
Laplace approximation suggests that is possible estimate Σ, through the inverse of the Hcssian matrix, because E,- E.,, [H.]. Recently, Botev et. al. (2017); Ritter et al. (2018) have leveraged insights from second-order optimization for neural networks to construct Kronecker factored Laplace approximation. Differently from the classical sccond-order methods (Battiti, 1992; Shepherd, 2012), which suffer from high compulational costs for deep neural networks, it takes advantage of the fact that Hessian matrices at the 1-th layer can be Kronecker factored as explained in Martens & Grosse (2015); Botev et al. (2017). I.e.,
AL-1 Ø Hi, (e((fw
0 Hi a1
Oluuh
(9)
A1-1
H
where A E RM--IXN-1 indicates the subspace spanned by the post-activation of the previous layer, and Hi E NNN, is the Hessian matrix of the loss with respect to the pre-activation the current layer, with NI-1 and Nt being the number neurons at the (I -1-1)- 1))thh and I-th layer, respectively. Further, through the derivation in Appendix I, we can estimate vol(w;) efficiently by having
dot(Ex) det ((E((A--1))+)) N: det ((E,(7G))- N-11
(10)
and
Il Eil [II (EE MM (E.(HL) dal
Ni-1
(11)
Note that, for the case with dropout, dropout noise n is taken into account to compute E, [A and E, [P)]. That is, [-[A-1]] = Ex. [ā--( and [?A]] = E.,.. ( [f)(( where ä:- and h4 are the activation and pre-activation of the corresponding layer in a dropout network.
ah.th,
4.1 Other disentanglement noise
Whilst introducing dropout noise expand weight volume (as shown in the experiments of Sections 5.1, 5.2), it is interesting to know (1) whether there are other noises that can lead to weight expansion, and, if so, whether they also lead to better generalization performance, and (2) whether there are noises that can lead to weight contraction, and, if so, whether those noises lead to worse generalization performance.
As indicated in Equation (9) and Appendix I, the weight covariance matrix can be approximated by the Kronecker product of (E[A,-1])- and (E(H,))-1. Disentanglement noises, attempt, can be injected into either term during the gradient update procedure to increase det(B[())))- and det(E(H))) for the purpose of weight expansion. The first option is to inject na to activations during forward propagation, follows.
Activation Reverse Noise no to increase (E(A)) We define na as 7o ~ N(urr(Ea)) where Σn (B(A))-1 and r(E) is to reverse the signs of off-diagonal elements of the matrix E.
The second option is inject Th into the node loss during backpropagation, as follows.
7

---

Published in Transactions on Machine Learning Research (09/2022)
Table 1: Weight volume on VGG networks.
Method
VGG11
VGGII(
VGG16
VGGI8(drepout)
VGG19
VGG19(drapant)
CUPAR-AN CIPAROI
Sampling
0.06220002
0.13:0.02
0.05200022
0.12:0.02
0.0120002
0.12:10.2
Laplace
0.0568
0,1523
0.6475
0.1397
0.0453
0.1443
Generalization Gap
0,8030
0.5215
0.8698
0.2996
11087
0.1055
Sampling
0.07+0:02
0.1440.02
0.06+0002
0.14#0.02
0.04+0.02
0.12+0.02
Laplace
0.0337
0.1578
0.6409
0.1620
0.0506
0.1409
Goneralization Gap
3.1208
2.1635
3.7541
1.2714
4.4787
0.4373
Node Loss Reverse Noise Th to increase dct(E())))'. We define 1i as mh ~ r(A)), where Σ = (B
Algorithm presents our simple and effective way to obtain disentanglement noise, where A and X are hyper-parameters to balance the strengths of 7/a and 1)-- In the experiments in Section 5.3, we will assign expansion noises and contraction noises according to their impact on the weight volume, where expansion noises are obtained from Algorithm 2 and contraction noises are obtained from stochastic noises weights.
5 Experiments
We have conducted a number of experiments to test our hypotheses that (1) weight expansion helps reduce the generalization error and (2) dropout increases the weight volume. More than 900 networks are trained. First, confirm the role of weight expansion in connecting dropout and generalization by estimating the weight volume with two independent methods (Section 5.1). Second, we extend the PAC-Bayesian theorem with the awareness to weight, volume, and show that the new complexity measure can significantly improve the prediction of the generalization error for dropout networks (Section 5.2). Finally, we consider other noises that can lead to either weight expansion or weight contraction, and study their respective impact on the generalization performance (Section 5.3). All empirical results are both significant and persistent across the models and data sets work with, and support either hypothesis (1) or hypothesis (2) both. In our experiments, we consider VGG-like models (Simonyan & Zisserman, 2014) and Alexket-like models (Krizhevsky et al., 2012) for CIFAR-10/ 10/100 (Krizhevsky et al., 2009), ImageNet-32 (Deng et al., 2009), and SVHN (Netzer et al., 2011). For sampling method (and sharpness method), we default to the settings of c = 0.05 for CIFAR-10/SVE 10/SH,, and c 0.1 for CIFAR-100) 10e11000mmmmeo---0 about 1,500s to collect samples and estimate weight volume for each model, and Laplace Approximation takes about 300s for each model.
Sampling mothod takes
5.1 Measuring weight volume
First experiment studies the correlation between weight volume and the application of dropout during training. Weight volume is hard to measure exactly, so use two different, yet independent, estimation methods Laplace approximation and the sampling method as in Section 4-to improve our confidence in the results and robustly test our hypotheses.
Weight Volume
0.12
0.10
CIFAR-10 Sampling
0.08
Table presents the estimated weight volume and the generalization gap (loss gap) on several VGG networks on CIFAR-10/100. We see that dropout leads, across all models, to persistent and significant increase of the weight volume (hypothesis (2)). At the same time, the generalization error reduced (hypothesis (1)). We default to the setting of weight decay with in the experiments of Table 1, Figure 3, and also present similar results with weight decay 0.0005 (and more results on ImageNet-32) in Appendix J, text. classification experiments in Appendix M.1.
CIFA--10 Laplace
CIFAR-100 Sampling
OIAAR 100 Laplace
0.04
OFAR-10 Sampling
CIF1--00 Sempling
0.15
0.3
0.45
Drapout
Figure 3: Weight volume w.r.t. dropout rate.
8

---

Published in Transactions on Machine Learning Research (09/2022)
Table 2: Mutual Information of complexity measures on CIFAR-10.
Complexity
VGG
AlexNet
Measure
Dropout
(G([=/=2)
GG([w/ GG([]/=1)
Dropout
GG([==1) GG||/=2)
GG(L=/=0)
Frob Distance
0.0233
0.0243
0.0127
0.0033
0.0644
0.0289
0.0139
0.00194
Spectral Norm
0.0231
0.0271
0.0139
0.0001
0.1016
0.0837
0,0832
0.0786
Parameter Norm
0.0535
0.0252
0.0122
0.0008
0.0567
0.0814
0.0793
0.0744
Path Norı
0.0897
0.0129
0.0053
0.0027
0.1530
0.0425
0.0307
0.0177
Sharpness o'
0.1605
0.0127
0.0053
0.0001
0.1583
0.0877
0.0639
0.0418
PAC Sha-pharpesss
0.0797
0,0200
0.01099
0.0023
0.1277
0.0552
0.0324
D.II19
PAC-S(Laplace)
0.5697
0.0837
0.0623
0.0116
0.7325
0.1248
0.1123
0.1052
PAC-8(Sampling)
0.6606
0.0857
0.0623
0.0433
0.5917
0.1014
0.0921
0.0847
A After confirming the qualitative correlation between weight volume and dropout, we consider the quantitative effect of the dropout rate on the weight volume. As Figure 3 shows, for the 4 VGG16 models with dropout rate 0.0, 0.15, 0.3, 0.45, respectively, we see a clear trend on CIFAR-10/100 that the weight volume increases with the dropout rate (hypothesis (2)). The generalization gap is also monotonically reduced (CIFAR-10: 0.86, 0.51, 0.30, 0.16; CIFAR-100: 3.75, 2.14, 1.07, 0.39) (hypothesis (1))-
In sumunary, these experiments provide evidence to the correlation between dropout, weight expansion, and generalization. This raises two follow-up questions: (1) can the weight volume serve as a proxy for the generalization gap? and (2) can techniques other than dropout exhibit similar behavior? Sections 5.2, 5.3 answer these questions in the affirmative, shedding light on the nature of the correlation between these concepts.
Limitations. There is a problem when dropout rate is large (i.e., close to 1), as it will seriously reduce the training accuracy. Although in this case the generalization gap will be small and weight volume will be large, the performance of the model is so bad that the dropout does not actually improve the training. Thus, we only consider suitable dropout rate (10, 0.45)) in our experiments.
5.2 Performance of the new complexity measure
A complexity measure can predict generalization gap without resorting to a test set. Inspired by the experiments in Jiang et al. (2019), we have trained 288*3=864 dropout networks (with different rates: O, 0.15, 0.3) on CIFAR-10 for VGG-like and AlexNet-like structures and CIFAR-100 for VGG-like structures to compare new complexity mcasure-which incorporates the weight volumc-wite complexity measures, including PAC-Sharpness, which was shown in Jiang et al. (2019) to perform better overall than many others. Following Jiang et al. (2019), we use mutual information (MI) to measure the correlation between various complexity mcasures, and dropout and generalization gap (GG, defined as Li - L. The details of experimental settings (e.g., GG([w/=2)) and MI are given in Appendix K.1.
set of cxisting
We compare our new complexity measures (Equation (5), with Laplace approximation and sampling method rospectively) with some existing ones, including Frobenius (Frob) Distance, Parameter Norm (Dziugaite & Roy, 2017; Neyshabur et al., 2018b; Nagarajan & Kolter, 2019), Spectral Norm (Yoshida & Miyato, 2017), Path Norm (Neyshabur et al., 201 2015a;b), Sharpness a', and PAC-Sharpness (Keskar et al., 2016; Neyshabur et al., 2017; Pitas et al., 2017). Among them, PAC-Sharpness is suggested in Jiang et al. (2019) as the the most promising one in their experiments. The variance 0² of our methods (PAC-S(Laplace), PAC-S(Sampling) and PAC-Sharpness is estimated by sharpness method, i.e., let vec(U") ~ (N(, (22)) be a sample from zero mcan Gaussian, where a chosen to be the largest number such that max=<=ll(ffw+0 et al., 2019). Note that, the perturbed models are used to estimate weight volume (by using sampling method) and sharpness PAC-Bayes sigma. Then, the complexity measures with these estimates are further used to predict the generalization of unperturbed models.
(((w)(()))))))))) c (Jiang
Table 2 presents the MI as described above on CIFAR-10 for VGG-like models and AlexNet-like models.
Details of the above complexity measures are given in Appendix K.2. More empirical results on CIFAR- are given Appendix K.3. We can see that, for most cases concerning the generalization error, i.c., |w| 0, 1,2 (fix O, 1, 2 hyper-parameten respectively, and compute corresponding expected MI over sub-spaces, details are shown in Appendix K.1), our new measures, PAC-S(Tapplac)) and PAC-S(Sampling), achieve the highest.
9

---

Published in Transactions on Machine Learning Research (09/2022)
V0005 Inaggred B7tnc
VO0OO CIPA
V:000233
2.0
V:0.0102
V:000328
V:0.0221
V:0.0602
Vi V.000560
3.0
E1m
V.0.0774
V:0.1442
V:0.1041
V:0.0954
0.0978
V-0.1546
V:0.1063
V.0.778
VolaTi oipaniion
VORAT2 CorFidton
rcenal
Figure 4: Disentanglement noise dropout. We have trained 2 VGG16 networks (Left) and 2 AlexNet (Right) on ImageNet-32 and CIFAR-100 respectively, with normal training, dropout training, disentanglement noise training (volume expansion) and stochastic noise training (volume contraction). For each data set, we present their test losses in different plots. More experiments on the CIFAR-10, SVHN are given in Appendix L.
MI with GG among all complexity measures. On AlexNet, the MI of other complexity measures increase, but our new measures still perform the best. Generally, the MI with GG for our measures are significantly higher than those for others (which supports hypothesis (1)), because our complexity measures can predict generalization-ine immrovvmeemm from weight expansion, which is mainly caused by dropout in our network configurations. Mcanwhile, hypothesis (2) is also empirically verified as complexity measures are most closely correlated with dropout (sce MI between complexity measures and dropout). In summary, the above results demonstrate that weight. expansion can be regarded an indicator of the enhanced generalization capability (see MI with GG) endowed by dropout (see MI with dropout).
5.3 Disentanglement noise other than dropout
The experiments in Sections 5.1, 5.2 provide support to the hypotheses (1) and (2) that dropout, weight expansion, and gencralization correlated, with an indication that weight expansion is the proxy to the connection between dropout and the generalization error. Our final experiment is designed to test this observation by considering the replacement of dropout.
We consider the disentanglement noise as presented in Algorithm 2, and train a set of VGG16 networks and AlexNet on CIFAR-10, CIFAR-100, Imagenet-32 and SVHN, with normal training, dropout training, disentanglement noise training (volume expansion), and stochastic noise training (volume contraction), respectively. As stochastic noises on weights may lead to either weight volume contraction or leave it unchanged, we collect the contraction cases for our experiments.
Figure 4 shows that volume expansion improves generalization performance, similar to dropout. Likewise, volume contraction leads to worse generalization performance. This relation, which further supports hypothesis (1), is persistent across the examined networks and data sets. This suggests that the weight expansion, instead of dropout, can serve as key indicator of good generalization performance, while dropout provides one method to implement volume expansion. Disentanglement noises may be one of valid replacements for dropout.
6 Discussion
6.1 Related work about dropout
Optimizing the training process to improve generalization has been a hot research area, among which lot of work studies it with respect to dropout. For example, Wager et al. (2013) considers training procedure with adaptive dropout; Gal & Ghahramani (2016) interprets dropout with a spike-and-slab distribution by variational approximation; and Kingma et al. (2015) proposes a variational dropout, which is an extension of the Gaussian dropout which the dropout rates are Icarned during training. By taking into account continuous distributions and Bernoulli noise (i.e. dropout), Nalisnick al. (2019) provides a framework for comprehending multiplicative noise in neural networks. Instead of considering directly, and mostly
10

---

Published in Transactions on Machine Learning Research (09/2022)
qualitatively, how dropout improves the training as above, we understand dropout and generalization through a measurable quantity- - weight volume.
Recent research has started to provide a narrative understanding of dropout's performance (Zhang et al., 2021), and providing variational dropout (Wang & Manning, 2013; Gal & Ghahramani, 2016; Gal et al., 2017; Achille & Soatto, 2018; Fan et al., 2020; Lee et al., 2020; Pham & Le, 2021; Fan et al., 2021). For example, Gao et al. (2019) suggests that dropout can be separated into forward dropout (for the forward pass) and backward dropout (for the backward pass), and different dropout ratios might be needed to fully utilize the benefit of these two passes. Wei et al. (2020) empirically and theoretically explains the effectiveness of dropout on LSTM and transformer architectures from two entangled sources of regularization. It is unclear if this also holds for convolutional networks. Lengerich et al. (2020) considers input and activation dropout, suggesting that dropout reduces spurious high order interaction between input features. Different from these, we consider explaining dropout through the rigorously defined concept of weight expansion.
Providing a theoretical analysis to understand dropout from the perspective of regularizing the inductive bias, Cavazza et al. (2018) considers dropout for matrix factorization, and shows that dropout achieves the global minimum of convex approximation problem with (squared) nuclear norm regularization. This is extended to deep linear network (Mianjy et al., 2018), deep linear network with squared loss (Mianjy & Arora, 2019), and DNN with only one last dropout layer (Arora et al., 2019). Mianjy & Arora (2020) presents precise iteration complexity results for dropout training in two-layer ReLU networks, using the logistic loss when the data distribution is separable in the reproducing kernel Hilbert space. However, it is unclear how to extend this framework to large networks From another perspective, we present theoretical analysis that dropout increases weight volume to improve generalization.
6.2 Related work about other generalization factors
Other than dropout, there are works studying how other factors affect the generalization of neural net- works (Chatterji et al., 2020; Dziugaite et al., 2020a; Wu et al., 2020; Yang et al., 2020; Harutyunyan et al., 2020; Haghifam et al., 2020), and the relation between gradient noises and generalization (Keskar & Socher, 2017; Jastrzębski et al., 2017; Smith & Le, 2017; Xing el al., 2018; Chaudhari & Soatto, 2018; Jastrzebski et al., 2018; Li et al., 2019). For example, Wen et al. (2019) adds diagonal Fisher noise to large-batch gradient updates, making it match the behavior of small-batch SGD. Zhu ct al. (2019) investigates unbiasod noise in a generic lype of gradient based optimization dynamics that combines SGD and standard Langevin dynamics. Menon et al. (2020) demonstrates that in some circumstances, post-hoc processing can enhance the performance of overparameterized models on under-represented subgroups. Moreover, Jin al. (2020) studies the relation between generalization and a definition of correlation on weight matrices. There was also NeurIPS 2020 Competition on "Predicting Generalization in Deep Learning" (Jiang et al., 2020; Lassance et al., 2020; Natekar & Sharma, 2020). To study generalization, Huang et al. (2020) represents the Fisher information matrix of DNNs as block diagonal matrix, Huang et al. (2021) lets the filter of convolutional layer obey a block diagonal covariance matrix Gaussian. In this work, we introduce a novel factor weight volume based on random weights, which also has a close relationship with generalization.
6.3 Related work about PAC-Bayes
PAC-Bayes is a general framework to efficiently study generalization for numerous machine learning models. Since the invention of the PAC-Bayes (McAllester, 1999), there have been a number of canonical works developed in the past decades, e.g., the works about generalization error bounds for learning problems (Germain et al., 2009; Welling & Teh, 2011; Parrado-Henánnder et al., 2012; Dwork et al., 2015a;b; Blundell et al., 2015; Alquier et al., 2016; Thiemann et al., 2017; Letarte et al., 2019; Rivasplata et al., 2020; Pérez- Ortiz et al., 2021), especially the paranctrization of the PAC-Bayesian bound with Gaussian distribution (Seeger, 2002; Langford & Caruana, 2002; Langford & Shawe-Taylor, 2003; Dziugaite & Roy, 2018; Jin el al., 2022). Specifically, Langford& & Shawe- Shawe-Taylor (2003); Germain et al. (2009); Farrado-Hernánee demonstrate that one can obtain tight PAC-Bayesian generalization bounds through multiplying the weight vector norm with a constant this is different from the idea of expanding weight volume, as weight volume is a property of the weight covariance matrix, while weight vector norm is property directly over weights.
et al. (2012)
11

---

Published in Transactions on Machine Learning Research (09/2022)
Alquier et al. (2016) conducts a general study of the properties of fast Gibbs posterior approximations under PAC-Bayes. Letarte et al. (2019) uses the PAC-Bayesian theory to give complete analysis of multilayer neural networks with binary activation. McAllester (2013) defines a natural notion of an uncorrelated posterior and provides a PAC-Bayesian bound for dropout training, while we consider the correlated posterior. It is interesting that. Alquier al. (2016) also optimizes PAC-Bayesiad bound through a Gaussian posterior with a full covariance matrix. We use a similar assumption (correlated Gaussian posterior), but the main differences are that we develop under another PAC-Bayesian branch (Dziugaite & Roy, 2017; Neyshabur et al., 2017; 2018a) and apply it on DNNs with connection to dropout noise.
6.4 Weight expansion and flatness
The flatness of the loss curve thought to be related to the generalization performance of machine learning models (Hochreiter & Schmidhuber, 1994; 1997), especially DNNs (Dinh et al., 2017; Chaudhari et al., 2017 7; Yao et al., 2018; Liang et al., 2019). For example, in pioneering works (Hochreiter & Schmidhuber, 1994; 1997), the minimum description length (MDL) principle (Rissanen, 1978) was used to show that the flatness of the minimum is a good generalization measure to look at. Keskar et al. (2017) studies the reason for the gencralization drop in large-batch networks and shows that large-batch algorithms tend to converge to sharp minima, resulting in worse generalization. Jiang el al. (2019) conducts a large-scale empirical analysis and discovers that Ratness-based measures have a stronger correlation with generalization than weight norms measures. Foret et al. (2021) proposes Sharpness- Aware Minimization
and (margin and optimization)- (SAM) to find parameters that lie in neighborhoods with uniformly low loss, as they think gradient descent can be performed more efficiently with these parameters.
Of most relevance is the work by Dziugaite & Roy (2017); Neyshabur et al. (2017; 2018a), which relates a new concept of expected flatness over random weights to the PAC-Bayesian framework. By relaxing their i.i.d. assumption of random weights, our proposed concept of weight volume generalizes the expected flatness (that considers solely the varionce of random weights) with weight correlation taken into account (that considers the weight, covariance). Specifically, weight volume measures the expected flatness using the determinant of normalized covariance matrix of random weights. That is, a larger weight volume means random weights lie in a flatter local minimum (obeying more spherical distribution), which can lead to better generalization. In particular, assuming a quadratic approximation of the loss holds, say Laplace approximation, weight volume approximately represents the inverse of the product of all eigenvalues of the Hessian (without normalization). That is, vol(w) R II: ,. where A is the i-th eigenvalue of the Hessian matrix H. Not only does this confirm the connection between weight volume and sharp/flat minima (i.c., parg/ssmal cigenvalues) of the Hessian, but also provides another, perhaps more comprehensive, view of flatness with respect to the whole spectra of the Hessian.
7 Conclusion and future work
This paper introduces 'weight expansion', an intriguing phenomenon that appears whenever dropout is applied during training. From a perspective of covariance, weight volume can be seen a measure of flatness and weight expansion can be thought as random weights lying in more spherical (flatter) local minimum. We have provided both theoretical and empirical arguments, showing that weight expansion can be one of the key indicators of the reduced generalization error, and that dropout is effective because it is (an inexpensive) method to obtain such weight, expansion.
Future work. The following few directions may be worthy of future exploration and research. First, at the moment, we consider weights vector, without taking into consideration the structural information that the weights can be closely coupled according to the structures of the layers. It is interesting to know whether or not weight volume may capture some structural information of network (e.g., convolutional layers). Second, as activation functions have been argued to have significant impact on not only the trained models but also the training process, it is interesting to study how weight volume is related to activation functions (e.g., ReLU, tanh). Last but not least, could be a significant topic to study whether the weight volume- when combined with the PAC-Bayes and adversarial samples (Farnia et al., 2018) can provide any guidance to the adversarial robustness.
12

---

Published in Transactions on Machine Learning Research (09/2022)
Acknowledgement.
This project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement. No 956123, and from UK Dstl under project SOLITUDE. GJ and XH are also partially supported by the UK EPSRC under projects [EP/R026173/,, EP CPPTO020 TT02699229//// This work is also supported by the CAS Project for Young Scientists in Basic Research, Grant No. YSBR-040.
References
Alessandro Achille and Stefano Soatto. Information dropout:
Learning optimal representations through noisy computation. IEFE trassactions on pollern analysis and machine intelligence, (0(12):2897-2990 2018.
Pierre Alquier, James Ridgway, and Nicolas Chopin. On the properties of variational approximations of gibbs posteriors. The Journal of Machine Learning Research, 17(1):8374-8414 2016.
Sanjeev Arora, Naday Cohen, Wei Hu, and Yuping Tuo. Implicit regularization in deep matrix factorization. Advances in Neural Information Processing Systems, 32:7413 7424, 2019.
Pierre Baldi and Peter J Sadowski. Undcrstanding dropout. Advances in neural information processing systems, 26:2814 2822, 2013.
Nitin Bansal, Xiaohan Chen, and Zhangyang Wang. Can we gain more from orthogonality regularizations in training deep networks? Advances in Neural Information Processing Systems, 31:4261-4271, 2018.
Roberto Battiti. First-and second-order methods for learning: between steepest descent and newlon's 's method. Neural computation, 4(2):141-100) 1992.
Christopher M Bishop. Pattern recognition and machine learning. 2006.
Charles Blundell, Julien Cornebise, Koray Kavukeuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International Conference on Machine Learning, pp. 1613 1622. PMLR, 2015.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146, 2017.
Alcksandar Botev, Hippolyt Ritter, and David Barber. Practical ganss-newton optimisation for deep learning. In International Conference on Machine Learning, pp. 557-565. PMLR, 2017.
Jacopo Cavazza, Pietro Morerio, Benjamin Hacffele, Connor Lane, Vittorio Murino, and Rene Vidal. Dropout a low-rank regularizer for matrix factorization. In International Conference on Arlificial Intelligence and Statistics, pp. 435 444. PMLR, 2018.
Niladri Chatterji, Behnam Neyshabur, and Hanie Sedghi. The intriguing role of module criticality in the generalization of deep networks. International Conference on Learning Representations, 2020.
Pratik Chaudhari and Stefano Soatto. Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks. In 2018 Information Theory and Applications Workshop (ITA), pp. 1-10. IEEE, 2018
Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer T. Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-sgd: Biasing gradient descent into wide valleys. In International Conference on Learning Representations, 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In 2009 IEEE conference on compuler vision and pallern recognilion, pp. 248 255. leee, 2009.
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In Proceedings of the 94th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 1019-1028. PMLR, 2017.
13

---

Published in Transactions on Machine Learning Research (09/2022)
Cynthia Dwork, Vitaly Feldman, Moritz Hardt, Toni Pitassi, Omer Reingold, and Aaron Roth. Generalization in adaptive data analysis and holdout reuse. Advances in Newral Information Processing Systems, 28: 2350 2358, 2015a.
Cynthia Dwork, Vitaly Feldman, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Aaron Leon Roth. Preserving statistical validity in adaptive data analysis. In Proceedings of the forty-seventh annual ACM symposium on Theory of computing, pp. 117-126, 2015b.
Gintare Karolina Dziugaite and Daniel M Roy. Computing nonvacuous generalization bounds for deep (stochas- tic) neural networks with many more parameters than training data. arXiv preprint arXiv:1703. 11008, 2017.
Gintare Karolina Dziugaite and Daniel M Roy. Data-dependent pac- pac-bayes bayes priors via differential privacy. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 8440-8450, 2018.
Gintare Karolina Dziugaite, Alexandre Drouin, Brady Neal, Nitarshan Rajkumar, Ethan Caballero, Linbo Wang, Joannis Mitliagkas, and Daniel M Roy. In search of robust measures of generalization. Advances in Neural Information Processing Systems, 33, 2020a.
Gintare Karolina Dziugaite, Alexandre Drouin, Brady Neal, Nitarshan Rajkumar, Ethan Caballero, Linbo Wang, Toannis Mitliagkas, and Daniel M Roy. In search of robust measures of generalization. Advances in Neural Information Processing Systems, 33, 2020b.
Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth on demand with structured dropout. International Conference on Leurning Representations, 2020.
Xinjie Fan, Shujian Zhang, Korawat Tanwisuth, Xiaoning Qian, and Mingyuan Zhou. Contextual dropout: An efficient sample-dependenl dropout module. International Conference on Learning Representations, 2021.
Farzan Farnia, Jesse M Zhang, and David Tse. Generalizable adversarial training via spectral normalization. International Conference on Learning Representations, 2018.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. In International Conference Learning Representations, 2021.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050 1059. PMLR, 2016.
Yarin Gal, Jiri Hron, and Alex Kendall. Concrete dropout. Advances in Neural Information Processing Systems, 30, 2017.
Hongchang Gao, Jian Pei, and Heng Huang. Demystifying dropout. In International Conference on Machine Learning, pp. 2112 2121. PMLR, 2019.
Pascal Germain, Alexandre Lacasse, François Laviolette, and Mario Marchand. Pac-bayesian learning of linear classifiers. In Proceedings of the 26th Annual International Conference on Machine Learning, pp. 353-300 003-360,22 2009.
Mahdi Haghifam, Jeffrey Negrea, Ashish Khisti, Daniel M Roy, and Gintare Karolina Dziugaite. Sharpened generalization bounds based on conditional mutual information and an application to noisy, iterative algorithms. Advances in Neural Information Processing Systems, 33:9925-9935, 2020.
Hrayr Harutyunyan, Kyle Reing, Greg Ver Steeg, and Aram Galstyan. Improving generalization by controlling label-noisc information in neural network weights. In International Conference on Machine Learning, pp. 4071-4081. PMLR, 2020.
David P Helmbold and Philip M Long. On the inductive bias of dropout. The Journal of Machine Learning Research, 16(1):3403- 3454, 2015.
14

---

Published in Transactions on Machine Learning Research (09/2022)
David P Helmbold and Philip M Long. Surprising properties of dropout in deep networks. In Conference Learning Theory, pp. 1143- PMLR, 2017.
Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Llya Sutskever, and Ruslan Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint orXi:: 1tt::227.055 2012.
Sepp Hochreiter and Jürgen Schmidhuber. Simplifying neural nets by discovering flat minima. In Gerald Tesauro, David S. Touretzky, and Todd K. Leen (eds.), Advances in Newral Information Processing Systems, pp. 529-536. MIT Press, 1994.
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Newral Comput., 9(1):1-42, 1997.
Lei Huang, Jie Qin, Yi Zhou, Fan Zhu, Li Liu, and Ling Shao. Normalization techniques in training dnns: Methodology, analysis and application. arXiv preprint ar arXit:2009. 0 12836, 2020.
Zhongzhan Huang, Wenqi Shao, Xinjiang Wang, Liang Lin, and Ping Luo. Rethinking the pruning criteria for convolutional ncural network. Advances in Neural Information Processing Systems, 34:16305-16318, 2021.
Stanislaw Jastrzebski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors inftuencing minima in sgd. arXiv preprint orXiv:1711.044: 2017.
Stanistaw Jastrzębski, Zachary Kenton, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. On the relation between the sharpest directions of dnn loss and the sgd step length. ar preprint wrXiv:187.050311 2018.
Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic gencralization measures and where to find them. arXiv preprint arXiv:1912.021 19111.172.. 2019.
Yiding Jiang, Pierre Foret, Scott Yak, M. Daniel Roy, Hossein Mobahi, Karolina Gintare Dziugaite, Samy Bengio, Suriya Gunasekar, Isabelle Guyon, and Behnam Neyshabur. Neurips 2020 competition: Predicting generalization in deep learning. 2020.
Gaojie Jin, Xinping Yi, Liang Zhang, Lijun Zhang, Sven Schewe, and Xiaowei Huang. How does weight correlation affect generalisation ability of deep neural networks? Advances in Newral Information Processing Systems, 33, 2020.
Gaojie Jin, Xinping Yi, Wei Huang, Sven Schewe, and Xiaowei Huang. Enhancing adversarial training with socond-order statistics of weights. In Proceedings of the IFFE/CC /VFF Conference on Computer Vision and Pottern Recognition, pp. 15273 15283, 2022.
Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. Fasttext.zip: Compressing text classification models. arXiv preprint arXi::1612.0300 2016.
Armand Joulin, Édouard Grave, Piotr Bojanowski, and Tomá Mikolov. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pp. 427-431, 2017.
Nitish Shirish Keskar and Richard Socher. Improving generalization performance by switching from adam to sgd. arXiv preprint Xr 1711:111220 076228, 2017.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocodal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint 1TF::660.0448 2016.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Representations, 2017.
Yoon Kim. Convolutional neural networks for sentence classification. In EMNLP, 2014.
15

---

Published in Transactions on Machine Learning Research (09/2022)
Durk P Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameterizatioo trick. Advances in neural information processing systems, 28:2575-2588 2015.
Kocherlakota and K Kocherlakota. Generalized variance. Encyclopedia of Stalistical Sciences, 2004.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Πya Sutskever, and GeofTrey Hinton. Imagenet classification with deep convolutional neural networks. Advances in newral information processing systems, 25:1097-1105, 2012.
John Langford and Rich Caruana. (not) bounding the true error. Advances in Neural Information Processing Systems, 2:809-816, 2002.
John Langford and John Shawe- Taylor. Pac-bayes & margins. Advances in neural information processing systems, pp. 439-446, 2003.
Carlos Lassance, Louis Béthune, Myriam Bontonou, Mounia Hamidouche, and Vincent Gripon. Ranking deep learning generalization using label variation in latent geometry graphs. arXiv preprint arXiw:2011.19737, 2020.
Beom Hae Lee, Taewook Nam, Eunho Yang, and Ju Sung Hwang. Meta dropout: Learning to perturb latent features for gencralization. International Conference on Learning Representations, 2020.
Benjamin Lengerich, Eric P Xing, and Rich Caruana. On dropout, overfitting, and interaction effects in deep neural networks. arXiv preprint orXiv:2007.0082) 2020.
Gaël Letarte, Pascal Germain, Benjamin Guedj, and Francois Laviolette. Dichotomize and generalize: Pac-bayesian binary activated deep neural networks. Advances in Neural Information Processing Systems, 32:6872-6888 6882, 2019.
Yuanzhi Li, Colin Wei, and Tengyu Ma. Towards explaining the regularization effect of initial large learning rate in training neural networks. Advances in Neural Information Processing Systems, 32, 2019.
Tengyuan Liang, Tomaso A. Poggio, Alexander Rakhlin, and James Stokes. Fisher-rao metric, geometry, and complexity of neural networks. In The 22nd International Conference on Arlificial Intelligence and Slatistics, ATSTATS 2019, 16-18 April 2019, Naha, Okinawa, Japan, volume 89 of Proceedings of Machine Leurning Research, pp. 888 896. PMLR, 2019.
Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. Recurrent neural network for text classification with multi-task learning, Xiv preprint arXiv: arXiv:160..00
2016.
James Martens and Roger Grosse. Optimizing neural networks with wooecker-factoree approximate curvature. In International conference on machine learning, pp.
PMLR, 2015.
David McAllester. A pac-bayesian lutorial with a dropout bound. ar preprint ar arXiv:1307.2116 2013.
David A McAllester. Pac-bayesian model averaging. In Proceedings of the twelfth annual conference on Computational learning theory, pp. 164-170, 1999.
Adilya Krishna Menon, Ankit Singh Rawat, and Sanjiv Kumar. Overparameterristioo and worst-case generalisation: friend or foo? In International Conference on Learning Representations, 2020.
Poorya Mianjy and Raman Arora. On dropout and nuclear norm regularization. In International Conference on Machine Learning, pp. 4575 4584. PMLR, 2019.
Poorya Mianjy and Raman Arora. On convergence and genoralization of dropout training. Advances in Neural Information Processing Systems, 33, 2020.
Poorya Mianjy, Raman Arora, and Rene Vidal. On the implicit bias of dropout. In International Conference Machine Learning, pp. 3540-20 3548. PMLR, 2018.
16

---

Published in Transactions on Machine Learning Research (09/2022)
Dmytro Mishkin and Jiri Matas. All you need is good init. arXiv preprint arXiv:1511.0644 2015.
Vaishnavh Nagarajan and J Zico Kolter. Generalization in deep networks: The role of distance from initialization. arXiv preprint 1XXi:190110000 2019.
Fric Nalisnick, José Miguel HernándezzLobato and Padhraic Smyth. Dropout as a structured shrinkage prior. In International Conference on Machine Learning, pp. 4712 4722. PMLR, 2019.
Parth Natekar and Manik Sharma. Representation based complexity measures for predicting generalization in deep learning. arXiv preprint arXiv:2012.02775, 2020.
Jeffrey Negrea, Mahdi Haghifam, Gintare Karolina Dziugaite, Ashish Khisti, and Daniel M Roy. Information- theoretic generalization bounds for sgld via data-dependent estimates. Advances in Neural Information Processing Systems, 32:11015 11025, 2019.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Behnam Neyshabur, Ruslan Salakhutdinov, and Nathan Srebro. Path-sgd: path-normalized optimization deep neural networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems- Volume 2, pp. 2422 2430, 2015a.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm- capacity control in neural networks In Conference Learning Theory, pp. 1376 1401. PMLR, 2015b.
Bchnam Neyshabur, Srinadh Bhojanapalli, David Mcallester, and Nati Srebro. Exploring generalization in deep learning. Advances in Neural Information Processing Systems, 30:5947-5956, 2017.
Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro. A pac-bayesian approach to spectrally- normalized margin bounds for neural networks. International Conference Learning Representations, 2018a.
Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. Towards understand- ing the role of over- wve-pparamtttiaatoon in generalization of neural networks. arXiv preprint arXiv: 1tt::1880.12000 12076, 2018b.
Emilio Parrado-Hernánder Amiran Ambroladze, John Shawe-Taylor, and Shiliang Sun. Pac-bayes bounds with data dependent priors. The Journal of Machine Leurning Research, 13(1):3507 3531, 2012.
Maria Pérez-Ortiz, Omar Rivasplata, John Taylo,,aylor, and Csaba Szepesvári. Tighter risk certificates for neural networks. Journal of Machine Learning Research, 22, 2021.
Hieu Pham and Quoe Le. Autodropout: Learning dropout patterns to regularize deep networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 9351-9359, 2021.
Konstantinos Pitas, Mike Davies, and Pierre Vandergheynst. Pac-bayesian margin bounds for convolutional neural networks. arXiv preprint arXiv:1801.0011/ 2017.
Jorma Rissanen. Modeling by shortest data description. Aulom., 14(5):465 471, 1978.
Hippolyt Ritter, Aleksandar Botev, and David Barber. A scalable laplace approximation for neural networks. Tn 6th International Conference on Learning Representations, TCLR 2018-Conference Track Proceedings, volume 6, International Conference on Representation Learning, 2018.
Omar Rivasplata, Ilja Kuzborskij, Csaba Szepesvári, and John Shawe-Taylor. Pac-bayes analysis beyond the usual bounds. In NeurIPS, 2020.
Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. Xiv preprint arXio:1312. 6120, 2013.
17

---

Published in Transactions on Machine Learning Research (09/2022)
Matthias Seeger. Pac-bayesian generalisation error bounds for gaussian process classification. Journal of machine learning research, 3(Oct):233-P))) 269, 2002.
Adrian J Shepherd. Recond-order methods for neural networks: Fast and reliable training methods for multi-layer perceptrons. Springer Science Business Media, 2012.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for largo-scale image recognition.
arXiv preprint 4rX:::1409.15 2014.
Samuel L Smith and Quoc Lc. A bayesian perspective on generalization and stochastic gradient descent. arXiv preprint 1rX00:1700.0000
2017.
Nitish Srivastava, GeofTrey Hinton, Alex Krizhevsky, Nya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting, The journal of machine learning research, 15(1): 1929-1958, 2014.
Maosong Sun, Jingyang Li, Zhipeng Guo, Z Yu, Y Zheng, X Si, and Z Liu. Thuctc: an efficient chineso text. classifier. GilHuu Repository, 2016.
Niklas Thiemann, Christian Igel, Olivier Wintenberger, and Yevgeny Seldin. A strongly quasiconvex pac- bayesian bound. In International Conference on Algorithmic Learning Theory, pp. 466-492. PMT.R, 2017.
Stefan Wager, Sida Wang, and Percy Liang. Dropout training as adaptive regularization. Advonces in neural information processing systems, 26:351-359, 2013.
Sida Wang and Christopher Manning. Fast dropout training. In international conference machine learning, pp. 118-126. PMT.R, 2013.
Colin Wei, Sham Kakade, and Tengyu Ma. The implicit and explicit regularization effects of dropout. In International Conference on Machine Learning, pp. 10181-10192. PMLR, 2020.
Max Welling and Yoe W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proccedings of the 28th international conference machine learning (ICML-11), pp. 681 688. Citeseer, 2011.
Yoming Wen, Kevin Luk, Maxime Gazcau, Guodong Zhang, Harris Chan, and Jimmy Ba. Interplay between optimization and generalization of stochastic gradient descent with covariance noise. arXiv preprint arXiv:1902 102.:1902.08244 2019.
Sen Wu, Hongyang Zhang, Gregory Valiant, and Christopher Ré. On the generalization effects of linear transformations in data augmentation. In International Conference on Machine Learning, pp. 10410-10420. PMLR, 2020.
Chen Xing, Devansh Arpit, Christos Tsirigotis, and Yoshua Bengio. walk with sgd. Xixiv preprinl arXiv:1802.08771 2018.
Aolin Xu and Maxim Raginsky. Information-theoretic analysis of generalization capability of learning algorithms. Advances in Neural Information Processing Systems, 2017:2525-2534 2017.
Zitong Yang, Yaodong Yu, Chong You, Jacob Steinhardt, and Yi Ma. Rethinking variance trade-off for generalization of neural networks. In International Conference on Machine Learning, pp. 10767-10777. PMLR, 2020.
Zhewei Yao, Amir Gholami, Qi Lei, Kurt Keutzer, and Michael W. Mahoney. Hessian-based analysis of large batch training and robustness to adversaries. In Advances in Neural Information Processing Systems, pp. 4954-4964, 2018.
Yuichi Yoshida and Takeru Miyato. Spectral norm regularization for improving the generalizability of deep learning. Xixio preprint arXi:: 00441, 2017.
18

---

Published in Transactions on Machine Learning Research (09/2022)
Hao Zhang, Sen Li, YinChao Ma, Mingjie Li, Yichen Xie, and Quanshi Zhang. Towards understanding and improving dropout in game theory. International Conference on Learning Representations, 2021.
Zhanxing Zhu, Jingfeng Wu, Bing Yu, Lei Wu, and Jinwen Ma. The anisotropic noise in stochastic gradient descent: Its behavior of escaping from sharp minima and regularization effects. Tn International Conference Machine Learning, pp. 7654-7663. PMLR, 2019.
19

---

Published in Transactions on Machine Learning Research (09/2022)
0.25
Normal Dropoul
1
Weight Volume
100
of weight Epoch
(a) Partal normalized covariance matri (normal)
(c) normalized covariance matrix (dropout)
Figure We have trained two VGG16 networks (with/withoul dropout) on CIFAR-10. (b) shows the changes in weight volune along with epochs. There are four sampling points (0230) to track the absolute normalized covariance matrix of normal gradient updates (a) and dropout gradient updates (c).
Table 3: Network Structure for Figure 1.
Normal CNN
Conv (3,3)
Conv (3.3)
MaxPool (7.7)
Dense 128
Dense-2 2
Softmax
Dropout CNN
Conv-61-(3,3)
Conv-128-(3.3)
MaxPool-(7.7)
Dense-128
Dropout-0.3
Dense-2
Softmax
A Details of Figure 1
As Table 3 shows, use the above two networks to train MNIST with ReLU activation function, 0.01 learning rate, 128 minibatch size, stochastic gradient descent (SGD), 150 epochs, with and without dropout. Normal CNN and dropout CNN contain multiple convolutional layers with 64 and 128 channels, following with 128 and 2 hidden units. We visualize the features of Dense-2 layer (layer with units) in Figure 1.
B Weight volume and weight orthogonality
For clarification, weight volume is not the orthogonality of weight matrices, but the "normalized generalized variance" of weight matrices. It is statistical property of weight matrices when treated as random variables, which is different from the orthogonality technique in Saxe et al. (2013); Mishkin & Matas (2015); Bansal et al. (2018) that treat weight matrices as deterministic variables.
Specifically, by treating the elements in weight matrices as random variables (to fit the PAC-Bayes framework), the weight volune is defined as the determinant of the "covariance matrix" of weight matrices (see Defini- tion 3.1). The optimization of the determinant of the covariance will not necessarily lead to orthogonality of weight matrices. It appears that weight volume considers the statistical correlation between any two weights (treated as random variables), whereas weight orthogonality considers the linear space correlation between any two columns in weight matrices.
E.g., multivariate random variable w made up of two terms, say w = vec(W) + Uw, where veo(W) the vectorization of weight matrix and Uw the 0 mean multivariate random variable. Weight volume relies on the covariance matrix of Uw, while weight orthogonality depends W.
C Supplementary for Figure 2
Figure also shows that the correlation among gradient updates in the normal network (Figure 5a) is larger than the correlation in dropout network (Figure 5c) and large correlation of gradient updates can lead small weight volume (large correlation of weights), while small correlation of gradient updates can lead large weight volume (small correlation of weights).
20

---

Published in Transactions on Machine Learning Research (09/2022)
D Detailed derivation of PAC-Bayes
McAllester (1999); Dziugaite & Roy (2017) consider training set S with m E N samples drawn from a distribution D. Given the prior and posterior distributions P and Q on the weights w respectively, for any > O, with probability over the draw of training set, we have
DKL(Q]IP) + In m
E Bw-qlLD((ww) S [wwolLs([w]] +
(12)
2(m 1)
Equation 4: Let P (up,, Ep) and Q= - EQ); we can get
DKI(N(PQ) E2)1 Ep)) / - (m((
= / 1 det Ep
1
1/ In del ΣQ
7
( 1
2" (o - m2) + = -
2
det Ep
= In det IQ
tr(l[(x - p2) " + tr tr< "P (x - pP)l
(13)
2
2
2
In del Ep
1
1
(r((T)) + マ/ wnq / (nQ pp) + 1 -ir(Eip Ep)
det IQ
2
(rppp Eq) + (nQ pp) (nQ pP) k: In (Ep)
det (Eq)
Equation 5: Neyshabur et al. (2017); Jiang et al. (2019) instantiate the prior P to be a vec(Wo) mean, o² variance Gaussian distribution and the posterior Q to be a veo(WF) mean spherical Gaussian with variance o² in each direction. Relax their assumption, we assume Q non-spherical Gaussian (the off-diagonal correlations for same layer not 0, those for different layers are 0), then we can get
|| 10
(2'(
DKL(Q|IP)
In
20
2
del (Eq)
W.F
W
det (EI,p)
+
In
20
2
det (2)
W. W0 σ²
det
In
2
dot dct(E1,
(14)
W. w1112
I1. P1, det (Et.))
2
o2
1
W.F.F
-W9112
In II [i,Qla det (
Σ
2
1
Σ
IIF W
In vol(wi)
a²
21

---

Published in Transactions on Machine Learning Research (09/2022)
E Proof of Lemma 3.2
Proof E.1 Let Asg, Dey be the gradient updates for Waj, Wey of some layer in a normal network, i 8', As and Dey be the gradient updates for Waj; Wyz in the same setting dropout network. Let vec(A) ~ Ma Es) and vec(A)= = (1 @0 r)) @ vee( (') vec( Ma(Us, Ea), Note that, as we aim to study the impact of ñ on the correlation among gradient updates, we assume vec( and vec( are i.i.d. As dropout noises for different nodes (units) are independent, we have
Cov(A.j: Avy.) = [x - (3,,)( 13 )1
= (1 [( 1 - -Ma ) A/y3
[Ha,,.]] + +(1 g)9
-99
E3 1( 1-q Mavy) (-HAW)) +(1 g)glE 3/:((
1-9 اا,;) (-
(15)
+ [ (-Has)))
= -E Ma.,) (á'.z - -b
=Ea((Ar - -PA,) (Bpg - -Har,))
= Cov(Aig. A,jr),
and
Var( (yy) = E& (Aij 13.,) )²}
(1 --)1531 / 11 A
mA,,) )2 q(-m,,,,)
Ex, 1(A// mai,) )
q12
(16)
1-
1 q
EA (Ai, MA. )2
9112,,
1-
(i)
+
1 7
303 1-9
Similarly, Var(Avy)= = Var(A,y))
Thus, we have
1-g
Cov(A Avy))
|3
Varl VVr(Aur) (A (r(()r)))
[Cov(Aij, Apr)
Var(Anr + I-4
Var(A
qu3 2 1-9
1-4
1-
(17)
(1 MolCov(AAp) Awye)
Var A.j Var( Ary))
(1 4) PAisiAgy!
|PAn.Ayyl
22

---

Published in Transactions on Machine Learning Research (09/2022)
F Proof of Lemma 3.3
Proof F.1 As w ~ Nw we have
Cov(wij, Avg') E., [Ww, (wwy) (ww) (Argr - Mārry1
(1 g) [(wes mww,) M Day
1-9 -HaNs)) + 9Bw [Wess Mwi,) -
(18)
w [ pwwss) (4 |1
Ew,A (Wij Hwwes) ) (Avr - p
Cov(wijj Avj'),
Cov (Wvj, Cor(Wyy Avg') and Cov(wey's Ais) are similar. Prom Equations 15, 16, we can get
Pw,, (wn ICov(w., + Aij, Way + Avy)]
Var(wiy + Ais) Avj')
[Cov(wyy, way Cov Aij Avy.) Cov(way, Ary) Cov(wey,, Ab))
V (ar((w))) + Var (b))) 20ov Wij, A k))) (Var(wey)) Var( 2Cov(weg) Ary))
|Cov(wyy W Cov( Cov( A Cor(w,,,
(Varr(ws)) + Var (y) 2Cov w A (1)) (Var(W:)))) Var Var(
+ 2Cor(W Ary))
[Co(Wayj wayr + Cov A.j, ) + Cov (oo(w.j Avy,) + Cov(wey, An)
w²..
(Var(wis) + Var(A.)) I-g
+
2Cov(wij, Ai))
1-4
1
(Var(wyy)) + Var(A 1-q [Cov(wij + Aij. way + Arg:)
+
9/2ggg 1-9
+ +2Cov(Wey, Avj'))
(aarwwy + Aij) + (Var(An) 1-9
(Var(wsgg + Anj) + 9 (Var(Ar,))
1-2
(oo((w. + Aij. Wars + Ary)
Var(Ws; + Aij) Var Wey + Azjy)
Aey Anyl
G Weight volume and correlation
Since the definition of weight volume in Definition 3.1, the weight volume equals the determinant of the weight correlation matrix.
It difficult thcoretically argue an 'exact' relationship (e.g., perfect positive or negative) between weight volume and correlation of weights, but they have rough negative relationship. Take a simple example, let. n-dimensional weight correlation matrix C have the same correlation p, it is easy to infer that (absolute) correlation is negative related with weight volume,
det(C) = (1-p)" (1-p)=-1(14 (n- 1)p),
(19)
where ddet((C) (n (n)))) p))--2 and > 2. Also, demonstrate the relationship between weight volume dp and average (absolute) correlation with 10000 samples for 3-dimensional corrclation matrix (Figure 6a) and 4-dimensional correlation matrix (Figure 6b) respectively. Although weight volume is not monotonically increasing (absolute) correlation decreases, we can find rough negative relationship.
23

---

Published in Transactions on Machine Learning Research (09/2022)
Rommal
Dropoue
08-
Weight Decay
Batch Norma loation
0.8
0.-
2.
Deter
0.-
0.-
3,0
6.4
0.0
0.4
Absakute Comdation
Absoltte Comelation (b) Corvlaation wett detaminent (4-d corralation matris)
(0) Conciation w.rt delaminant (3-3 consiation matric)
Weigght volume wirl epoch (diffrnen (diaaren rogularzzrr)
Figure 6: (a) We sample 10000 3-dimensional correlation matrices and show their determinant and average absolute correlation. (b) We sample 10000 4-dimensional correlation matrices and show their determinant and average absolute correlation. (c) We train four small NNs (64-32-16-1) 10 with normal, dropout, weight decay, and batch normalization, respectively) on MNIST. The figure shows the changes in weight volume along with epochs for these lour networks.
H Other regularizers with weight volume
As dropout can gencrate independent noises but other regularizers, like weight docay and batch normalization, camnot. Thus, dropout can introduce randomness into the network, while weight decay and batch normalization cannot (these two regularizers improve generalization not through weight volume, through other keys). We have described in detail why this dropout noise can expand weight volume and why weight volume is correlated with generalization, theoretically (Section 3) and empirically (Section 5, Figures 2, 5).
Also, in Figure 6c, we can see that dropout expands weight volume compared with normal network, while weight decay and batch normalization have hardly any impact on weight volume.
Estimate weight volume
The approximation of H,[H] is two-fold: (1) Only the diagonal blocks of the weight matrix are captured, so that the correlation between blocks is decoupled; (2) Assume A and H4 are independent (Botev al., 2017; Ritter et al., 2018). Then, we have
H = [A @ Hi] Ro [A @ [.).
(20)
Finally, Σ, can be approximated as follows:
(A-1)) ",, (Pt
== (B, (E,
(21)
Therefore, by separately computing the matrix inverse of [A[A[-]] and Ez(H,,, whose sizes are significantly smaller, we can estimate Σ, E M efficiently.
Further, the weight volume can be approximated by using
det(F.)) R det ((E (E)(AA-)))-1) No det ((E, (Hi)() 11) Ni
(22)
and
II [e) R [E. [E.(A- M [E, (alli)--(aaa N Ni
(23)
J Supplementary for Section 5.1
As Appendix 4 shows, on the ImageNet-32, dropout still leads persistent, and significant, increase on the weight volume, across all three models. At the same time, the generalization error is reduced.
24

---

Published in Transactions on Machine Learning Research (09/2022)
Table Weight Volume on VGG Networks (ImageNett322 weight decay 0).
Method
VGG11
VGG16
VGGI8(dropont)
VGG19
VGGI9(dropout)
Sampling
0.015+0.01
0.089
0.016-0.01
0.097
0.017+0.01
0.086 0.01
Laplace
0.0198
0.0985
0.0194
0.0942
0.0214
0.0977
Generalization Cap
7.0388
2.0501
6.3022
1.2430
7.5187
0.7910
Table 5: VGG11 Structure
Come-64
Maxpooling
Comy-128
Maxpooling
Come-256
D0ogont
Comy-256
Maxgooling Demso-512
Cony-512
(ropont(0.3)
Come-512
Maxpooling
Conv-512
Doopost(0.3)
Conv-512
Maxpooling
Dropont(0.1)
Dropout/0.1)
Denae-quun classes)
Table 6: VGG16 Structure
Conv-64
Drepont
Comy-64
Maxpooling
Comy-128
D
Conv-128
Maxpooling
Comv-256
Droporr/O.5)
Cons-206
Dropont (0.3)
Conv-20i
Mixpooling
Conv-112
(..
Canv-512
Dropout(O.3)
Comv-512
Maxpooling
Conv-512
(0.3)
Conv-512
Dropout(0.3)
Comv-512
Maxpooling
Dropoutt(l.]
Dense-512
(1op
Jease-(aum clusses
Table 7: VGG19 Structure
Conv-64
Dropont
Cone-64
Maxpooling
Cons-128
(0.3)at(0.3)
Conv-128
Moopooling
Come-25i
Drogout((.3)
Conv-2bli
Dropont (0.3)
Conx-256
(10
256
Maspooling
Cony-512
Dropont(0.3)
S12
Deopous
Dropont (0.3)
Coco 512
Maxppooiigg
(103
(1.poat(.3)
Cony
(1.3
Conv-512
Masgooling
Drepom/(o1)
Derse-512 512
(0.1)
Donse- Dense-(mmn (mui classss)
Table 8:
Weight Volume VGG Networks (TmageNet-32, weight decay 0)-
Method
VGG11
VCGII(rropout)
VGG16
VGG16(dropout)
Sampling
0.015+0.01
0.089 0.01
0.016+0.01
0,097
Laplace
0.0198
0.0985
0.0194
0.0942
Lass
0.01233
1.157I
0.0085
L8577
Test Loss
7,4811
3.2072
6.4017
2,9007
Loss Gap
7.0388
2.0501
6,3922
1.2430
Top-1
030033
0.7088
0.9066
0.0049
Top-1 Test Ace
0,2738
0.3694
0.3116
0.3908
Top-1 Acc Gap
0.7215
0.3394
0.6837
0.2041
Top-5 Train
0.90000
0.8023
0.9080
0.8248
Top-5 Test Ace
0.5014
0.6170
0.5502
0.6444
Top-5 Acc Gap
0.4985
0.1853
0.4197
0.1804
Table 9: Weight Volume on VGG Networks (ImageNet-322, weight decay 0.0005).
Method
VGG11
(dGII(rropout)
VGG16
VCG1e(dropout)
Sampling
0.021
0007
0.019+0.DD1
0.091+0.001
Laplace
0.0268
0.0877
0.0284
0.0891
Train Loss
0.7321
2.3611
0.6708
2.6208
Test Loss
4.8745
3.2244
4.6388
3.0396
Losss Gup
4.1410
0.8603
3,9680
0.4188
Top-1 Train Ace
0.9926
0.5312
0.9532
0.4819
Top-1 Test Acc
0.3265
0.4033
0.3732
0.4286
Top-1 Acc Gap
0.6661
0.1279
0.6200
0.0583
Top-5 Train
0.9020
0.7539
0.3230
0.7419
Top-5 Test Ace
0.5691
0.6531
0.6145
0.6777
Top-5 Acc Gap
(14330
0.1308
0.3864
0.0642
Table 10: Weight volume on VGG networks (weight decay 0).
Method
VGG11
VGGLI(dropout)
VGG16
VGG16(dropout)
Sampling
0.06
0.1310.02
0.0520.02
0.1210002
Laplace
0.0568
0.1523
0.0475
0,1397
CIPAR-10
Train Loss
0,0833
0.0866
0.0200
0,3001
Loss
0.6021
0.8001
0.5098
Loss Gap
0.5030
0.5215
0.8898
0.2996
Train Acc
0.9973
0.9698
0.9933
0.8813
Test Ace
0.8563
0.8645
0.8435
0.8101
Gap
0.1411
0.1053
0.1198
Sampling
0012
0.1420.02
0.0620022
0.141:0.02
Laplace
0.0637
0.1578
0.0409
0.1620
CIEAR-1O0 100
Train Loss
0.0187
0.2688
0.0706
1,0321
Test. Loss
3.1395
2.4244
3.8247
2.3038
Loss Gap
3.1208
2.1635
3.7541
1.2714
Train Ace
0.9913
0.9271
0.9912
0.6398
0.5764
0.3911
Acc Gap
0.1179
0.3363
0.5015
0.1210
In addition, we show the VGG network structures in Tables 5, 6, 7, and also demonstrate the details (including training loss, training accuracy, test loss, test accuracy) of the VGG experiments on ImageNet-32 (Tables 8, 9) and CIFAR-10/ 10/100 (Tables 10, 11). All models for CIFAR-10/ 1100 are trained for 140 epochs using SGD with momentum 0.9, batch size 128, weight decay 0 or 10-1 and an initial learning rate of 0.1 that is
25

---

Published in Transactions on Machine Learning Research (09/2022)
Table 11: Weight volume on VGG networks (weight decay = 0.0005).
Method
VGG11
VGGLI(dropout)
VGG16
VGG16(dropout)
Sampling
0.05=0.02
0.1320.02
0.05=0.02
0.13:10.02
Laplace
0.0512
0.1427
0.0433
0.1346
CIPAR-10
Train Loss
0.2166
0.2984
0.1739
0.2438
Test Loss
0.6368
0.3997
0.5316
0.4788
Gap
0.3013
0.3607
Train Acc
0.9910
0.9801
0.3958
0,9878
Test Ace
0.9048
0.91
0.9234
0.9340
Gap
0.0724
Sampling
005-0.02
0.15++0.02
0.06+00.02
0.15-1-0.02
Laplace
0.0496
0.1469
0.0413
0.1659
CIPAR-100
Train Loss
0.4359
0.7612
0.5403
0.8890
Loss
2.
2.3672
2.31499
Loss Gap
1 19977
1.6060
1.7746
1,2626
Train Ace
0.9976
0.9651
0.9888
0.9133
0.6765
0.0080
0.6070
Acc Gap
0.3465
0.2889
0.2909
0.2163
divided by after every 20 epochs. All models for ImageNet-32 are trained for 100 epochs using SGD with
momentum 0.9, batch size 1024, weight decay 0 or 5 x 10-4, and an initial learning rate of 0.1 that is divided by 2 for each 20 epochs, and learning rate of 0.0005 after 50th epoch.
All results Tables 8, 9, 10, 11 also suggest that dropout can expand the weight volume and narrow generalization gap.
K Details of experiments in Section 5.2
K.1 Experimental settings for Section 5.2
We consider both VGG-like models, with dropout rate= 0.15, 0.30J, parameterized over - size=(64, 256), initial learning rate=(0.1, 0.03), depth=(11, 16, 19), activation function=(ReLUU, tanh), weight decay=(0.0,, 0.0005), width=(1.0, 0.5) }, and AlexNet-like models, parameterized over Q which has the same parameters except that depth=(7, 9, 10). Therefore, either VGG or AlexNet has 2
288 configurations. Let
the set of configurations.
Formally, we use a normalized MI to quantify the relation between random variables Veo and Vpa with pa QU (GG), where 'co' denotes complexity mcasure and 'GG' gencralization gap. For example, Vdepth is the random variable for network depth.
Given a set of models, we have their associated hyper-parameter or generalization gap (pa(fw.))l) E ) and their respective values of the complexity measure {co(Swa))l E V). Let Veo(y;, 151) - sigu(co(Sw.st)) pa(fw,,,)) We compute their joint probability p(Veo: Vpa)
co co(fw,e,)) )), (pa 1by)) A sign(pa(fw.. and marginal probabilities v(Voa) and p( p(Pa)). The MI between Vco and Vpa can be computed
ΣΣ P(Voo, Vpa) log P(Vo, Vpa). p( p(Veo) ((vc))(Vpa))"
i(Voo: Vpa) I(Veo, Vpn) HVPpa)
T(Voo: Vpa)=
(24)
Veo Vpa
where (Voo. Vpa) can be further normalized Í(V.o. V/pa) to enable fair comparison between complexity measures, (-) is the marginal entropy. For the generalization gap, we consider not only i(Voo. VGG), but also the MI with some of the hyper-parameters fixed, i.e., Ewes Bucca(((eo. Vea (w)), for the cases of |w| = 0, 1, 2. Note that, when |w| = 0, it is the same as i(V.o. Vea). Intuitively, Euca (iVoo, Vealw)) with |ل| = 2 is the expected value of f(V... VGG) when 2 hyper-parameters in Ω are fixed. Note that, as aim to study the relationship between dropout and generalization, only compute Bucco(((V Voalw)) with |w| = 1, and 1(Voo, Vdropout) for all complexity measures.
The learning rate decays 50% per 20 epochs. 3 For VGG-like models, width means the model has 64, 128, 256, 512, 512 channels in structure, while width 0.5 that the model has 32, 64, 128, 256, 256 channels its structure.
26

---

Published in Transactions on Machine Learning Research (09/2022)
Table 12: Mutual Information of Complexity Measures on CIFAR-100.
Complexity
VGG
Measure
Dropout
GG[[u =2)
GG(|=/=1)
GG([w/=D)
Froh Distarce
0.0128
0.0172
D.I0668
0.0001
Spectral
Parameter Norm
0.0122
0,0168
0.0088
0.0011
Path Norm
0.0370
0.01.55
0.0025
0.0009
Sharpoess
0.0342
0.0164
0.0030
PAC-Sharpness
0.0390
0.0346
0.0166
0.0019
PAC-S ((Laplacc)
0.6225
0.1439
0.1256
0,1083
PAC-S (AC-S(Sampling)
0.3364
0.1001
D,0852
0,0787
V.0000
V:000247
V:0.7771
Vs V.0399
6:90
go
V:0.1684
V.0.1552
V:0.1273
V:0.1198
V: 0.0.1464
V:0.1502
Votari expariiion
d'ppour
Figure Disentanglement noise v.s. dropout. We train VGG16 networks (Left) and 2 AlexNet (Right) on CIFAR-10 and SVHN respoctively, with normal training, dropout training, disentanglement noise training (volume expansion) and stochastic noise training (volume contraction). For each dataset, we present their test losses in different plots.
K.2 Complexity measures
In the following, show the details of complexity mcasures. Frobenius (Frob) Distance and Parameter Norm (Dziugaite & Roy, 2017; Nagarajan & Kolter, 2019; Neyshabur et al., 2018b):
cOFrob(fw) Σ - W
(25)
co Σ 11W
(26)
Spectral Norm (Yoshida Miyalo, 2017):
(oSpon[SW) Σ I/ W
(27)
Path Norm: Path-norm was introduced in (Neyshabur et al., 2015b) as scale invariant complexity measure for generalization and is shown to be useful geometry for optimization (Neyshabur et al., 2015a).
To calculate path-norm, square the parameters of the network, do forward pass on an all-ones input and then take square root of sum of the network outputs. We define the following measures based on the path-norm:
copau((f) Ewa
(28)
where W/2 W W the element-wist square operation on the parameters. Sharpness-based based PAC-Bayes:
W.F W7IP
COPAC-S COPAC-s(fW)
Σ
(29)
20
27

---

Published in Transactions on Machine Learning Research (09/2022)
where is estimated by sharpness method. That is, let vec(U") ~ NN(0, be sample from mean Gaussian, where a is chosen to be the largest number such that max (([w+ur) - [(sw)] s (Keskar et al., 2016; Pitas et al., 2017; Neyshabur et al., 2017; Jiang et al., 2019).
PAC-Bayes with Weight Volume:
W - 1
CoPAC--SLLaacce//ammnnggg Σ
+ In
(30)
20
vol(w,)
where vol(wi) is estimated by Laplace approximation (PAC-S(Laplace) and sampling method (PAC- SSSmmpling).
K.3 Supplementary for Section 5.2
As Table 12 shows, for all cases concerning the generalization error on CIFAR-100 (VGG), i.e., w/= 0,1,2, PAC-S(Laplacc) or PAC-S(Sampling) is still a clear winner and also closely correlated with dropout.
L Supplementary for Experiment 5.3
As Figure shows, the CIFAR-10 and SVHN, weight volume expansion improves generalization performance, similar to dropout. This is in contrast to volume contraction, which leads to worse generalization performance.
M Additional experiments
M.1 Experiments on text classification
Table 13: Text classification for THUCNews
Method
FastTexx
Fast (drrooot))))
TextRNN
TextRNN
TexTCNN
(extCINN(drooout)
Sampling
0.04+0.02
0.08=0.02
0:03+0002
0.06=00.02
0.06==0.02
0.09-00.02
Laplace
DJ0366
0.0831
0.0126
0.0458
0.0453
0.0850
Train Loss
Test Loss
0.06
0.25
0.46
0.29
0.61
0.42
Loss Gap
0.65
0.17
0.42
0.22
0.59
0.38
Ace
Test Ace
0.9011
0.9217
0.9037
0,9102
0,9022
0,9129
Acc: Gap
0.0047
0.0627
0.0804
0.0649
0.0022
0.0770
make our empirical results more robust, we also implement FastText (Bojanowski et al., 2017; Joulin et al., 2016; 2017), TextRNN (Liu al., 2016) and TextCNN (Kim, 2014) classify THUCNeWs data set (Sun et al., 2016) (180000 training data, 10000 validation data, 10000 test data) with and without dropout. The results of those text classification tasks in Table 13 also indicate that dropout noise narrow gencralization gap and expand the weight volume.
M.2 Complexity measures with average sign-error
Table 14: Average sign-error of complexity measures on CIFAR-10.
Complexity
VGG
AlexNet GG
Measure
GG
Frob Distance
0.5374
0.5560
Spectral Norm
0.5045
0.6783
Parameter Norm
0.4817
0.6656
Path Norm
0.4852
0.4982
Sharpaess
PAC- SAC--hhapneess
0.5077
0.4434
(Lapaae))
0.3763
0.3247
PAC-S(Sampling)
28

---

Published in Transactions on Machine Learning Research (09/2022)
Figure 8 Some weight distributions of VGG 16 on CIFAR-10, red line is the true Gaussian, blue bar is the distribution of weights.
We also average sign-error (E [0, 1)) from Dxiugaite al. (2020b) to measure the effectiveness of complexity measures across all models. The smaller average sign-error means the better performance of complexity measures. Table 14 shows that our new complexity mcasures can also get better performances under the measure of average sign-error.
M.3 Distribution of weights
Gaussian distributed weights are commonly assumed in the analysis of DNN models under the PAC-Bayesian framework. To verify its rationality, we conducted a new experiment in Figure 8. We collect weight samples with sampling method and the empirical results show that these random weights (blue bar) approximatcly obey a Gaussian (red line).
29


Reviews:
[{"review_id": "LtIT2Caxfd", "reviewer": "Reviewer_YqAv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nPros:\n1. The idea is novel. It builds the connection between the generalization performance and the weight volume. \n2. The methods for calculating the weight volume are effective. \n3. The experimental results confirm the performance of the proposed methods.\n4. The writing is easy to follow. \n\nCons:\n1. In this paper, the weight is assumed to follow a Gaussian distribution, and then the generalization bound is obtained. Is this true for real-world applications? It would be good if the authors could conduct experiments to show the distribution of the weight of some DNNs.\n2. The proposed two approaches for computing the weight volume look slow. It would be good if the authors could show the consumed time. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the connection between dropout and generalization with a new concept: weight expansion. In particular, the authors show that the PAC-Bayesian generalization error is related to the weight volume, and dropout can lead to a large weight volume. Based on this finding, this paper developed two approaches to approximately calculate the weight volume. Finally, the experimental results confirm that weight volume is a good indicator of generalization performance. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1. In this paper, the weight is assumed to follow a Gaussian distribution, and then the generalization bound is obtained. Is this true for real-world applications? It would be good if the authors could conduct experiments to show the distribution of the weight of some DNNs.\n2. The proposed two approaches for computing the weight volume look slow. It would be good if the authors could show the consumed time. ", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MHzo7qNKHD", "reviewer": "Reviewer_yQtZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, I enjoyed reading the paper and found it fairly interesting. One of my main concerns is that under local perturbations of the weights (such as Gaussian), weight volume is basically equivalent to flatness. To my understanding, weight volume is a particular way to measure flatness. Since the authors do not clearly state these connections, the novelty of the idea seems to be a bit overrepresented in the paper. I think there should be a separate paragraph explaining how weight volume and flatness connect, and the relevant citations need to be discussed (of which there should be quite a significant number).\n\nPresentation: parts of the paper are fairly clearly written, parts are hard to follow (or make little sense, see below). \n\nAll theorem/lemma statements seem somewhat informal. The assumptions should be clearly listed in lemma statements, or listed prior to the statement and numbered, so that the theorem statement could refer to the assumptions. For example, it was too obscured in my opinion that P and Q share the same diagonal for obtaining the result in Lemma 3.1. I only realized after looking at the proof in the appendix. In addition, Lemma’s 3.1 and 3.2 present one-step analysis of correlations of the weights. The authors should make sure that the claims made in various discussions are accurate and point out that the analysis is one-step, or present multi-step analysis.\n\nSection 7 (Conclusion and future work) is very poorly written. The section is full of vacuous statements, and reads more like a space filler. It is hard to follow the logical structure (if there is structure). What’s the connection to adversarial generalization? What exactly is adversarial generalization? Why would the optimal solution not be practical? It was just said it is intractable, so it seems one cannot even go searching for this optimal solution, practical or not. \n\nClaims: i personally would prefer to see all the claims made in the paper and the respective evidence listed somewhere (with a precise description of the evidence, e.g., distinguishing between one-step and multi-step analysis, the extent of the experimental procedure, etc.)\n\nExperiments: the authors present two hypotheses that relate generalization, dropout, and weight expansion. They then present some empirical evidence that the quantities of interest are correlated. However, to truly test the hypothesis, some interventions changing the quantities of interest are needed. Let’s, for example, consider Hypothesis 1: “weight expansion reduces the generalization error”. In my opinion, to test such a hypothesis, one would need to think of experiments that could alter the weight expansion, and not through dropout only. I know the authors consider disentanglement noise, but that seems to be directly related to training neural networks essentially for perturbation robustness.\n\nIn Section 5.2 the authors claim to follow Jiang et al and Dziugaite et al for measuring correlations between generalization measures and generalization. Note, however, that the two mentioned papers propose different ways to evaluate the relationship between generalization and a generalization measure: one considers an average loss, the other considers a distributionally-robust loss. The authors should more carefully check both pieces of work, and have an appropriate discussion why they choose one over the other (or measure both).\n\nRegarding the evaluation of weight volume as a generalization measure: \n1) how is sigma for the prior chosen?  This detail may be mentioned in the paper but was not easy to find, and is important for the correctness of the evaluation of weight volume as a generalization measure (at least if the measure is justified through a PAC-Bayes bound).\n2) Is generalization measured of a perturbed or unperturbed classifier?\n\nMissing citations:  Huang et al ‘20 also used block-diagonal structure of the covariance matrix; McAllister ‘13 “A PAC-Bayesian tutorial with a dropout bound” – different from the submission, as the noise considered is diagonal Gaussian, and thus the weight volume part does not appear.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies dropout's effects on generalization. It introduces a notion of weight volume, and shows that under a particular perturbation of the weights, weight volume under dropout is preserved due to correlations among the weights being reduced over a single gradient step. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Critical changes based on the comments above:\n\n - clarifying the connections to flatness; add related work;\n - list all evidence toward supporting the two hypothesis presented in the introduction\n- list assumptions for theoretical results clearly; clarify wording around one-step vs multi-step analysis\n- add Section 5.2 extended evaluation based on two main competing methods\n- re-write conclusion section;\n - respond to questions (1) and (2) above to prove correctness of the empirical evaluation of weight volume as a generalization measure.\n\nChanges that would strengthen the paper:\n- evaluate hypothesis one by perturbing weight volume in different ways. E.g., what if after optimization I optimized for the maximum weight  volume further, would that increase generalization?", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tUN6UsRV3Q", "reviewer": "Reviewer_6GSD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is well-written and organized. Clear hypotheses are presented, a theoretical justification is provided, and experiments are used to test the hypotheses.\n- Experiments are systematic and well organized. Experiments are not on state-of-the-art models, but reasonable for this type of experiment.\n- The idea of having a measureable quantity that would help predict the generalization ability of a model is quite appealing from a practical perpsective.\n\nWeaknesses:\n- The theoretical arguments rely on a long list of approximations and assumptions. The authors do a nice job of referring to other papers in the literature where these assumptions are used, but don't discuss the limits of these assumptions and where they break down. \n- One simple example where it seems the theory might break down is in the analysis of dropout's connection to weight volume. When the dropout probabability q goes to 1, the gradient correlation goes to 0, but the weights themselves become highly correlated within a layer; the columns of W_l will converge to the same value, which will depend on the weights in the other layers. This scenario is not even unrealistic, as practicioners sometimes use high dropout rates in practice (e.g. 0.9), while the experiments in the paper only explore dropout rates up to 0.45.\n- Marginal experimental improvement of proposed algorithm over standard dropout. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a new explanation for how the dropout training algorithm improves generalization of neural network models. A quantity called \"weight volume\" --- defined as the normalized determininant of the weight covariance matrix --- is connected to generalization using the PAC-Bayes framework, and to dropout training by analyzing the correllation between gradients during dropout training. Experiments show correllations between dropout, weight volume, and generalization performance, performed with convolutional neural networks on CIFAR-10 and CIFAR-100. This new perspective suggests other forms of regularization that could be similar or better to dropout.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "- Perform a grammar check. There are some typos in the related works section.", "broader_impact_concerns": "None. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work explored a connection between weight expansion (measured by the determinant of normalized weight covariance) and dropout in neural network training. In particular, the authors demonstrated that weight expansion is an effective means of increasing generalization in a PAC-Bayesian setting, and argued that dropout, among other methods, indeed led to weight expansion. The authors performed some initial theoretical analysis and conducted extensive experiments to validate their claims. 

All three reviewers found this work interesting in the sense of providing a new perspective on the widely adopted dropout. Some questions on writing, related work, and assumptions were posed, which the authors satisfactorily addressed in the response. In the end, all reviewers recommended acceptance, which the AE concurs. 

Some minor suggestions for the final version: 
- Please consider abandoning the abbreviation "do" for dropout, since "do" already has an agreed-on meaning in causality. Using the tilde notation (e.g. $\tilde\eta$) probably suffices. 
- In the newly added Lemma 3.2 (as well as the text above it), ${\Delta^{do}}'$ was never defined. The assumption on the same distribution of ${\Delta^{do}}'$ seems a bit strong and falls into the single-step analysis pointed out by the reviewers. 
- Representing dropout as $(1+\eta)\odot a$ seems a bit awkward. Simply let $\tilde X = \frac{\delta}{1-q} X$ (with $X$ representing any quantity of interest), where $\delta \sim \mathrm{Bernoulli}(1-q)$, i.e., $\delta = 1$ with prob. $1-q$, would be more direct. For instance, the proof of Lemma 3.2 can be largely trivialized: $E(\tilde X \tilde Z) = E(\frac{\delta_X}{1-q} X \frac{\delta_Z}{1-q}  Z) = E(XZ)$, by conditioning on $X$ and $Z$ and taking expectation on the independent Bernoulli $\delta_X$ and $\delta_Z$. Similarly for $\mathrm{Var}(\tilde X)$ as well as Lemma 3.3. 
Decision: Accept (poster)