Abstract:
While dropout is known to be a successful regularization technique, insights into the mechanisms that lead to this success are still lacking. We introduce the concept of weight expansion, an increase in the signed volume of a parallelotope spanned by the column or row vectors of the weight covariance matrix, and show that weight expansion is an effective means of increasing the generalization in a PAC-Bayesian setting. We provide a theoretical argument that dropout leads to weight expansion and extensive empirical support for the correlation between dropout and weight expansion. To support our hypothesis that weight expansion can be regarded as an indicator of the enhanced generalization capability endowed by dropout, and not just as a mere by-product, we have studied other methods that achieve weight expansion (resp. contraction), and found that they generally lead to an increased (resp. decreased) generalization ability. This suggests that dropout is an attractive regularizer, because it is a computationally cheap method for obtaining weight expansion. This insight justifies the role of dropout as a regularizer, while paving the way for identifying regularizers that promise improved generalization through weight expansion.

heading: 1 INTRODUCTION
text: Research on why dropout is so effective in improving the generalization ability of neural networks has been intensive. Many intriguing phenomena induced by dropout have also been studied in this research (Gao et al., 2019; Lengerich et al., 2020; Wei et al., 2020). In particular, it has been suggested that dropout optimizes the training process (Baldi & Sadowski, 2013; Wager et al., 2013; Helmbold & Long, 2015; Kingma et al., 2015; Gal & Ghahramani, 2016; Helmbold & Long, 2017; Nalisnick et al., 2019) and that dropout regularizes the inductive bias (Cavazza et al., 2018; Mianjy et al., 2018; Mianjy & Arora, 2019; 2020).
A fundamental understanding of just how dropout achieves its success as a regularization technique will not only allow us to understand when (and why) to apply dropout, but also enable the design of new training methods. In this paper, we suggest a new measure, weight expansion, which both furthers our understanding and allows the development of new regularizers.
Broadly speaking, the application of dropout leads to non-trivial changes in the weight covariance matrix. As the weight covariance matrix is massively parameterized and hard to comprehend, we abstract it to the signed volume of a parallelotope spanned by the vectors of the matrix—weight volume (normalized generalized variance (Kocherlakota & Kocherlakota, 2004)) for short. Dropout increases this weight volume. Figure 1 illustrates the weight expansion obtained by dropout and visualizes how it improves the generalization ability.
This leads to our hypotheses that (1) weight expansion reduces the generalization error of neural networks (Section 3.1), and that (2) dropout leads to weight expansion (Section 3.2). We hypothesize that weight expansion can be an indicator of the increased generalization ability and dropout is ‘merely’ an efficient way to achieve it.
The weight volume is defined as the normalized determinant of the weight covariance matrix (determinant of weight correlation matrix), which is the second-order statistics of weight vectors when treated as a multivariate random variable. For example, when the weight vector is an isotropic Gaussian, the weight volume is large and the network generalizes well. Likewise, if the weights
are highly correlated, then the weight volume is small and the network does not generalize well. Technically, as a determinant, the weight volume serves as a measure of the orthogonality of the rows/columns of the weight covariance matrix. The more orthogonal the rows/columns are, the larger is the weight volume—which makes the different classes more distinguishable. Figure 1 visualizes that, with increasing weight volume, the decision boundaries of different classes become clearer. Note that, this is different from the “weight orthogonality” proposed in Saxe et al. (2013); Mishkin & Matas (2015); Bansal et al. (2018): Weight volume is a statistical property of weight matrices treated as random variables, while orthogonality treats weight matrices as deterministic variables (more discussions are provided in Appendix B).
In addition to our theoretical arguments that weight expansion promotes generalization and dropout leads to weight expansion (Section 3), our empirical results also support our hypotheses. To mitigate estimation errors, we consider two independent weight volume estimation methods in Section 4, i.e., sampling method and Laplace approximation. Section 5.1 demonstrates that the two methods lead to consistent results, which show that using dropout brings about weight expansion (coherence to hypothesis (2)) and generalization-improvement (coherence to hypothesis (1)) across data sets and networks.
Then, we embed the weight volume (a measurable quantity) to a new complexity measure that is based on our adaptation of the PAC-Bayesian bound (Section 3.1). Our experiments in Section 5.2, conducted on 3 × 288 network configurations, show that, comparing to existing complexity measures which do not consider weight volume, the new complexity measure is most closely correlated with generalization (coherence to hypothesis (1)) and dropout (coherence to hypothesis (2)) for these dropout networks with different dropout rates.
Finally, we use disentanglement noises (Section 4.1) to achieve weight expansion and further to improve generalization (or other noises to achieve contraction and further to destroy generalization, Section 5.3). This in particular supports hypothesis (1) and indicates that weight expansion is a key indicator for a reduced generalization error, while dropout is a computationally inexpensive means to achieve it. (Disentanglement noises, e.g., are more expensive.)

heading: 2 PRELIMINARIES
text: Notation. Let S be a training set with m samples drawn from the input distribution D, and s ∈ S a single sample. Loss, expected loss, and empirical loss are defined as `(fW (s)), LD(fW ) = Es∼D[`(fW (s))], and LS(fW ) = 1m ∑ s∈S [`(fW (s))], respectively, where fW (·) is a learning model. Note that, in the above notation, we omit the label y of the sample s, as it is clear from the context. Let W , Wl, and wij be the model weight matrix, the weight matrix of l-th layer, and the element of the i-th row and the j-th column in W , respectively. Note that we omit bias for convenience. To make a clear distinction, let W be the multivariate random variable of vec(W ) (vectorization ofW ), and wij be the random variable of wij . W 0 andWF denote the weight matrix before and after training, respectively. Further, we use W ∗ to represent the maximum-a-posteriori
(MAP) estimate of W . Note that WF can be seen as an approximation of W ∗ after training. We consider a feedforward neural network fW (·) that takes a sample s0 ∈ S as input and produces an output hL, after L fully connected layers. At the l-th layer (l = 1, . . . , L), the latent representation before and after the activation function is denoted as hl = Wlal−1 and al = actl(hl), respectively, where act(·) is the activation function. Node Dropout. We focus on node dropout (Hinton et al., 2012; Srivastava et al., 2014), a popular version of dropout that randomly sets the output of a given activation layer to 0. We adopt a formal definition of node dropout from Wei et al. (2020), which is more general than the usual Bernoulli formalism. For the layer l with the output of the activation function al ∈ RNl and a given probability ql ∈ [0, 1), we sample a scaling vector ηdol ∈ RNl (do is short for dropout) with independently and identically distributed elements
(ηdol )i = { −1 with probability ql, ql
1−ql with probability 1− ql. (1)
for i = 1, . . . , Nl, where Nl is the number of neurons at the layer l. With a slight abuse of notation, we denote by ηdol a vector of independently distributed random variables, each of which has a zero mean, and also a vector with each element being either −1 or ql1−ql when referring to a specific realization. As such, applying node dropout, we compute the updated output after the activation as adol = (1 + η do l ) al. That is, with probability ql, an element of the output with node dropout adol is reset to 0, and with probability 1 − ql it is rescaled with a factor 11−ql . The index of the layer l will be omitted when it is clear from the context. With different realizations of ηdo, a number of subnetworks with different connectivity will be produced during training.

heading: 3 DROPOUT AND WEIGHT EXPANSION
text: In this section, we first formally define the weight volume, which has been informally introduced in Figure 1. We aim to answer the following two questions: 1. Why does large weight volume promote generalization? 2. Why does dropout expand the weight volume?
Definition 3.1 (Weight Volume) Let Σl = E[(Wl − E(Wl))(Wl − E(Wl))ᵀ] be the weight covariance matrix in a neural network. The weight volume (also called as normalized generalized variance or determinant of weight correlation matrix) is defined as
vol(Wl) , det(Σl)∏ i[Σl]ii ∈ [0, 1], (2)
where the denominator is the product of the diagonal elements in Σl.
Intuitively, the weight volume comes from the geometric interpretation of the determinant of the matrix det(Σl), with normalization over the diagonal elements of the matrix. Because Σl is a covariance matrix and thus positive semi-definite, we have vol(Wl) ∈ [0, 1] since 0 ≤ det(Σl) ≤∏ i[Σl]ii. We note that vol(Wl) = 0 implies that the the weight covariance matrix does not have full rank, suggesting that some weights are completely determined by the others, while vol(Wl) = 1 suggests that the weights Wl are uncorrelated. The larger vol(Wl) is, the more dispersed are the weights.
Before presenting empirical results, we first provide theoretical support for our first hypothesis by adapting the PAC-Bayesian theorem (McAllester, 1999; Dziugaite & Roy, 2017) (Section 3.1): We show that a larger weight volume leads to a tighter bound for the generalization error. In Section 3.2, we argue that dropout can expand weight volume through the theoretical derivation.

heading: 3.1 WHY DOES LARGE WEIGHT VOLUME PROMOTE GENERALIZATION?
text: First, we support hypothesis (1) on the relevance of weight expansion through an extension of the PAC-Bayesian theorem, and then verify it empirically in Section 5. The PAC-Bayesian framework, developed in McAllester (1999); Neyshabur et al. (2017); Dziugaite & Roy (2017), connects weights with generalization by establishing an upper bound on the generalization error with respect to the Kullback-Leibler divergence (DKL) between the posterior distribution Q and the prior distribution P of the weights. In the following, we extend this framework to work with vol(Wl).
Let P be a prior and Q be a posterior over the weight space. For any δ > 0, with probability 1 − δ over the draw of the input space, we have (McAllester, 1999; Dziugaite & Roy, 2017)
EW∼Q[LD(fW)] ≤ EW∼Q[LS(fW)] +
√ DKL(Q||P ) + ln mδ
2(m− 1) . (3)
Given a learning setting, Dziugaite & Roy (2017); Neyshabur et al. (2017); Jiang et al. (2020b) assume P = N (µP ,ΣP ) and Q = N (µQ,ΣQ). Thus, DKL(Q||P ) can be simplified to
DKL(Q||P ) = 1
2
( tr(Σ−1P ΣQ)− k + (µP − µQ) ᵀ(ΣP ) −1(µP − µQ) + ln det(ΣP )
det(ΣQ)
) , (4)
where k is the dimension of W, tr denotes the trace, and det denotes the determinant. The derivation is given in Appendix D. To simplify the analysis, Neyshabur et al. (2017; 2018a) instantiate the prior P to be a vec(W 0) (or 0) mean and σ2 variance Gaussian distribution, and assume that the posterior Q to also be a vec(WF ) mean spherical Gaussian with variance σ2 in each direction.
We relax their assumption by letting Q be a non-spherical Gaussian (the off-diagonal correlations for same layer are not 0, whereas those for different layers are 0), while retaining the assumption on the σ2 variation in every direction (but allowing arbitrary covariance between them). This retains both tr(Σ−1P ΣQ) = k and ∏ i[Σl,P ]ii = ∏ i[Σl,Q]ii for all l, which provides the following.
DKL(Q||P ) = 1
2 ∑ l ( ||WFl −W 0l ||2F σ2l + ln ∏ i[Σl,P ]ii det(Σl,Q) ) = 1
2 ∑ l ( ||WFl −W 0l ||2F σ2l + ln 1 vol(Wl) ) .
(5)
Details of the derivation are provided in Appendix D. From Equations (3) and (5), we notice that the PAC-Bayesian upper bound of the generalization error becomes smaller when vol(Wl) increases, which is also demonstrated by wide experiments in Section 5.

heading: 3.2 WHY DOES DROPOUT EXPAND THE WEIGHT VOLUME?
text: In the above PAC-Bayesian framework, we assume the (non-spherical) Gaussian distributed weights and connect weight volume with generalization. In this subsection, to connect weight volume with dropout through correlation (of gradient updates and weights), and further to support hypothesis (2), we consider the arbitrarily distributed (with finite variance) gradient updates of l-th layer, ∆l for a normal network and ∆dol for the corresponding dropout network (all learning settings are same except for dropout). That is, vec(∆l) ∼M∆(µ∆l ,Σ∆l), and dropout noise ηdol is applied on gradient updates through
vec(∆dol ) = (1 + 1⊗ ηdol ) vec(∆do ′ l ),
where the random vector (1 + 1⊗ ηdol ) ∈ RNl−1Nl with ⊗ being Kronecker product. As we aim to study the impact of ηdol on the correlation of gradient updates, we assume that vec(∆ do′
l ) also obeys the distributionM∆(µ∆l ,Σ∆l) (for simplifying correlation analysis). Thus, with a slight abuse of notation, denote (1+1⊗ηdol ) vec(∆do ′ l ) as η do l to be applied on gradient updates of corresponding neurons for notational convenience. Then, we can now obtain Lemma 3.1.
Lemma 3.1 (Dropout reduces correlation of gradient updates.) Let ∆ij and ∆i′j′ be the gradient updates for wij and wi′j′ of some layer in a normal network, where i 6= i′. Then the (absolute) correlation between ∆doij and ∆ do i′j′ in a dropout network can be upper bounded by that in the normally trained network with the same setting, i.e.,∣∣ρ∆doij ,∆doi′j′ ∣∣ ≤ (1− q)∣∣ρ∆ij ,∆i′j′ ∣∣. (6) Details of the proof are given in Appendix E. The equality holds if, and only if, q = 0 or ρ∆ij ,∆i′j′ = 0. Lemma 3.1 shows that the dropout noise decreases correlation among gradient updates, which is also shown by experiments in Figure 2 and Appendix C. That is, the correlation among gradient updates in a normal network (Figure 2a) is obviously larger than the correlation in a dropout network (Figure 2c). Next, Lemma 3.1 leads to the following lemma.
Lemma 3.2 (Dropout reduces correlation of updated weights.) The (absolute) correlation of updated weights in a dropout network is upper bounded by that in the normal network. That is,∣∣ρwij+∆doij ,wi′j′+∆doi′j′ ∣∣ ≤ ∣∣Cov(wij + ∆ij ,wi′j′ + ∆i′j′)∣∣√( Var(wij + ∆ij) + qVar(∆ij) 1−q )( Var(wi′j′ + ∆i′j′) + qVar(∆i′j′ ) 1−q
) ≤ ∣∣ρwij+∆ij ,wi′j′+∆i′j′ ∣∣.
(7) The equality holds if and only if q = 0 or Cov(wij + ∆ij ,wi′j′ + ∆i′j′) = 0, and the full proof is given in Appendix F. It is clear that weight volume (equals the determinant of weight correlation matrix) is one of measures of correlation among weights (refer to Figure 1), and small correlation can lead to large weight volume. We also provide more discussions about the relationship between correlation and weight volume in Appendix G. Consistent with Lemma 3.2, we show in addition, that gradient updates with large correlation (Figure 2a) can lead to a large correlation of weights—and thus to a small weight volume (Figure 2b), while gradient updates with small correlation (Figure 2c, dropout) can lead to a small correlation of weights—and thus to a large weight volume.

heading: 4 WEIGHT VOLUME APPROXIMATION
text: To mitigate estimation errors, we use two methods to estimate the weight volume. One is the sampling method, and the other is Laplace approximation (Botev et al., 2017; Ritter et al., 2018) of neural networks, which models dropout as a noise in the Hessian matrix. Laplace approximation enables us to generalize dropout to other disentanglement noises (Section 4.1). Note that some of other popular regularizers, e.g., weight decay and batch normalization, improve generalization not through weight expansion (more discussions are provided in Appendix H).
Sampling Method. First, we use a sharpness-like method (Keskar et al., 2017; Jiang et al., 2020b) to get a set of weight samples drawn from (W + U) such that |L(fW+U ) − L(fW )| ≤ , where vec(U) ∼ N (0, σ2UI) is a multivariate random variable obeys zero mean Gaussian with σU = 0.1 (more details are given in Appendix I). The covariance between wij and wi′j′ can then be computed by Cov(wij ,wi′j′) = E[(wij − E[wij ])(wi′j′ − E[wi′j′ ])]. Finally, we can estimate vol(W) according to Equation (2).
Laplace Approximation. Laplace approximation has been used in posterior estimation in Bayesian inference (Bishop, 2006; Ritter et al., 2018). It aims to approximate the posterior distribution p(W|S) by a Gaussian distribution, based on the second-order Taylor approximation of the ln posterior around its MAP estimate. Specifically, for layer l and given weights with an MAP estimate W ∗l on S, we have
ln p(Wl|S) ≈ ln p ( vec(W ∗l )|S ) − 1
2
( Wl − vec(W ∗l ) )ᵀ Σ−1l ( Wl − vec(W ∗l ) ) , (8)
where Σ−1l = Es[Hl] is the expectation of the Hessian matrix over input data sample s, such that the Hessian matrix Hl is given by Hl =
∂2`(fW (s)) ∂vec(Wl)∂vec(Wl) .
It is worth noting that the gradient is zero around the MAP estimate W ∗, so the first-order Taylor polynomial is inexistent. Taking a closer look at Equation (8), one can find that its right hand side is exactly the logarithm of the probability density function of a Gaussian distributed multivariate
random variable with mean vec(W ∗l ) and covariance Σl, i.e., Wl ∼ N (vec(W ∗l ),Σl), where Σl can be viewed as the covariance matrix of Wl.
Laplace approximation suggests that it is possible to estimate Σl through the inverse of the Hessian matrix, because Σ−1l = Es[Hl]. Recently, Botev et al. (2017); Ritter et al. (2018) have leveraged insights from second-order optimization for neural networks to construct a Kronecker factored Laplace approximation. Differently from the classical second-order methods (Battiti, 1992; Shepherd, 2012), which suffer from high computational costs for deep neural networks, it takes advantage of the fact that Hessian matrices at the l-th layer can be Kronecker factored as explained in Martens & Grosse (2015); Botev et al. (2017). I.e.,
Hl = al−1a ᵀ l−1︸ ︷︷ ︸
Al−1
⊗ ∂ 2`(fW (s))
∂hl∂hl︸ ︷︷ ︸ Hl = Al−1 ⊗Hl, (9)
where Al−1 ∈ RNl−1×Nl−1 indicates the subspace spanned by the post-activation of the previous layer, and Hl,do ∈ RNl×Nl is the Hessian matrix of the loss with respect to the pre-activation of the current layer, with Nl−1 and Nl being the number of neurons at the (l − 1)-th and l-th layer, respectively. Further, through the derivation in Appendix J, we can estimate vol(Wl) efficiently by having
det(Σl) ≈ det ( (Es[Al−1])−1 )Nl · det ((Es[Hl])−1)Nl−1 (10) and ∏
i
[Σl]ii ≈ (∏
c
[ (Es[Al−1])−1 ] cc )Nl · (∏ d [ (Es[Hl])−1 ] dd )Nl−1 . (11) Note that, for the case with dropout, dropout noise ηdo is taken into account to compute Es[Al−1] and Es[Hl] (Appendix K). That is, Es[Adol−1] = Es,ηdol [a do l−1(a do l−1)
ᵀ] and Es[Hdol ] = Es,ηdol [ ∂2`(fW (s))
∂hdol ∂h do l
], where adol−1 and h do l are the activation and pre-activation of the corresponding
layer in a dropout network.

heading: 4.1 OTHER DISENTANGLEMENT NOISE
text: Whilst introducing dropout noise can expand weight volume (as shown in the experiments of Sections 5.1 and 5.2), it is interesting to know (1) whether there are other noises that can lead to weight expansion, and, if so, whether they also lead to better generalization performance, and (2) whether there are noises that can lead to weight contraction, and, if so, whether those noises lead to worse generalization performance.
Algorithm 1 Gradient update with disentanglement noise. Input: minibatch {si}ni=1, activation noise ηa, node loss noise ηh, noise strengths λ1, λ2. I Forward Propagation: apply noise λ1 ·ηa to activations. I Back Propagation: apply noise λ2 · ηh to node loss. I Calculate g = 1n ∑n i=1∇W ( `(fW (si)) ) by considering both noises as indicated above. I Use g for the optimization algorithm. As indicated in Equation (9) and Appendix J, the weight covariance matrix can be approximated by the Kronecker product of (E[Al−1])−1 and (E[Hl])−1. Disentanglement noises, as an attempt, can be injected into either term during the gradient update procedure to increase det(E[Al−1])−1 and det(E[Hl])−1 for the purpose of weight expansion. The first option is to inject ηa to activations during forward propagation, as follows.
Activation Reverse Noise ηa to increase det(E[A])−1. We define ηa as ηa ∼ N(µ, r(Σa)), where Σa = (E[A])−1 and r(Σ) is to reverse the signs of off-diagonal elements of the matrix Σ. The second option is to inject ηh into the node loss during backpropagation, as follows.
Node Loss Reverse Noise ηh to increase det(E[H])−1. We define ηh as ηh ∼ N(µ, r(Σh)), where Σh = (E[H])−1. Algorithm 1 presents our simple and effective way to obtain disentanglement noise, where λ1 and λ2 are hyper-parameters to balance the strengths of ηa and ηh. In the experiments in Section 5.3, we will assign expansion noises and contraction noises according to their impact on the weight volume, where expansion noises are obtained from Algorithm 1 and contraction noises are obtained from stochastic noises on weights.

heading: 5 EXPERIMENTS
text: We have conducted a number of experiments to test our hypotheses that (1) weight expansion helps reduce the generalization error and (2) dropout increases the weight volume. More than 900 networks are trained. First, we confirm the role of weight expansion in connecting dropout and generalization by estimating the weight volume with two independent methods (Section 5.1). Second, we extend the PAC-Bayesian theorem with the awareness to weight volume, and show that the new complexity measure can significantly improve the prediction of the generalization error for dropout networks (Section 5.2). Finally, we consider other noises that can lead to either weight expansion or weight contraction, and study their respective impact on the generalization performance (Section 5.3). All empirical results are both significant and persistent across the models and data sets we work with, and support either hypothesis (1) or hypothesis (2) or both. In our experiments, we consider VGG-like models (Simonyan & Zisserman, 2014) and AlexNet-like models (Krizhevsky et al., 2012) for CIFAR-10, CIFAR-100 (Krizhevsky et al., 2009), ImageNet-32 (Deng et al., 2009), and SVHN (Netzer et al., 2011). For sampling method, we default to the settings of = 0.05 for CIFAR-10/SVHN, and = 0.1 for CIFAR-100/ImageNet-32.

heading: 5.1 MEASURING WEIGHT VOLUME
text: (hypothesis (2)). At the same time, the generalization error is reduced (hypothesis (1)). We default to the setting of weight decay with 0 in the experiments of Table 1, Figure 3, and also present similar results with weight decay 0.0005 (and more results on ImageNet-32) in Appendix L.
After confirming the qualitative correlation between weight volume and dropout, we consider the quantitative effect of the dropout rate on the weight volume. As Figure 3 shows, for the 4 VGG16 models with dropout rate 0.0, 0.15, 0.3, 0.45, respectively, we see a clear trend on CIFAR-10/100 that the weight volume increases with the dropout rate (hypothesis (2)). The generalization gap is also monotonically reduced (CIFAR-10: 0.86, 0.51, 0.30, 0.16; CIFAR-100: 3.75, 2.14, 1.07, 0.39) (hypothesis (1)).
In summary, these experiments provide evidence to the correlation between dropout, weight expansion, and generalization. This raises two follow-up questions: (1) can the weight volume serve as a proxy for the generalization gap? and (2) can techniques other than dropout exhibit a similar behavior? Answers to these questions can shed light on the nature of the correlation between these concepts.

heading: 5.2 PERFORMANCE OF THE NEW COMPLEXITY MEASURE
text: A complexity measure can predict generalization gap without resorting to a test set. Inspired by the experiments in Jiang et al. (2020b), we have trained 288*3=864 dropout networks (with dif-
ferent rates: 0, 0.15, 0.3) on CIFAR-10 for VGG-like and AlexNet-like structures and CIFAR-100 for VGG-like structures to compare our new complexity measure—which incorporates the weight volume—with a set of existing complexity measures, including PAC-Sharpness, which was shown in Jiang et al. (2020b) to perform better overall than many others. Following Jiang et al. (2020b); Dziugaite et al. (2020a), we use mutual information to measure the correlation between various complexity measures, and dropout and generalization gap (GG, defined as test loss - training loss). The details of experimental settings (e.g., GG(|ω|=2)) and mutual information are given in Appendix M.1.
We compare our new complexity measures (Equation (5), with Laplace approximation and sampling method respectively) with some existing ones, including Frobenius (Frob) Distance, Parameter Norm (Dziugaite & Roy, 2017; Neyshabur et al., 2018b; Nagarajan & Kolter, 2019), Spectral Norm (Yoshida & Miyato, 2017), Path Norm (Neyshabur et al., 2015a;b), Sharpness α′, and PACSharpness (Keskar et al., 2017; Neyshabur et al., 2017; Pitas et al., 2017). Among them, PACSharpness is suggested in Jiang et al. (2020b) as the the most promising one in their experiments.
Table 2 presents the mutual information as described above on CIFAR-10 for VGG-like models and AlexNet-like models. Details of the above complexity measures are given in Appendix M.2. More empirical results on CIFAR-100 are given in Appendix M.3. We can see that, for most cases concerning the generalization error, i.e., |ω| = 0, 1, 2 (fix 0, 1, 2 hyper-parameters respectively, and compute corresponding expected mutual information over sub-spaces, details are shown in Appendix M.1), our new measures, PAC-S(Laplace) and PAC-S(Sampling), achieve the highest mutual information among all complexity measures. On AlexNet, the mutual information of other complexity measures increase, but our new measures still perform best. Generally, the mutual information values of GG for our measures are significantly higher than those for others (which supports hypothesis (1)), because our complexity measures can predict generalization-improvement from weight expansion, which is mainly caused by dropout in our network configurations. Meanwhile, hypothesis (2) is also empirically verified as our complexity measures are most closely correlated with dropout.

heading: 5.3 DISENTANGLEMENT NOISE OTHER THAN DROPOUT
text: The experiments in Sections 5.1 and 5.2 provide support to the hypotheses (1)&(2) that dropout, weight expansion, and generalization are correlated, with an indication that weight expansion is the proxy to the connection between dropout and the generalization error. Our final experiment is designed to test this observation by considering the replacement of dropout. We consider the disentanglement noise as presented in Algorithm 1, and train a set of VGG16 networks and AlexNet on CIFAR-10, CIFAR-100, Imagenet-32 and SVHN, with normal training,
dropout training, disentanglement noise training (volume expansion), and stochastic noise training (volume contraction), respectively. As stochastic noises on weights may lead to either weight volume contraction or leave it unchanged, we collect the contraction cases for our experiments. Figure 4 shows that volume expansion improves generalization performance, similar to dropout. Likewise, volume contraction leads to a worse generalization performance. This relation, which further supports hypothesis (1), is persistent across the examined networks and data sets. This suggests that the weight expansion, instead of dropout, can serve as a key indicator of good generalization performance, while dropout provides one method to implement volume expansion. Disentanglement noises may be a valid replacements for dropout.

heading: 6 RELATED WORK
text: From the perspectives of optimizing the training process, related work includes (Baldi & Sadowski, 2013; Wager et al., 2013; Kingma et al., 2015; Helmbold & Long, 2015; 2017; Gal & Ghahramani, 2016; Nalisnick et al., 2019). For example, Wager et al. (2013) considers dropout training as adaptive regularization; Gal & Ghahramani (2016) interprets dropout with a spike-and-slab distribution by variational approximation; and Kingma et al. (2015) re-parameterizes noise on the weights as uncertainty in the hidden units, and proposes variational dropout, a generalization of Gaussian dropout.
Recent research has started to provide a narrative understanding of dropout’s performance (Zhang et al., 2021), and providing variational dropout (Wang & Manning, 2013; Kingma et al., 2015; Gal & Ghahramani, 2016; Gal et al., 2017; Achille & Soatto, 2018; Fan et al., 2020; Lee et al., 2020; Pham & Le, 2021; Fan et al., 2021). And Cavazza et al. (2018); Mianjy et al. (2018); Mianjy & Arora (2019); Arora et al. (2019); Mianjy & Arora (2020) provide a theoretical analysis to understand dropout from the perspective of regularizing the inductive bias. Different from these, we consider explaining dropout through the rigorously defined concept of weight expansion.
Other than dropout, there are works studying how other factors affect the generalization of neural networks (Neyshabur et al., 2018a; Chatterji et al., 2020; Dziugaite et al., 2020b; Sen et al., 2020; Zitong et al., 2020; Hrayr et al., 2020; Mahdi et al., 2020; Menon et al., 2021; Montero et al., 2021), and the relation between gradient noises and generalization (Keskar et al., 2017; Keskar & Socher, 2017; Jastrzebski et al., 2017; Smith & Le, 2018; Xing et al., 2018; Chaudhari & Soatto, 2018; Jastrzebski et al., 2019; Li et al., 2019; Zhu et al., 2019; Jin et al., 2020; Wen et al., 2020) was studied. There was also a NeurIPS 2020 Competition on Predicting Generalization in Deep Learning (Jiang et al., 2020a; Lassance et al., 2020; Natekar & Sharma, 2020).
Since the invention of the PAC-Bayesian framework (McAllester, 1999), there have been a number of canonical works developed in the past decades, including the parametrization of the PACBayesian bound with Gaussian distribution (Seeger, 2002; Langford & Caruana, 2002; Langford & Shawe-Taylor, 2003), the early works about generalization error bounds for learning problems (Maurer, 2004; Germain et al., 2009; Welling & Teh, 2011; Parrado-Hernández et al., 2012) and the recent works about PAC-Bayes for machine learning models (Dwork et al., 2015a;b; Blundell et al., 2015; Alquier et al., 2016; Thiemann et al., 2017; Dziugaite & Roy, 2018; Rivasplata et al., 2019; Letarte et al., 2019; Rivasplata et al., 2020; Pérez-Ortiz et al., 2021b; Haddouche et al., 2021; Pérez-Ortiz et al., 2021a). Specifically, Langford & Shawe-Taylor (2003); Germain et al. (2009); Parrado-Hernández et al. (2012) demonstrate that one can obtain tight PAC-Bayesian generalization bounds through multiplying the weight vector norm with a constant, this is different with the idea of expanding weight volume, as weight volume is a statistical property of weight matrices treated as random variables, while weight vector norm treats weights as deterministic variables (weight vector norm can be seen as the first term in Equation (5), while weight volume is the second term). It is interesting that Alquier et al. (2016) also optimizes PAC-Bayesian bound through a Gaussian posterior with a full covariance matrix. We use a similar assumption (correlated Gaussian posterior) but the main differences are: we develop it under another PAC-Bayesian branch (Dziugaite & Roy, 2017; Neyshabur et al., 2017; 2018a) and apply it on DNNs with connection to dropout noise.

heading: 7 CONCLUSION
text: The paper introduces ‘weight expansion’, an intriguing phenomenon that appears whenever dropout is applied during training. We have provided both theoretical and empirical arguments, which show that weight expansion can be one of the key indicators of the reduced generalization error, and that dropout is effective because it is (an inexpensive) method to obtain such weight expansion.

heading: 8 ETHICS STATEMENT AND REPRODUCIBILITY STATEMENT
text: Ethics Statement. In this work, we present theoretical and empirical results to justify two hypotheses: (1) weight expansion helps reduce the generalization error and (2) dropout increases the weight volume. All the empirical results are obtained on publicly available benchmark data sets (e.g., CIFAR-10/100, SVHN, ImageNet-32). Thus, our work does not involve any ethics issues.
Reproducibility Statement. In Section 3, to theoretically support hypothesis (1) (through PACBayesian framework) and hypothesis (2) (through correlation of gradient updates and weights), we have clearly explained (or cited) the assumptions (e.g., Gaussian distributed weights, arbitrarily distributed gradient updates), and have made the complete proofs (or derivations, for Section 3.1, Lemma 3.1 and Lemma 3.2) in Appendices D to F, and have discussed the relationship between correlation of weights and weight volume in Appendix G (with a simple proof and two simulations). As for the empirical results in Sections 5.1 to 5.3, experiments in Sections 5.1 and 5.2 verify both hypothesis (1) and hypothesis (2), while Section 5.3 further supports hypothesis (1). We have uploaded the corresponding scripts to the supplementary file. All these results are conducted and validated on multiple data sets and multiple network architectures.

heading: A Details of Figure 1 16

heading: B Weight Volume and Weight Orthogonality 16

heading: C Supplementary for Figure 2 16

heading: D Detailed Derivation of PAC-Bayes 17

heading: E Proof of Lemma 3.1 18

heading: F Proof of Lemma 3.2 19

heading: G Weight Volume and Correlation 19

heading: H Other Regularizers with Weight Volume 20

heading: I Supplementary for Sampling Method 20

heading: J Estimate Weight Volume 20

heading: K Embedding Dropout Noise ηdo to Covariance Matrix Σdol 21

heading: L Supplementary for Section 5.1 21

heading: M Details of Experiments in Section 5.2 22

heading: N Supplementary for Experiment 5.3 24

heading: O Experiments on Text Classification 24

heading: P Future Work 24

heading: Q Rebuttal to Reviewer iaGH 26

heading: R Rebuttal to Reviewer wW51 29

heading: S Rebuttal to Reviewer w6em 33
text: T Rebuttal to Reviewer nB1c 34

heading: A DETAILS OF FIGURE 1
text: As Table 3 shows, we use the above two networks to train MNIST with ReLU activation function, 0.01 learning rate, 128 minibatch size, stochastic gradient descent (SGD), 150 epochs, with and without dropout. Normal CNN and dropout CNN contain multiple convolutional layers with 64 and 128 channels, following with 128 and 2 hidden units. We visualize the features of Dense-2 layer in Figure 1.

heading: B WEIGHT VOLUME AND WEIGHT ORTHOGONALITY
text: For clarification, weight volume is not the orthogonality of weight matrices, but the “normalized generalized variance” of weight matrices. It is a statistical property of weight matrices when treated as random variables, which is different from the orthogonality technique in Saxe et al. (2013); Mishkin & Matas (2015); Bansal et al. (2018) that treat weight matrices as deterministic variables.
Specifically, by treating the elements in weight matrices as random variables (to fit the PAC-Bayes framework), the weight volume is defined as the determinant of the “covariance matrix” of weight matrices (see Definition 3.1). The optimization of the determinant of the covariance will not necessarily lead to orthogonality of weight matrices. It appears that weight volume considers the statistical correlation between any two weights (treated as random variables), whereas weight orthogonality considers the linear space correlation between any two columns in weight matrices.
E.g., multivariate random variable W is made up of two terms, say W = vec(W ) + UW , where vec(W ) is the vectorization of weight matrix and UW is the 0 mean multivariate random variable. Weight volume relies on the covariance matrix of UW , while weight orthogonality depends on W .

heading: C SUPPLEMENTARY FOR FIGURE 2
text: Figure 5 also shows that the correlation among gradient updates in the normal network (Figure 5a) is larger than the correlation in dropout network (Figure 5c) and large correlation of gradient updates can lead to small weight volume (large correlation of weights), while small correlation of gradient updates can lead to large weight volume (small correlation of weights).

heading: D DETAILED DERIVATION OF PAC-BAYES
text: McAllester (1999); Dziugaite & Roy (2017) consider a training set S with m ∈ N samples drawn from a distribution D. Given the prior and posterior distributions P and Q on the weights W respectively, for any δ > 0, with probability 1− δ over the draw of training set, we have
EW∼Q[LD(fW)] ≤ EW∼Q[LS(fW)] +
√ DKL(Q||P ) + ln mδ
2(m− 1) . (12)
Equation (4): Let P = N (µP ,ΣP ) and Q = N (µQ,ΣQ), we can get DKL(N (µQ,ΣQ)||N (µP ,ΣP )) = ∫ [ln(Q(x))− ln(P (x))]Q(x)dx
= ∫ [ 1
2 ln det ΣP det ΣQ − 1 2 (x− µQ)ᵀΣ−1Q (x− µQ) + 1 2 (x− µP )ᵀΣ−1P (x− µP )]Q(x)dx
= 1
2 ln det ΣP det ΣQ − 1 2 tr{E[(x− µQ)(x− µQ)ᵀ]Σ−1Q }+ 1 2 tr{E[(x− µP )ᵀΣ−1P (x− µP )]}
= 1
2 ln det ΣP det ΣQ − 1 2 tr(Ik) + 1 2 (µQ − µP )ᵀΣ−1P (µQ − µP ) + 1 2 tr(Σ−1P ΣQ)
= 1
2
[ tr(Σ−1P ΣQ) + (µQ − µP ) ᵀΣ−1P (µQ − µP )− k + ln det(ΣP )
det(ΣQ)
] .
(13)
Equation (5): Neyshabur et al. (2017); Jiang et al. (2020b) instantiate the prior P to be a vec(W 0) mean, σ2 variance Gaussian distribution and the posterior Q to be a vec(WF ) mean spherical Gaussian with variance σ2 in each direction. Relax their assumption, we assume Q is a non-spherical Gaussian (the off-diagonal correlations for same layer are not 0, those for different layers are 0), then we can get
DKL(Q||P ) = ||WF −W 0||2F
2σ2 +
1 2 ln det(ΣP )
det(ΣQ) = ∑ l ||WFl −W 0l ||2F 2σ2 + 1 2 ln ∏ l det(Σl,P ) det(Σl,Q)
= 1
2 ∑ l ( ||WFl −W 0l ||2F σ2 + ln det(Σl,P ) det(Σl,Q) ) = 1
2 ∑ l ( ||WFl −W 0l ||2F σ2l + ln ∏ i[Σl,P ]ii det(Σl,Q) ) = 1
2 ∑ l ( ||WFl −W 0l ||2F σ2l + ln ∏ i[Σl,Q]ii det(Σl,Q) ) = 1
2 ∑ l ( ||WFl −W 0l ||2F σ2l + ln 1 vol(Wl) ) .
(14)

heading: E PROOF OF LEMMA 3.1
text: Proof E.1 Let ∆ij , ∆i′j′ be the gradient updates for wij , wi′j′ of some layer in a normal network, i 6= i′, ∆doij and ∆doi′j′ be the gradient updates for wij , wi′j′ in the same setting dropout network. Let vec(∆) ∼M∆(µ∆,Σ∆) and vec(∆do) = (1+1⊗ηdo) vec(∆do ′ ), vec(∆do ′ ) ∼M∆(µ∆,Σ∆). Note that, as we aim to study the impact of ηdo on the correlation among gradient updates, we assume vec(∆) and vec(∆do ′ ) obey the same arbitrary distribution M∆. As dropout noise and gradient update are independent, dropout noises for different neurons (kernels) are independent, we have
Cov(∆doij ,∆ do i′j′) = E∆do [ (∆doij − µ∆doij )(∆ do i′j′ − µ∆do i′j′ ) ]
= (1− q)2E∆do′ [ ( ∆do ′ ij
1− q − µ∆ij )(
∆do ′
i′j′
1− q − µ∆i′j′ )
] + (1− q)q
E∆do′ [ ( ∆do ′ i′j′
1− q − µ∆i′j′ )(−µ∆ij )
] + (1− q)qE∆do′ [ ( ∆do ′ ij
1− q − µ∆ij )(−µ∆i′j′ ) ] + q2 [ (−µ∆i′j′ )(−µ∆ij )
] = E∆do′ [ (∆do ′ ij − µ∆ij )(∆do ′ i′j′ − µ∆i′j′ ) ]
= E∆ [ (∆ij − µ∆ij )(∆i′j′ − µ∆i′j′ ) ] = Cov(∆ij ,∆i′j′),
(15)
and Var(∆doij ) = E∆do [ (∆doij − µ∆doij ) 2 ]
= (1− q)E∆do′ [ ( ∆do ′ ij
1− q − µ∆ij )2
] + q(−µ∆ij )2
= E∆do′
[ (∆do ′ ij − µ∆ij )2 ]
1− q + qµ2∆ij 1− q
= E∆ [ (∆ij − µ∆ij )2 ] 1− q + qµ2∆ij 1− q
= Var(∆ij) 1− q + qµ2∆ij 1− q .
(16)
Similarly, Var(∆doi′j′) = Var(∆i′j′ ) 1−q + qµ2∆ i′j′
1−q . Thus, we have∣∣ρ∆doij ,∆doi′j′ ∣∣ = ∣∣Cov(∆doij ,∆doi′j′)∣∣√ Var(∆doij )Var(∆ do i′j′)
= ∣∣Cov(∆ij ,∆i′j′)∣∣√( Var(∆ij)
1−q + qµ2∆ij 1−q
)( Var(∆i′j′ )
1−q + qµ2∆ i′j′ 1−q ) ≤ (1− q) ∣∣Cov(∆ij ,∆i′j′)∣∣√
Var(∆ij)Var(∆i′j′) = (1− q) ∣∣ρ∆ij ,∆i′j′ ∣∣
≤ ∣∣ρ∆ij ,∆i′j′ ∣∣.
(17)

heading: F PROOF OF LEMMA 3.2
text: Proof F.1 As dropout noise and weight are independent, and W ∼ NW, we have Cov(wij ,∆ do i′j′) = Ew,∆do [ (wij − µwij )(∆doi′j′ − µ∆do i′j′ ) ]
= (1− q)Ew,∆do′ [ (wij − µwij )( ∆do ′ i′j′
1− q − µ∆i′j′ )
] + qEw [ (wij − µwij )(−µ∆i′j′ ) ] = Ew,∆do′ [ (wij − µwij )(∆do ′ i′j′ − µ∆i′j′ ) ]
= Ew,∆ [ (wij − µwij )(∆i′j′ − µ∆i′j′ ) ] = Cov(wij ,∆i′j′),
(18)
Cov(wij ,∆ do ij ),Cov(wi′j′ ,∆ do i′j′) and Cov(wi′j′ ,∆ do ij ) are similar. From Equations (15) and (16),
we can get∣∣ρwij+∆doij ,wi′j′+∆doi′j′ ∣∣ =
|Cov(wij + ∆doij ,wi′j′ + ∆doi′j′)|√ Var(wij + ∆doij )Var(wi′j′ + ∆ do i′j′)
= ∣∣Cov(wij ,wi′j′) + Cov(∆doij ,∆doi′j′) + Cov(wij ,∆doi′j′) + Cov(wi′j′ ,∆doij )∣∣√( Var(wij) + Var(∆doij ) + 2Cov(wij ,∆ do ij ) )( Var(wi′j′) + Var(∆doi′j′) + 2Cov(wi′j′ ,∆ do i′j′)
) =
∣∣Cov(wij ,wi′j′) + Cov(∆ij ,∆i′j′) + Cov(wij ,∆i′j′) + Cov(wi′j′ ,∆ij)∣∣√( Var(wij) + Var(∆doij ) + 2Cov(wij ,∆ij) )( Var(wi′j′) + Var(∆doi′j′) + 2Cov(wi′j′ ,∆i′j′)
) =
∣∣Cov(wij ,wi′j′) + Cov(∆ij ,∆i′j′) + Cov(wij ,∆i′j′) + Cov(wi′j′ ,∆ij)∣∣√( Var(wij) + Var(∆ij) 1−q + qµ2∆ij 1−q + 2Cov(wij ,∆ij)
) 1√(
Var(wi′j′) + Var(∆i′j′ ) 1−q + qµ2∆ i′j′ 1−q + 2Cov(wi′j′ ,∆i′j′) )
≤ ∣∣Cov(wij + ∆ij ,wi′j′ + ∆i′j′)∣∣√(
Var(wij + ∆ij) + qVar(∆ij)
1−q
)( Var(wi′j′ + ∆i′j′) + qVar(∆i′j′ )
1−q ) ≤
∣∣Cov(wij + ∆ij ,wi′j′ + ∆i′j′)∣∣√ Var(wij + ∆ij)Var(wi′j′ + ∆i′j′)
= ∣∣ρwij+∆ij ,wi′j′+∆i′j′ ∣∣.

heading: G WEIGHT VOLUME AND CORRELATION
text: Since the definition of weight volume in Definition 3.1, the weight volume equals the determinant of the weight correlation matrix.
It is difficult to theoretically argue an ‘exact’ relationship (e.g., perfect positive or negative) between weight volume and correlation of weights, but they have a rough negative relationship. Take a simple example, let n-dimensional weight correlation matrix C have the same correlation ρ, it is easy to infer that (absolute) correlation is negative related with weight volume, as
det(C) = (1− ρ)n−1 ( 1 + (n− 1)ρ ) , (19)
where d det(C)dρ = (n− n 2)ρ(1− ρ)n−2 and n ≥ 2. Also, we demonstrate the relationship between weight volume and average (absolute) correlation with 10000 samples for a 3-dimensional correlation matrix (Figure 6a) and a 4-dimensional correlation matrix (Figure 6b) respectively. Although weight volume is not monotonically increasing as (absolute) correlation decreases, we can find a rough negative relationship.

heading: H OTHER REGULARIZERS WITH WEIGHT VOLUME
text: As dropout can generate independent noises but other regularizers, like weight decay and batch normalization, cannot. Thus, dropout can introduce randomness into the network, while weight decay and batch normalization cannot (these two regularizers improve generalization not through weight volume, through other keys). We have described in detail why this dropout noise can expand weight volume and why weight volume is correlated with generalization, theoretically (Section 3) and empirically (Section 5, Figures 2 and 5).
Also, in Figure 6c, we can see that dropout expands weight volume compared with normal network, while weight decay and batch normalization have hardly any impact on weight volume.

heading: I SUPPLEMENTARY FOR SAMPLING METHOD
text: Sampling Method: As mentioned in Section 4, we use sharpness-like method (Keskar et al., 2017) to get a set of weight samples (W +U). To get the samples from the posteriori distribution steadily and fastly, we train the convergent network with learning rate 0.0001, noise U and 50 epochs, then collect corresponding 50 samples. As the samples are stabilized at training loss and validation loss but with different weights, we can treat them as the samples from same posteriori distribution. Details are given in the code.

heading: J ESTIMATE WEIGHT VOLUME
text: The approximation of Es[Hl] is two-fold: (1) Only the diagonal blocks of the weight matrix are captured, so that the correlation between blocks is decoupled; (2) AssumeAl−1 andHl are independent (Botev et al., 2017; Ritter et al., 2018). Then, we have
Es[Hl] = Es[Al−1 ⊗Hl] ≈ Es[Al−1]⊗ Es[Hl]. (20)
Finally, Σl can be approximated as follows:
Σl=(Es[Hl])−1 ≈(Es[Al−1])−1⊗(Es[Hl])−1. (21)
Therefore, by separately computing the matrix inverse of Es[Al−1] and Es[Hl], whose sizes are significantly smaller, we can estimate Σl ∈ RNl−1Nl×Nl−1Nl efficiently. Further, the weight volume can be approximated by using
det(Σl) ≈ det ( (Es[Al−1])−1 )Nl · det ((Es[Hl])−1)Nl−1 (22) and ∏
i
[Σl]ii ≈ (∏
c
[ (Es[Al−1])−1 ] cc )Nl · (∏ d [ (Es[Hl])−1 ] dd )Nl−1 . (23)
Table 4: Weight Volume on VGG Networks (ImageNet-32, weight decay = 0).
Method VGG11 VGG11(dropout) VGG16 VGG16(dropout) VGG19 VGG19(dropout)
Sampling 0.015±0.01 0.089±0.01 0.016±0.01 0.097±0.01 0.017±0.01 0.086±0.01 Laplace 0.0198 0.0985 0.0194 0.0942 0.0214 0.0977
Generalization Gap 7.0388 2.0501 6.3922 1.2430 7.5187 0.7910
Table 5: VGG11 Structure
Conv-64 Maxpooling Conv-128 Maxpooling Conv-256 Dropout(0.3) Conv-256 Maxpooling Conv-512 Dropout(0.3) Conv-512 Maxpooling Conv-512 Dropout(0.3) Conv-512 Maxpooling Dropout(0.1) Dense-512 Dropout(0.1) Dense-(num classes)
K EMBEDDING DROPOUT NOISE ηdo TO COVARIANCE MATRIX Σdol
For Laplace approximation (Section 4), the standard dropout, as a noise term in adol = (1+η do l ) al, can be framed into the posterior estimation via Hessian matrices in Equation (9).

heading: L SUPPLEMENTARY FOR SECTION 5.1
text: As Table 4 shows, on the ImageNet-32, dropout still leads to persistent, and significant, increase on the weight volume, across all three models. At the same time, the generalization error is reduced.
In addition, we show the VGG network structures in Tables 5 to 7, and also demonstrate the details (including training loss, training accuracy, test loss, test accuracy) of the VGG experiments on ImageNet-32 (Tables 8 and 9) and CIFAR-10/100 (Tables 10 and 11). All models for CIFAR10/100 are trained for 140 epochs using SGD with momentum 0.9, batch size 128, weight decay 0 or 5×10−4, and an initial learning rate of 0.1 that is divided by 2 after every 20 epochs. All models for ImageNet-32 are trained for 100 epochs using SGD with momentum 0.9, batch size 1024, weight decay 0 or 5× 10−4, and an initial learning rate of 0.1 that is divided by 2 for each 20 epochs, and learning rate of 0.0005 after 50th epoch.
All results in Tables 8 to 11 also suggest that dropout can expand the weight volume and narrow generalization gap.

heading: M DETAILS OF EXPERIMENTS IN SECTION 5.2
text: M.1 EXPERIMENTAL SETTINGS FOR SECTION 5.2
We consider both VGG-like models, with dropout rate={0.0, 0.15, 0.30}, parameterized over ΩVGG={batch size={64, 256}, initial learning rate={0.1, 0.03},1 depth={11, 16, 19}, activation function={ReLU, tanh}, weight decay={0.0, 0.0005}, width={1.0, 0.5}2 }, and AlexNet-like models, parameterized over ΩAlex, which has the same parameters except that depth={7, 9, 10}. Therefore, either VGG or AlexNet has 2 ∗ 2 ∗ 3 ∗ 2 ∗ 2 ∗ 2 ∗ 3 = 288 configurations. Let Ψ be the set of configurations.
Formally, we use a normalized mutual information to quantify the relation between random variables Vco and Vpa with pa ∈ Ω ∪ {GE}, where ‘co’ denotes complexity measure and ‘GE’ generalization error. For example, Vdepth is the random variable for network depth.
Given a set of models, we have their associated hyper-parameter or generalization gap {pa(fΘ,ψ)|ψ ∈ Ψ}, and their respective values of the complexity measure {co(fΘ,ψ)|ψ ∈ Ψ}. Let Vco(ψi, ψj) , sign(co(fΘ,ψi) − co(fΘ,ψj )), Vpa(ψi, ψj) , sign(pa(fΘ,ψi) − pa(fΘ,ψj )). We can compute their joint probability p(Vco, Vpa) and marginal probabilities p(Vco) and p(Vpa). The mutual information between Vco and Vpa can be computed as
I(Vco, Vpa) = ∑ Vco ∑ Vpa p(Vco, Vpa) log p(Vco, Vpa) p(Vco)p(Vpa) , Î(Vco, Vpa) = I(Vco, Vpa) H(Vpa) , (24)
where I(Vco, Vpa) can be further normalized as Î(Vco, Vpa) to enable a fair comparison between complexity measures, H(·) is the marginal entropy. For the generalization gap, we consider not only Î(Vco, VGE), but also the mutual information with some of the hyper-parameters fixed, i.e., Eω⊂Ω ( Î(Vco, VGE|ω) ) , for the cases of |ω| = 0, 1, 2. Note that, when |ω| = 0, it is the same as
Î(Vco, VGE). Intuitively, Eω⊂Ω ( Î(Vco, VGE|ω) ) with |ω| = 2 is the expected value of Î(Vco, VGE)
1The learning rate decays 50% per 20 epochs. 2For VGG-like models, width = 1 means the model has 64, 128, 256, 512, 512 channels in its structure,
while width = 0.5 means that the model has 32, 64, 128, 256, 256 channels in its structure.
when 2 hyper-parameters in Ω are fixed. Note that, as we aim to study the relationship between dropout and generalization, we only compute Eω⊂Ω ( Î(Vco, VGE|ω) ) with |ω| = 0, 1, 2 and
Î(Vco, Vdropout) for all complexity measures.
M.2 COMPLEXITY MEASURES
In the following, we show the details of complexity measures.
Frobenius (Frob) Distance and Parameter Norm (Dziugaite & Roy, 2017; Nagarajan & Kolter, 2019; Neyshabur et al., 2018b):
coFrob(fW ) = ∑ l ||WFl −W 0l ||2F . (25)
coPara(fW ) = ∑ l ||WFl ||2F . (26)
Spectral Norm (Yoshida & Miyato, 2017): coSpec(fW ) = ∑ l ||WFl −W 0l ||22. (27)
Path Norm: Path-norm was introduced in (Neyshabur et al., 2015b) as an scale invariant complexity measure for generalization and is shown to be a useful geometry for optimization (Neyshabur et al., 2015a). To calculate path-norm, we square the parameters of the network, do a forward pass on an all-ones input and then take square root of sum of the network outputs. We define the following measures based on the path-norm:
coPath(fW ) = ∑ l fW 2(1), (28)
where W 2 = W ◦W is the element-wise square operation on the parameters. Sharpness-based PAC-Bayes:
coPAC−S(fW ) = ∑ l ‖WFl −W 0l ‖2F 2σ2l , (29)
where σ is estimated by sharpness method. That is, let vec(U ′) ∼ N (µU′ , σ′2I) be a sample from a zero mean Gaussian, where σ is chosen to be the largest number such that maxσ′≤σ|L(fW+U ′)− L(fW )| ≤ (Keskar et al., 2017; Pitas et al., 2017; Neyshabur et al., 2017; Jiang et al., 2020b). PAC-Bayes with Weight Volume:
coPAC−S(Laplace/Sampling)(fW ) = ∑ l [‖WFl −W 0l ‖2F 2σ2l + ln 1 vol(Wl) ] , (30)
where vol(Wl) is estimated by Laplace approximation (PAC-S(Laplace)) and sampling method (PAC-S(Sampling)).
M.3 SUPPLEMENTARY FOR SECTION 5.2
As Table 12 shows, for all cases concerning the generalization error on CIFAR-100 (VGG), i.e., |ω|= 0,1,2, PAC-S(Laplace) or PAC-S(Sampling) is still a clear winner and is also closely correlated with dropout.

heading: N SUPPLEMENTARY FOR EXPERIMENT 5.3
text: As Figure 7 shows, on the CIFAR-10 and SVHN, weight volume expansion improves generalization performance, similar to dropout. This is in contrast to volume contraction, which leads to worse generalization performance.

heading: O EXPERIMENTS ON TEXT CLASSIFICATION
text: To make our empirical results more robust, we also implement FastText (Bojanowski et al., 2017; Joulin et al., 2016; 2017), TextRNN (Liu et al., 2016) and TextCNN (Kim, 2014) to classify THUCNews data set (Sun et al., 2016) (180000 training data, 10000 validation data, 10000 test data) with and without dropout. The results of these text classification tasks in Table 13 also indicate that dropout noise can narrow generalization gap and expand the weight volume.

heading: P FUTURE WORK
text: In this paper, we introduce ‘weight expansion’, an intriguing phenomenon that appears whenever dropout is applied during training. We have provided both theoretical and empirical arguments, which show that weight expansion is one of the key indicators of the reduced generalization error, and that dropout is effective to obtain weight expansion.
For the next steps, we believe the following few directions are worthy of exploration and research.
• First, at the moment, we consider weights as a vector, without taking into consideration the structural information that the weights can be closely coupled according to the structures of the layers. It is interesting to know whether or not weight volume may capture some structural information of network (e.g., convolutional layer).
• Second, it will be useful to develop more PAC-Bayesian optimization methods (Dziugaite & Roy, 2017; Letarte et al., 2019; Pérez-Ortiz et al., 2021b) to help expand weight volume.
While the methods in the paper have been shown effective and efficient, they are by no means optimal. It is not hard to see that such optimization problem is highly intractable, so an optimal solution may not be practical. Nevertheless, we believe there are rooms for improvement, and any progress in this direction will be significant for not only the problem in this paper but also other problems that may involve PAC-Bayesian optimization.
• Third, activation functions have been argued to have significant impact on not only the trained models but also the training process, related to the first point, it is interesting to understand how weight volume is related to activation functions (e.g., ReLU, tanh).
• Last but not least, it can be a significant topic to study whether the weight volume – when combined with the PAC-Bayesian adversarial generalization bound (Farnia et al., 2018) – can provide any guidance to the hot topic of adversarial generalization.

Review summaries:
The paper proposes the "weight volume" as a new metric for neural network model selection. To compute the weight volume, one should first consider a vector of random variables that take values "around" the weights of the neural network without changing its outputs. Then the weight volume is measured from the normalized determinant of the covariance matrix. The paper shows mathematically that increasing the weight volume decreases a PAC-Bayesian generalization bound, and (2) the dropout method naturally endeavors weight volume expansion. Then, the paper presents two methods for estimating the weight volume of a neural network (which is mandatory as neural networks commonly have a large number of parameters). Finally, extensive experiments show empirically a clear correlation between the introduced metric and the generalization gap of neural networks, 

Authors study the generalization error through the notion of weight expansion. It is shown that a weight expansion reduces the generalization error and that dropout is an efficient method that increases the weight volume. Authors also argue that weight expansion should be regarded as the cause of better generalization.

This paper studies the important problem of answering the question "Why does drop-out help with generalization"? The answer presented in this paper comes in two stages by relating drop-out to weight expansion and weight-expansion to generalization. They show both their claims are theoretically supported and then experimentally validate the findings on various applications that are of interest. Their findings are not entirely theoretical however, as they lead to a new method via weight expansion.