Weight Expansion: A New Perspective on Dropout and Generalization

1 INTRODUCTION . Research on why dropout is so effective in improving the generalization ability of neural networks has been intensive . Many intriguing phenomena induced by dropout have also been studied in this research ( Gao et al. , 2019 ; Lengerich et al. , 2020 ; Wei et al. , 2020 ) . In particular , it has been suggested that dropout optimizes the training process ( Baldi & Sadowski , 2013 ; Wager et al. , 2013 ; Helmbold & Long , 2015 ; Kingma et al. , 2015 ; Gal & Ghahramani , 2016 ; Helmbold & Long , 2017 ; Nalisnick et al. , 2019 ) and that dropout regularizes the inductive bias ( Cavazza et al. , 2018 ; Mianjy et al. , 2018 ; Mianjy & Arora , 2019 ; 2020 ) . A fundamental understanding of just how dropout achieves its success as a regularization technique will not only allow us to understand when ( and why ) to apply dropout , but also enable the design of new training methods . In this paper , we suggest a new measure , weight expansion , which both furthers our understanding and allows the development of new regularizers . Broadly speaking , the application of dropout leads to non-trivial changes in the weight covariance matrix . As the weight covariance matrix is massively parameterized and hard to comprehend , we abstract it to the signed volume of a parallelotope spanned by the vectors of the matrix—weight volume ( normalized generalized variance ( Kocherlakota & Kocherlakota , 2004 ) ) for short . Dropout increases this weight volume . Figure 1 illustrates the weight expansion obtained by dropout and visualizes how it improves the generalization ability . This leads to our hypotheses that ( 1 ) weight expansion reduces the generalization error of neural networks ( Section 3.1 ) , and that ( 2 ) dropout leads to weight expansion ( Section 3.2 ) . We hypothesize that weight expansion can be an indicator of the increased generalization ability and dropout is ‘ merely ’ an efficient way to achieve it . The weight volume is defined as the normalized determinant of the weight covariance matrix ( determinant of weight correlation matrix ) , which is the second-order statistics of weight vectors when treated as a multivariate random variable . For example , when the weight vector is an isotropic Gaussian , the weight volume is large and the network generalizes well . Likewise , if the weights are highly correlated , then the weight volume is small and the network does not generalize well . Technically , as a determinant , the weight volume serves as a measure of the orthogonality of the rows/columns of the weight covariance matrix . The more orthogonal the rows/columns are , the larger is the weight volume—which makes the different classes more distinguishable . Figure 1 visualizes that , with increasing weight volume , the decision boundaries of different classes become clearer . Note that , this is different from the “ weight orthogonality ” proposed in Saxe et al . ( 2013 ) ; Mishkin & Matas ( 2015 ) ; Bansal et al . ( 2018 ) : Weight volume is a statistical property of weight matrices treated as random variables , while orthogonality treats weight matrices as deterministic variables ( more discussions are provided in Appendix B ) . In addition to our theoretical arguments that weight expansion promotes generalization and dropout leads to weight expansion ( Section 3 ) , our empirical results also support our hypotheses . To mitigate estimation errors , we consider two independent weight volume estimation methods in Section 4 , i.e. , sampling method and Laplace approximation . Section 5.1 demonstrates that the two methods lead to consistent results , which show that using dropout brings about weight expansion ( coherence to hypothesis ( 2 ) ) and generalization-improvement ( coherence to hypothesis ( 1 ) ) across data sets and networks . Then , we embed the weight volume ( a measurable quantity ) to a new complexity measure that is based on our adaptation of the PAC-Bayesian bound ( Section 3.1 ) . Our experiments in Section 5.2 , conducted on 3 × 288 network configurations , show that , comparing to existing complexity measures which do not consider weight volume , the new complexity measure is most closely correlated with generalization ( coherence to hypothesis ( 1 ) ) and dropout ( coherence to hypothesis ( 2 ) ) for these dropout networks with different dropout rates . Finally , we use disentanglement noises ( Section 4.1 ) to achieve weight expansion and further to improve generalization ( or other noises to achieve contraction and further to destroy generalization , Section 5.3 ) . This in particular supports hypothesis ( 1 ) and indicates that weight expansion is a key indicator for a reduced generalization error , while dropout is a computationally inexpensive means to achieve it . ( Disentanglement noises , e.g. , are more expensive . ) 2 PRELIMINARIES . Notation . Let S be a training set with m samples drawn from the input distribution D , and s ∈ S a single sample . Loss , expected loss , and empirical loss are defined as ` ( fW ( s ) ) , LD ( fW ) = Es∼D [ ` ( fW ( s ) ) ] , and LS ( fW ) = 1m ∑ s∈S [ ` ( fW ( s ) ) ] , respectively , where fW ( · ) is a learning model . Note that , in the above notation , we omit the label y of the sample s , as it is clear from the context . Let W , Wl , and wij be the model weight matrix , the weight matrix of l-th layer , and the element of the i-th row and the j-th column in W , respectively . Note that we omit bias for convenience . To make a clear distinction , let W be the multivariate random variable of vec ( W ) ( vectorization ofW ) , and wij be the random variable of wij . W 0 andWF denote the weight matrix before and after training , respectively . Further , we use W ∗ to represent the maximum-a-posteriori ( MAP ) estimate of W . Note that WF can be seen as an approximation of W ∗ after training . We consider a feedforward neural network fW ( · ) that takes a sample s0 ∈ S as input and produces an output hL , after L fully connected layers . At the l-th layer ( l = 1 , . . . , L ) , the latent representation before and after the activation function is denoted as hl = Wlal−1 and al = actl ( hl ) , respectively , where act ( · ) is the activation function . Node Dropout . We focus on node dropout ( Hinton et al. , 2012 ; Srivastava et al. , 2014 ) , a popular version of dropout that randomly sets the output of a given activation layer to 0 . We adopt a formal definition of node dropout from Wei et al . ( 2020 ) , which is more general than the usual Bernoulli formalism . For the layer l with the output of the activation function al ∈ RNl and a given probability ql ∈ [ 0 , 1 ) , we sample a scaling vector ηdol ∈ RNl ( do is short for dropout ) with independently and identically distributed elements ( ηdol ) i = { −1 with probability ql , ql 1−ql with probability 1− ql . ( 1 ) for i = 1 , . . . , Nl , where Nl is the number of neurons at the layer l. With a slight abuse of notation , we denote by ηdol a vector of independently distributed random variables , each of which has a zero mean , and also a vector with each element being either −1 or ql1−ql when referring to a specific realization . As such , applying node dropout , we compute the updated output after the activation as adol = ( 1 + η do l ) al . That is , with probability ql , an element of the output with node dropout adol is reset to 0 , and with probability 1 − ql it is rescaled with a factor 11−ql . The index of the layer l will be omitted when it is clear from the context . With different realizations of ηdo , a number of subnetworks with different connectivity will be produced during training . 3 DROPOUT AND WEIGHT EXPANSION . In this section , we first formally define the weight volume , which has been informally introduced in Figure 1 . We aim to answer the following two questions : 1 . Why does large weight volume promote generalization ? 2 . Why does dropout expand the weight volume ? Definition 3.1 ( Weight Volume ) Let Σl = E [ ( Wl − E ( Wl ) ) ( Wl − E ( Wl ) ) ᵀ ] be the weight covariance matrix in a neural network . The weight volume ( also called as normalized generalized variance or determinant of weight correlation matrix ) is defined as vol ( Wl ) , det ( Σl ) ∏ i [ Σl ] ii ∈ [ 0 , 1 ] , ( 2 ) where the denominator is the product of the diagonal elements in Σl . Intuitively , the weight volume comes from the geometric interpretation of the determinant of the matrix det ( Σl ) , with normalization over the diagonal elements of the matrix . Because Σl is a covariance matrix and thus positive semi-definite , we have vol ( Wl ) ∈ [ 0 , 1 ] since 0 ≤ det ( Σl ) ≤∏ i [ Σl ] ii . We note that vol ( Wl ) = 0 implies that the the weight covariance matrix does not have full rank , suggesting that some weights are completely determined by the others , while vol ( Wl ) = 1 suggests that the weights Wl are uncorrelated . The larger vol ( Wl ) is , the more dispersed are the weights . Before presenting empirical results , we first provide theoretical support for our first hypothesis by adapting the PAC-Bayesian theorem ( McAllester , 1999 ; Dziugaite & Roy , 2017 ) ( Section 3.1 ) : We show that a larger weight volume leads to a tighter bound for the generalization error . In Section 3.2 , we argue that dropout can expand weight volume through the theoretical derivation . 3.1 WHY DOES LARGE WEIGHT VOLUME PROMOTE GENERALIZATION ? . First , we support hypothesis ( 1 ) on the relevance of weight expansion through an extension of the PAC-Bayesian theorem , and then verify it empirically in Section 5 . The PAC-Bayesian framework , developed in McAllester ( 1999 ) ; Neyshabur et al . ( 2017 ) ; Dziugaite & Roy ( 2017 ) , connects weights with generalization by establishing an upper bound on the generalization error with respect to the Kullback-Leibler divergence ( DKL ) between the posterior distribution Q and the prior distribution P of the weights . In the following , we extend this framework to work with vol ( Wl ) . Let P be a prior and Q be a posterior over the weight space . For any δ > 0 , with probability 1 − δ over the draw of the input space , we have ( McAllester , 1999 ; Dziugaite & Roy , 2017 ) EW∼Q [ LD ( fW ) ] ≤ EW∼Q [ LS ( fW ) ] + √ DKL ( Q||P ) + ln mδ 2 ( m− 1 ) . ( 3 ) Given a learning setting , Dziugaite & Roy ( 2017 ) ; Neyshabur et al . ( 2017 ) ; Jiang et al . ( 2020b ) assume P = N ( µP , ΣP ) and Q = N ( µQ , ΣQ ) . Thus , DKL ( Q||P ) can be simplified to DKL ( Q||P ) = 1 2 ( tr ( Σ−1P ΣQ ) − k + ( µP − µQ ) ᵀ ( ΣP ) −1 ( µP − µQ ) + ln det ( ΣP ) det ( ΣQ ) ) , ( 4 ) where k is the dimension of W , tr denotes the trace , and det denotes the determinant . The derivation is given in Appendix D. To simplify the analysis , Neyshabur et al . ( 2017 ; 2018a ) instantiate the prior P to be a vec ( W 0 ) ( or 0 ) mean and σ2 variance Gaussian distribution , and assume that the posterior Q to also be a vec ( WF ) mean spherical Gaussian with variance σ2 in each direction . We relax their assumption by letting Q be a non-spherical Gaussian ( the off-diagonal correlations for same layer are not 0 , whereas those for different layers are 0 ) , while retaining the assumption on the σ2 variation in every direction ( but allowing arbitrary covariance between them ) . This retains both tr ( Σ−1P ΣQ ) = k and ∏ i [ Σl , P ] ii = ∏ i [ Σl , Q ] ii for all l , which provides the following . DKL ( Q||P ) = 1 2 ∑ l ( ||WFl −W 0l ||2F σ2l + ln ∏ i [ Σl , P ] ii det ( Σl , Q ) ) = 1 2 ∑ l ( ||WFl −W 0l ||2F σ2l + ln 1 vol ( Wl ) ) . ( 5 ) Details of the derivation are provided in Appendix D. From Equations ( 3 ) and ( 5 ) , we notice that the PAC-Bayesian upper bound of the generalization error becomes smaller when vol ( Wl ) increases , which is also demonstrated by wide experiments in Section 5 .

Summary: This paper studies the important problem of answering the question "Why does drop-out help with generalization"? The answer presented in this paper comes in two stages by relating drop-out to weight expansion and weight-expansion to generalization. They show both their claims are theoretically supported and then experimentally validate the findings on various applications that are of interest. Their findings are not entirely theoretical however, as they lead to a new method via weight expansion.