Towards Understanding Regularization in Batch Normalization

1 INTRODUCTION . Batch Normalization ( BN ) is an indispensable component in many deep neural networks ( He et al. , 2016 ; Huang et al. , 2017 ) . BN has been widely used in various areas such as machine vision , speech and natural language processing . Experimental studies ( Ioffe & Szegedy , 2015 ) suggested that BN improves convergence and generalization by enabling large learning rate and preventing overfitting when training deep networks . Understanding BN theoretically is a key question . This work investigates regularization of BN as well as its optimization and generalization in a singlelayer perceptron , which is a building block of deep models , consisting of a kernel layer , a BN layer , and a nonlinear activation function such as ReLU . The computation of BN is written by y = g ( ĥ ) , ĥ = γ h− µB σB + β and h = wTx . ( 1 ) This work denotes a scalar and a vector by using lowercase letter ( e.g . x ) and bold lowercase letter ( e.g . x ) respectively . In Eqn . ( 1 ) , y is the output of a neuron , g ( · ) denotes an activation function , h and ĥ are hidden values before and after batch normalization , w and x are kernel weight vector and network input respectively . In BN , µB and σB represent the mean and standard deviation of h. They are estimated within a batch of samples for each neuron independently . γ is a scale parameter and β is a shift parameter . In what follows , Sec.1.1 overviews assumptions and main results , and Sec.1.2 presents relationships with previous work . 1.1 OVERVIEW OF RESULTS . We overview results in three aspects . • First , Sec.2 decomposes BN into population normalization ( PN ) and gamma decay . To better understand BN , we treat a single-layer perceptron with ReLU activation function as an illustrative case . Despite the simplicity of this case , it is a building block of deep networks and has been widely adopted in theoretical analyses such as proper initialization ( Krogh & Hertz , 1992 ; Advani & Saxe , 2017 ) , dropout ( Wager et al. , 2013 ) , weight decay and data augmentation ( Bös , 1998 ) . The results in Sec.2 can be extended to deep neural networks as presented in Appendix C.4 . Our analyses assume that neurons at the BN layer are independent similar to ( Salimans & Kingma , 2016 ; van Laarhoven , 2017 ; Teye et al. , 2018 ) , as the mean and the variance of BN are estimated individually for each neuron of each layer . The form of regularization in this study does not rely on ∗The first three authors contribute equally . Corresponding to pluo.lhi @ gmail.com , { wangxinjiang , pengzhanglin } @ sensetime.com , weqish @ link.cuhk.edu.hk . Gaussian assumption on the network input and the weight vector , meaning our assumption is milder than those in ( Yoshida et al. , 2017 ; Ba et al. , 2016 ; Salimans & Kingma , 2016 ) . Sec.2 tells us that BN has an explicit regularization form , gamma decay , where µB and σB have different impacts : ( 1 ) µB discourages reliance on a single neuron and encourages different neurons to have equal magnitude , in the sense that corrupting individual neuron does not harm generalization . This phenomenon was also found empirically in a recent work ( Morcos et al. , 2018 ) , but has not been established analytically . ( 2 ) σB reduces kurtosis of the input distribution as well as correlations between neurons . ( 3 ) The regularization strengths of these statistics are inversely proportional to the batch size M , indicating that BN with large batch would decrease generalization . ( 4 ) Removing either one of µB and σB could imped convergence and generalization . • Second , by using ordinary differential equations ( ODEs ) , Sec.3 shows that gamma decay enables the network trained with BN to converge with large maximum learning rate and effective learning rate , compared to the network trained without BN or trained with weight normalization ( WN ) ( Salimans & Kingma , 2016 ) that is a counterpart of BN . The maximum learning rate ( LR ) represents the largest LR value that allows training to converge to a fixed point without diverging , while effective LR represents the actual LR in training . Larger maximum and effective LRs imply faster convergence rate . • Third , Sec.4 compares generalization errors of BN , WN , and vanilla SGD by using statistical mechanics . The “ large-scale ” regime is of interest , where number of samples P and number of neurons N are both large but their ratio P/N is finite . In this regime , the generalization errors are quantified both analytically and empirically . Numerical results in Sec.5 show that BN in CNNs has the same traits of regularization as disclosed above . 1.2 RELATED WORK . Neural Network Analysis . Many studies conducted theoretical analyses of neural networks ( Opper et al. , 1990 ; Saad & Solla , 1996 ; Bs & Opper , 1998 ; Pennington & Bahri , 2017 ; Zhang et al. , 2017b ; Brutzkus & Globerson , 2017 ; Raghu et al. , 2017 ; Mei et al. , 2016 ; Tian , 2017 ) . For example , for a multilayer network with linear activation function , Glorot & Bengio ( 2010 ) explored its SGD dynamics and Kawaguchi ( 2016 ) showed that every local minimum is global . Tian ( 2017 ) studied the critical points and convergence behaviors of a 2-layered network with ReLU units . Zhang et al . ( 2017b ) investigated a teacher-student model when the activation function is harmonic . In ( Saad & Solla , 1996 ) , the learning dynamics of a committee machine were discussed when the activation function is error function erf ( x ) . Unlike previous work , this work analyzes regularization emerged in BN and its impact to both learning and generalization , which are still unseen in the literature . Normalization . Many normalization methods have been proposed recently . For example , BN ( Ioffe & Szegedy , 2015 ) was introduced to stabilize the distribution of input data of each hidden layer . Weight normalization ( WN ) ( Salimans & Kingma , 2016 ) decouples the lengths of the network parameter vectors from their directions , by normalizing the parameter vectors to unit length . The dynamic of WN was studied by using a single-layer network ( Yoshida et al. , 2017 ) . Li et al . ( 2018 ) diagnosed the compatibility of BN and dropout ( Srivastava et al. , 2014 ) by reducing the variance shift produced by them . Moreover , van Laarhoven ( 2017 ) showed that weight decay has no regularization effect when using together with BN or WN . Ba et al . ( 2016 ) demonstrated when BN or WN is employed , backpropagating gradients through a hidden layer is scale-invariant with respect to the network parameters . Santurkar et al . ( 2018 ) gave another perspective of the role of BN during training instead of reducing the covariant shift . They argued that BN results in a smoother optimization landscape and the Lipschitzness is strengthened in networks trained with BN . However , both analytical and empirical results of regularization in BN are still desirable . Our study explores regularization , optimization , and generalization of BN in the scenario of online learning . Regularization . Ioffe & Szegedy ( 2015 ) conjectured that BN implicitly regularizes training to prevent overfitting . Zhang et al . ( 2017a ) categorized BN as an implicit regularizer from experimental results . Szegedy et al . ( 2015 ) also conjectured that in the Inception network , BN behaves similar to dropout to improve the generalization ability . Gitman & Ginsburg ( 2017 ) experimentally compared BN and WN , and also confirmed the better generalization of BN . In the literature there are also implicit regularization schemes other than BN . For instance , random noise in the input layer for data augmentation has long been discovered equivalent to a weight decay method , in the sense that the inverse of the signal-to-noise ratio acts as the decay factor ( Krogh & Hertz , 1992 ; Rifai et al. , 2011 ) . Dropout ( Srivastava et al. , 2014 ) was also proved able to regularize training by using the generalized linear model ( Wager et al. , 2013 ) . 2 A PROBABILISTIC INTERPRETATION OF BN . The notations in this work are summarized in Appendix Table 2 for reference . Training the above single-layer perceptron with BN in Eqn . ( 1 ) typically involves minimizing a negative log likelihood function with respect to a set of network parameters θ = { w , γ , β } . Then the loss function is defined by 1 P P∑ j=1 ` ( ĥj ) = − 1 P P∑ j=1 log p ( yj |ĥj ; θ ) + ζ‖θ‖22 , ( 2 ) where p ( yj |ĥj ; θ ) represents the likelihood function of the network and P is number of training samples . As Gaussian distribution is often employed as prior distribution for the network parameters , we have a regularization term ζ‖θ‖22 known as weight decay ( Krizhevsky et al. , 2012 ) that is a popular technique in deep learning , where ζ is a coefficient . To derive regularization of BN , we treat µB and σB as random variables . Since one sample x is seen many times in the entire training course , and at each time x is presented with the other samples in a batch that is drawn randomly , µB and σB can be treated as injected random noise for x . Prior of µB , σB . By following ( Teye et al. , 2018 ) , we find that BN also induces Gaussian priors for µB and σB . We have µB ∼ N ( µP , σ 2 P M ) and σB ∼ N ( σP , ρ+2 4M ) , where M is batch size , µP and σP are population mean and standard deviation respectively , and ρ is kurtosis that measures the peakedness of the distribution of h. These priors tell us that µB and σB would produce Gaussian noise in training . There is a tradeoff regarding this noise . For example , when M is small , training could diverge because the noise is large . This is supported by experiment of BN ( Wu & He , 2018 ) where training diverges when M = 2 in ImageNet ( Russakovsky et al. , 2015 ) . When M is large , the noise is small because µB and σB get close to µP and σP . It is known that M > 30 would provide a moderate noise , as the sample statistics converges in probability to the population statistics by the weak Law of Large Numbers . This is also supported by experiment ( Ioffe & Szegedy , 2015 ) where BN with M = 32 already works well in ImageNet . 2.1 A REGULARIZATION FORM . The loss function in Eqn . ( 2 ) can be written as an expected loss by integrating over the priors of µB and σB , that is , 1P ∑P j=1 EµB , σB [ ` ( ĥj ) ] where E [ · ] denotes expectation . We show that µB and σB impose regularization on the scale parameter γ by decomposing BN into population normalization ( PN ) and gamma decay . To see this , we employ a single-layer perceptron and ReLU activation function as an illustrative example . A more rigorous description is provided in Appendix C.1 . Regularization of µB , σB . Let ` ( ĥ ) be the loss function defined in Eqn . ( 2 ) and ReLU be the activation function . We have 1 P P∑ j=1 EµB , σB ` ( ĥ j ) ' 1 P P∑ j=1 ` ( h̄j ) ︸ ︷︷ ︸ PN + ζ ( h ) γ2︸ ︷︷ ︸ gamma decay , and ζ ( h ) = ρ+ 2 8M I ( γ ) ︸ ︷︷ ︸ from σB + 1 2M 1 P P∑ j=1 σ ( h̄j ) ︸ ︷︷ ︸ from µB , ( 3 ) where h̄j = γ h j−µP σP + β and hj = wTxj represent the computations of PN . ζ ( h ) γ2 represents gamma decay , where ζ ( h ) is an adaptive decay factor depended on the hidden value h. Moreover , ρ is the kurtosis of distribution of h , I ( γ ) represents an estimation of the Fisher information of γ and I ( γ ) = 1P ∑P j=1 ( ∂ ` ( ĥj ) ∂γ ) 2 , and σ ( · ) is a sigmoid function . From Eqn . ( 3 ) , we have several observations that have both theoretical and practical values . • First , PN replaces the batch statistics µB , σB in BN by the population statistics µP , σP . In gamma decay , computation of ζ ( h ) is data-dependent , making it differed from weight decay where the coefficient is determined manually . In fact , Eqn . ( 3 ) recasts the randomness of BN in a deterministic manner , not only enabling us to apply methodologies such as ODEs and statistical mechanics to analyze BN , but also inspiring us to imitate BN ’ s performance by WN without computing batch statistics in empirical study . • Second , PN is closely connected to WN , which is independent from sample mean and variance . WN ( Salimans & Kingma , 2016 ) is defined by υ w Tx ||w||2 that normalizes the weight vector w to have unit variance , where υ is a learnable parameter . Let each diagonal element of the covariance matrix of x be a and all the off-diagonal elements be zeros . h̄j in Eqn . ( 3 ) can be rewritten as h̄j = γ wTxj − µP σP + β = υ wTxj ||w||2 + b , ( 4 ) where υ = γa and b = − γµP a||w||2 + β . Eqn . ( 4 ) removes the estimations of statistics and eases our analyses of regularization for BN . • Third , µB and σB produce different strengths in ζ ( h ) . As shown in Eqn . ( 3 ) , the strength from µB depends on the expectation of σ ( h̄j ) ∈ [ 0 , 1 ] , which represents excitation or inhibition of a neuron , meaning that a neuron with larger output may exposure to larger regularization , encouraging different neurons to have equal magnitude . This is consistent with empirical result ( Morcos et al. , 2018 ) which prevented reliance on single neuron to improve generalization . The strength from σB works as a complement for µB . For a single neuron , I ( γ ) represents the norm of gradient , implying that BN punishes large gradient norm . For multiple neurons , I ( γ ) is the Fisher information matrix of γ , meaning that BN would penalize correlations among neurons . Both σB and µB are important , removing either one of them would imped performance . Extensions to Deep Networks . The above results can be extended to deep networks as shown in Appendix C.4 by decomposing the expected loss at a certain hidden layer . We also demonstrate the results empirically in Sec.5 , where we observe that CNNs trained with BN share similar traits of regularization as discussed above .

Summary: This is a thought provoking paper that aims to understand the regularization effects of batch-normalization (BN) under a probabilistic interpretation. The authors connect BN to population normalization (PN) and a gamma-decay term that penalizes the scale of the weights. They analyze the generalization error of BN for a single-layer perceptron using ideas in statistical physics.