Deconstructing the Regularization of BatchNorm

Batch normalization ( BatchNorm ) has become a standard technique in deep learning . Its popularity is in no small part due to its often positive effect on generalization . Despite this success , the regularization effect of the technique is still poorly understood . This study aims to decompose BatchNorm into separate mechanisms that are much simpler . We identify three effects of BatchNorm and assess their impact directly with ablations and interventions . Our experiments show that preventing explosive growth at the final layer at initialization and during training can recover a large part of BatchNorm ’ s generalization boost . This regularization mechanism can lift accuracy by 2.9 % for Resnet-50 on Imagenet without BatchNorm . We show it is linked to other methods like Dropout and recent initializations like Fixup . Surprisingly , this simple mechanism matches the improvement of 0.9 % of the more complex Dropout regularization for the state-of-the-art Efficientnet-B8 model on Imagenet . This demonstrates the underrated effectiveness of simple regularizations and sheds light on directions to further improve generalization for deep nets . 1 INTRODUCTION . Deep learning has made remarkable progress on a variety of domains in the last decade . While part of this progress relied on training larger models on larger datasets , it also depended crucially on the development of new training methods . A prominent example of such a development is batch normalization ( BatchNorm ) ( Ioffe and Szegedy , 2015 ) , which has become a standard component of training protocols . For example , state-of-the-art models in image recognition ( Szegedy et al. , 2017 ; He et al. , 2016 ; Tan and Le , 2019 ) , object detection ( He et al. , 2019 ; Du et al. , 2019 ) , and image segmentation ( Chen et al. , 2017 ) all use BatchNorm . Despite its prominence , the mechanisms behind BatchNorm ’ s effectiveness are not well-understood ( Santurkar et al. , 2018 ; Bjorck et al. , 2018 ; Yang et al. , 2019 ) . Perhaps at the core of the confusion is that BatchNorm has many effects . It has been correlated to reducing covariate shift ( Ioffe and Szegedy , 2015 ) , enabling higher learning rates ( Bjorck et al. , 2018 ) , improving initialization ( Zhang et al. , 2019 ) , and improving conditioning ( Desjardins et al. , 2015 ) , to name a few . These entangled effects make it difficult to properly study the technique . In this work , we deconstruct some of the effects of BatchNorm in search of much simpler components . The advantage of this approach compared to previous work is that it allows going beyond correlating these effects to BatchNorm by evaluating their impact separately . The mechanisms we consider in this work are purposefully simple . These simpler mechanisms are easier to understand and , surprisingly , they are competitive even at the level of the state-of-the-art . Our contributions can be summarized as follows : 1 . How does normalization help generalization ? We isolate and quantify the benefits of the different effects of BatchNorm using additive penalties and ablations . To our knowledge , we are the first to provide empirical evidence that BatchNorm ’ s effect of regularizing against explosive growth at initialization and during training can recover a large part of its generalization boost . Replicating this effect with Fixup initialization and the proposed additive penalty improves accuracy by 2.9 % for Resnet-50 without BatchNorm . 2 . Links to Fixup and Dropout . We draw novel connections between the regularization on the final layer , Dropout regularization , Fixup initialization and BatchNorm . 3 . Simplicity in regularization . The mechanism we identify can be useful as a standalone regularization . It produces 0.9 % improvement on the Efficientnet B8 architecture , matching the more complex Dropout regularization . 2 DECOMPOSING THE REGULARIZATION EFFECTS OF BATCH . NORMALIZATION Embedding Network Feature Embedding Input Embedding L2 Output Layer BatchNorm or Standardizing Loss Functional L2 Output Figure 1 : Diagram of a neural network showing where the mechanisms operate . In this section , we break BatchNorm into different mechanisms that can be studied separately . BatchNorm ( Ioffe and Szegedy , 2015 ) is a technique that accelerates training by standardizing the intermediate activations of a deep network . It achieves this standardization by using explicit normalization instead of relying on an additive regularization . While BatchNorm has been correlated to many effects , it is still unclear which effect , if any , explains most of its generalization boost . The effects we evaluate are the implicit regularizing effect on the norms at the final layer , and also its primary effect of standardizing the intermediate layers . In order to test these purposefully simple mechanisms we will rely on ablations and additive penalties . The use of additive penalties allows us to disentangle these effects where we control for the positive effect of BatchNorm on initialization by using the recently proposed Fixup initializer ( Zhang et al. , 2019 ) . 2.1 REGULARIZING AGAINST EXPLOSIVE GROWTH IN THE FINAL LAYER . First , we characterize the implicit effect of normalization on the final layer . Consider a neural network of the form NN ( x ) = WEmb ( x ) with loss L where x ∈ RI is the input of the network , W ∈ RK×H is the final weight matrix in the model , and Emb ( x ) : RI → RH is a feature embedding network with L layers . Let us take the common case where Emb ( x ) = Swish ( γBatchNorm ( PreEmb ( x ) ) + β ) where PreEmb ( x ) is the output of a residual network , Swish ( x ) = xσ ( ρx ) is the Swish activation ( Ramachandran et al. , 2017 ; Elfwing et al. , 2018 ) with scalar parameter ρ ( typically denoted β ) , and BatchNorm parameters γ , β. BatchNorm makes weight decay regularization on γ , β approximately equivalent to an additive penalty on the norm on the feature embedding L ( NN ( x ) ) + λ‖γ‖2 + λ‖β‖2 = L ( NN ( x ) ) + λ 4 E [ ‖Emb ( x ) ‖2 ] +O ( |ρ| ) . ( 1 ) See the Appendix A for the derivation . It means that the norm of the BatchNorm parameters alone is enough to directly control the norm of the feature embedding . It is a guarantee that the norm of the feature embedding can not grow explosively during training as long as these parameters are small . This regularization effect of BatchNorm can occur even without explicit weight decay due to the tendency of stochastic gradient descent to favor low norm parameters ( Wilson et al. , 2017 ) . This equivalency does not hold without BatchNorm because the activation of the embedding network become an important factor in the norm of the feature embedding ( ‖γ‖2 + ‖β‖2 6= E [ ‖γPreEmb ( x ) + β‖2 ] in general ) . Indeed , ( Balduzzi et al. , 2017 ; Gehring et al. , 2017 ; Zhang et al. , 2019 ) have shown that the activations of residual networks without BatchNorm tend explode exponentially in the depth of the network at initialization . This results in an extremely large embedding norm , even though the parameters are relatively small . We confirm experimentally in Section 4.3 that networks without BatchNorm have much larger feature embedding norms . Feature Embedding L2 ( EL2 ) We propose to assess the effect of this regularization mechanism by isolating it as the following additive penalty REL2 ( NN ) = 1 H E [ ‖Emb ( x ) ‖2 ] . ( 2 ) Adding this regularization to the loss allows us to test the impact of this mechanism independently on networks with and without BatchNorm . We dub this regularization embedding L2 for short in the following sections as it is L2 with a metric that is at the embedding network function level , with no additional penalties on the intermediate layers . It is applied right before the classification layer , in other words right after the final average pooling layer for residual networks . We will see with our experiments that this simple regularization can in large part recover the regularization boost of BatchNorm , has links to several known methods and is practically useful even at the level of the state-of-the-art . Functional L2 ( FL2 ) Regularizing the feature embedding has an implicit effect on the final output norm ( E [ ‖NN ( x ) ‖ ] ≤ 1/2 ‖W‖2 + H/2 · REL2 ( NN ) ) . In order to test the impact of this effect , we will also evaluate a direct penalty on the norm of the final output RFL2 ( NN ) = 1 K E [ ‖NN ( x ) ‖2 ] . ( 3 ) We dub this regularization functional L2 for short in the following sections as it is L2 with a metric that is at the full network function level . It is applied on the logits of the model for classification . In Section 4.4 , we investigate whether it is regularizing this norm or the feature embedding norm that is more closely correlated to better generalization . 2.2 STANDARDIZING THE INTERMEDIATE ACTIVATIONS OF THE MODEL . As a baseline mechanism , we will also consider an additive penalty that encourages the normalization of every intermediate layer called standardization loss ( Collins et al. , 2019 ) . This is a useful reference to gauge the effectiveness of the regularization on the embedding described in the previous sub-section . It also helps disentangle the side-effects of normalization . The penalty is DKL ( P ( x ) ||N ( x|0 , I ) ) = 1 2 ∑ i ( µ2i + σ 2 i − log σ2i − 1 ) ( 4 ) where µi is the mean and σi is the variance of an intermediate layer of the network . These are the same statistic that are computed by BatchNorm and a penalty is added to the loss for all the intermediate layers . This regularization has been considered in ( Collins et al. , 2019 ) . They found that it accelerated learning but fell significantly short of the generalization boost of batch normalization . However , they did not account for the positive effect of normalization on initialization . We correct for this in our experiments using the recently proposed Fixup initialization ( Zhang et al. , 2019 ) . 3 DRAWING LINKS TO OTHER METHODS . In this section , we will draw connections between the mechanisms considered for BatchNorm and other methods . 3.1 DROPOUT REGULARIZATION . Dropout ( Hinton et al. , 2012 ) is a regularization that prevents overfitting by randomly omitting subsets of features during training . Despite its early popularity , its use has declined with the rise of batch normalized convolutional networks . Ghiasi et al . ( 2018 ) , and Zagoruyko and Komodakis ( 2016 ) , find that it produces comparatively much smaller improvements when applied to the intermediate layers of such networks . However , Tan and Le ( 2019 ) has shown that state-of-the-art results can be obtained by applying Dropout only at the input of the final layer of the network . Interestingly , we can relate this particular use of Dropout to BatchNorm . Wang and Manning ( 2013 ) have shown that Dropout with MSE loss can be isolated as an additive regularization when applied at the last layer LDropout = 1 N ∑ E [ ( WDropout ( Emb ( xi ) ) − yi ) 2 ] = 1 N ∑ ( WEmb ( xi ) − yi ) 2 + λtr ( Wdiag ( E [ Emb ( x ) Emb ( x ) T ] ) WT ) where the additive Dropout regularization is RDropout ( NN ) = tr ( Wdiag ( E [ Emb ( x ) Emb ( x ) T ] ) WT ) and λ is the strength of the penalty . In this formulation , we can see that Dropout is related to the mechanisms in Section 2.1 as follows K ·RFL2 ( NN ) ≈ RDropout ( NN ) ≤ 1 4 ‖W‖4 + H 4 REL2 ( NN ) . ( 5 ) The approximate relationship to functional L2 relies on the assumption that the features are relatively decorrelated . To our knowledge this close but simple relationship between the expected norm of the output and Dropout with MSE loss had not been noted before . The upper bound with embedding L2 gives us a guarantee on Dropout robustness when training with BatchNorm and weight decay . In some sense , this means that networks with BatchNorm already incorporate a regularization effect similar to that conferred by Dropout . This can explain why networks with BatchNorm have tended to benefit relatively less from Dropout . The approximate relationship to functional L2 can be extended to networks with cross-entropy by using Taylor expansion and assuming the network has low-confidence . This assumption is likely only correct at initialization . In comparison , a related upper-bound can be found for embedding L2 using Taylor expansion . We will see in Section 4.5 that embedding and functional L2 can match or exceed Dropout for state-of-the-art architectures on Imagenet such as Efficientnet ( Tan and Le , 2019 ) . This is surprising because using an additive penalty at the final layer is much simpler than Dropout .

Summary: The paper empirically studies the regularization of BN. It proposes the point that the BN's effect is connected with the regularizing against explosive growth in the final layer. To motivate this point, it takes a single-layer case and shows the BN approximately penalizes on the norm of the feature embedding thereon. Two regularizations are proposed according to the point and are used to justify it.