Four Things Everyone Should Know to Improve Batch Normalization

1 INTRODUCTION . Neural networks have transformed machine learning , forming the backbone of models for tasks in computer vision , natural language processing , and robotics , among many other domains ( Krizhevsky & Hinton , 2009 ; He et al. , 2017 ; Levine et al. , 2016 ; Sutskever et al. , 2014 ; Graves et al. , 2013 ) . A key component of many neural networks is the use of normalization layers such as Batch Normalization ( Ioffe & Szegedy , 2015 ) , Group Normalization ( Wu & He , 2018 ) , or Layer Normalization ( Ba et al. , 2016 ) , with Batch Normalization the most commonly used for vision-based tasks . While the true reason why these methods work is still an active area of research ( Santurkar et al. , 2018 ) , normalization techniques typically serve the purpose of making neural networks more amenable to optimization , allowing the training of very deep networks without the use of careful initialization schemes ( Simonyan & Zisserman , 2015 ; Zhang et al. , 2019 ) , custom nonlinearities ( Klambauer et al. , 2017 ) , or other more complicated techniques ( Xiao et al. , 2018 ) . Even in situations where training without normalization layers is possible , their usage can still aid generalization ( Zhang et al. , 2019 ) . In short , normalization layers make neural networks train faster and generalize better . Despite this , it has been challenging to improve normalization layers . In the general case , a new approach would need to be uniformly better than existing normalization methods , which has proven difficult . It has even been difficult to tackle a simpler task : characterizing when specific changes to common normalization approaches might yield benefits . In all , this has created an environment where approaches such as Batch Normalization are still used as-is , unchanged since their creation . In this work we identify four techniques that everyone should know to improve their usage of Batch Normalization , arguably the most common method for normalization in neural networks . Taken together , these techniques apply in all circumstances in which Batch Normalization is currently used , ranging from large to very small batch sizes , including one method which is even useful when the batch size B = 1 , and for each technique we identify the circumstances under which it is expected to be of use . In summary , our contributions are : 1 . A way to more effectively use the current example during inference , fixing a discrepancy between training and inference that had been previously overlooked , 2 . Identifying Ghost Batch Normalization , a technique designed for very large-batch multiGPU training ( Hoffer et al. , 2017 ) , as surprisingly effective even in the medium-batch , single-GPU regime , 3 . Recognizing weight decay of the scaling and centering variables γ and β as a valuable source of regularization , an unstudied detail typically neglected , and 4 . Proposing a generalization of Batch and Group Normalization in the small-batch setting , effectively making use of cross-example information present in the minibatch even when such information is not enough for effective normalization on its own . Experimentally , we study the most common use-case of Batch Normalization , image classification , which is fundamental to most visual problems in machine learning . In total , these four techniques can have a surprisingly large effect , improving accuracy by over 6 % on one of our benchmark datasets while only changing the usage of Batch Normalization layers . We have released code at https : //github.com/ceciliaresearch/four_things_ batch_norm . 2 RELATED WORK/BACKGROUND ON NORMALIZATION METHODS . Most normalization approaches in neural networks , including Batch Normalization , have the general form of normalizing their inputs xi to have a learnable mean and standard deviation : x̂i = γ xi − µi√ σ2i + + β ( 1 ) where γ and β are the learnable parameters , typically initialized to 1 and 0 , respectively . Where approaches typically differ is in how the mean µi and variance σ2i are calculated . Batch Normalization ( Ioffe & Szegedy , 2015 ) , the pioneering work in normalization layers , defined µi and σ2i to be calculated for each channel or feature map separately across a minibatch of data . For example , in a convolutional layer , the mean and variance are computed across all spatial locations and training examples in a minibatch . During inference , these statistics are replaced with an exponential moving average of the mean and variance , making inference behavior independent of inference batch statistics . The effectiveness of Batch Normalization is undeniable , playing a key role in nearly all state-of-the-art convolutional neural networks since its discovery ( Szegedy et al. , 2016 ; 2017 ; He et al. , 2016a ; b ; Zoph & Le , 2017 ; Zoph et al. , 2018 ; Hu et al. , 2018 ; Howard et al. , 2017 ; Sandler et al. , 2018 ) . Despite this , there is still a fairly limited understanding of Batch Normalization ’ s efficacy — while Batch Normalization ’ s original motivation was to reduce internal covariate shift during training ( Ioffe & Szegedy , 2015 ) , recent work has instead proposed that its true effectiveness stems from making the optimization landscape smoother ( Santurkar et al. , 2018 ) . One weakness of Batch Normalization is its critical dependence on having a reasonably large batch size , due to the inherent approximation of estimating the mean and variance with a single batch of data . Several works propose methods without this limitation : Layer Normalization ( Ba et al. , 2016 ) , which has found use in many natural language processing tasks ( Vaswani et al. , 2017 ) , tackles this by calculating µi and σ2i over all channels , rather than normalizing each channel independently , but does not calculate statistics across examples in each batch . Instance Normalization ( Ulyanov et al. , 2016 ) , in contrast , only calculates µi and σ2i using the information present in each channel , relying on the content of each channel at different spatial locations to provide effective normalization statistics . Group Normalization ( Wu & He , 2018 ) generalizes Layer and Instance Normalization , calculating statistics in “ groups ” of channels , allowing for stronger normalization power than Instance Normalization , but still allowing for each channel to contribute significantly to the statistics used for its own normalization . The number of normalization groups per normalization layer is typically set to a global constant in group normalization , though alternatives such as specifying the number of channels per group have also been tried ( Wu & He , 2018 ) . Besides these most common approaches , many other forms of normalization also exist : Weight Normalization ( Salimans & Kingma , 2016 ) normalizes the weights of each layer instead of the inputs , parameterizing them in terms of a vector giving the direction of the weights and an explicit scale , which must be initialized very carefully . Decorrelated Batch Normalization ( Huang et al. , 2018 ) performs ZCA whitening in its normalization layer , and Iterative Normalization ( Huang et al. , 2019 ) makes it more efficient via a Newton iteration approach . Cho & Lee ( 2017 ) analyze the weights in Batch Normalization from the perspective of a Riemannian manifold , yielding new optimization and regularization methods that utilize the manifold ’ s geometry . Targeting the small batch problem , Batch Renormalization ( Ioffe , 2017 ) uses the moving average of batch statistics to normalize during training , parameterized in such a way that gradients still propagate through the minibatch mean and standard deviation , but introduces two new hyperparameters and still suffers somewhat diminished performance in the small-batch setting . Guo et al . ( 2018 ) tackle the small batch setting by aggregating normalization statistics over multiple forward passes . Recently , Switchable Normalization ( Luo et al. , 2019 ) aims to learn a more effective normalizer by calculating µi and σ2i as learned weighted combinations of the statistics computed from other normalization methods . While flexible , care must be taken for two reasons : First , as the parameters are learned differentiably , they are fundamentally aimed at minimizing the training loss , rather than improved generalization , which typical hyperparameters are optimized for on validation sets . Second , the choice of which normalizers to include in the weighted combination remains important , manifesting in Switchable Normalization ’ s somewhat worse performance than Group Normalization for small batch sizes . Differentiable Dynamic Normalization ( Luo et al. , 2019 ) fixes the latter point , learning an even more flexible normalization layer . Beyond these , there are many approaches we omit for lack of space ( Littwin & Wolf , 2018 ; Deecke et al. , 2019 ; Hoffer et al. , 2018 ; Klambauer et al. , 2017 ; Xiao et al. , 2018 ; Zhang et al. , 2019 ) . 3 IMPROVING NORMALIZATION : WHAT EVERYONE SHOULD KNOW . In this section we detail four methods for improving Batch Normalization . We also refer readers to the Appendix for a discussion of methods which do not improve normalization layers ( sometimes surprisingly so ) . For clarity , we choose to interleave descriptions of the methods with experimental results , which aids in understanding each of the approaches as they are presented . We experiment with four standard image-centric datasets in this section : CIFAR-100 , SVHN , Caltech-256 , and ImageNet , and report results on validation datasets in order to fully describe each approach without contaminating test-set results . We give results on test sets , and experimental details in Sec . 4 . 3.1 INFERENCE EXAMPLE WEIGHING . Batch Normalization has a disparity in function between training and inference : As previously noted , Batch Normalization calculates its normalization statistics over each minibatch of data separately while training , but during inference a moving average of training statistics is used , simulating the expected value of the normalization statistics . Resolving this disparity is a common theme among methods that have sought to replace Batch Normalization ( Ba et al. , 2016 ; Ulyanov et al. , 2016 ; Salimans & Kingma , 2016 ; Wu & He , 2018 ; Ioffe , 2017 ) . Here we identify a key component of this training versus inference disparity which can be fixed within the context of Batch Normalization itself , improving it in the general case : when using a moving average during inference , each example does not contribute to its own normalization statistics . To give an example of the effect this has , we consider the output range of Batch Normalization . During training , due to the inclusion of each example in its own normalization statistics , it can be shown1 that the minimum possible output of a Batch Normalization layer is : min x0 , ... , xB−1 γ x0 − µi√ σ2i + + β = −γ √ B − 1 + β ( 2 ) with a corresponding maximum value of γ √ B − 1+β , whereB is the batch size , and we assume for simplicity that Batch Norm is being applied non-convolutionally . In contrast , during inference the output range of Batch Normalization is unbounded , creating a discrepancy . Morever , this actually happens for real networks : the output range of a network with Batch Normalization is wider during inference than during training ( see Sec . B in Appendix ) . 1Proof in Appendix Sec . A Fortunately , once this problem has been realized , it is possible to fix — we need only figure out how to incorporate example statistics during inference . Denoting mx as the moving average over x and mx2 the corresponding moving average over x2 , we apply the following normalization : µi = αE [ xi ] + ( 1− α ) mx σ2i = ( αE [ x 2 i ] + ( 1− α ) mx2 ) − µ2i x̂i = γ xi − µi√ σ2i + + β ( 3 ) where α is the contribution of xi to the normalization statistics , and we have reparameterized the variance as σ2i = E [ x 2 i ] − E [ xi ] 2 . Given this formulation , a natural question is the choice of the parameter α , where α = 0 corresponds to the classical inference setting of Batch Normalization and α = 1 replicates the setting of techniques which do not use cross-image information in calculating normalization statistics . Intuitively , it would make sense for the optimal value to be α = 1B . However , this turns out to not be the case — instead , α is a hyperparameter best optimized on a validation set , whose optimal value may depend the model , dataset , and metric being optimized . While counterintuitive , this can be explained by the remaining set of differences between training and inference : for a basic yet fundamental example , the fact that the model has been fit on the training set ( also typically with data augmentation ) may produce systematically different normalization statistics between training and inference . An advantage of this technique is that we can apply it retroactively to any model trained with Batch Normalization , allowing us to verify its efficacy on a wide variety of models . In Fig . 1 we show the effect of α on the ImageNet ILSVRC 2012 validation set ( Russakovsky et al. , 2015 ) for three diverse models : ResNet-152 ( He et al. , 2016b ) , MobileNetV2 ( Sandler et al. , 2018 ) , and NASNet-A Large ( Zoph et al. , 2018 ) 2 . On ResNet-152 , for example , proper setting of α can increase accuracy by up to 0.6 % , top-5 accuracy by 0.16 % , and loss by a relative 4.7 % , which are all quite significant given the simplicity of the approach , the competitiveness of ImageNet as a benchmark , and the fact that the improvement is essentially “ free ” — it involves only modifying the inference behavior of Batch Normalization layers , and does not require any re-training . Across models , the optimal value for α was largest for NASNet-A , the most memory-intensive ( and therefore smallest batch size ) model of the three . We refer the reader to the Appendix for additional plots with larger ranges of α . Surprisingly , it turns out that this approach can have positive effects on models trained without any cross-image normalization at all , such as models trained with Group Normalization ( Wu & He , 2018 ) . We demonstrate this in Fig . 2 , where we find that adding a tiny amount of information from the moving average statistics can actually result in small improvements , with relatively larger improvements in accuracy on Caltech-256 and cross entropy loss on CIFAR-100 and SVHN . This finding is extremely surprising , since adding in any information from the moving averages at all represents a clear difference from the training setting of Group Normalization . Similar to the unintuitive optimal value for α , we hypothesize that this effect is due to other differences in the settings of training and inference : for example , models are generally trained on images with the application of data augmentation , such as random cropping . During inference , though , images appear unperturbed , and it might be the case that incorporating information from the moving averages is a way of influencing the model ’ s intermediate activations to be more similar to those of data augmented 2Models obtained from ( Silberman & Guadarrama ) . images , which it has been trained on . This mysterious behavior may also point to more general approaches for resolving training-inference discrepancies , and is worthy of further study . Last , we also note very recent work ( Singh & Shrivastava , 2019 ) which examines a similar approach for incorporating the statistics of an example during inference time , using per-layer weights and optimizing with a more involved procedure that encourages similar outputs to the training distribution . Summary : Inference example weighing resolves one disparity between training and inference for Batch Normalization , is uniformly beneficial across all models and very easy to tune to metrics of interest , and can be used with any model trained with Batch Normalization , even retroactively .

Summary: The paper performs an empirical study of four batch-normalization improvements and proposes a new normalization technique for small batch sizes, based on group and batch normalizations. Among others, the authors address the inconsistency between the train and the test stages and the problem of small batch sizes. The authors conducted an empirical ablation study of the four techniques and proposed an intuition when each method should be used.