Four Things Everyone Should Know to Improve Batch Normalization
Authors: Cecilia Summers
Michael J. Dinneen
Venue:  (2020)
Keywords: batch normalization

Abstract: A key component of most neural network architectures is the use of normalization layers, such as Batch Normalization. Despite its common use and large utility in optimizing deep architectures, it has been challenging both to generically improve upon Batch Normalization and to understand the circumstances that lend themselves to other enhancements. In this paper, we identify four improvements to the generic form of Batch Normalization and the circumstances under which they work, yielding performance gains across all batch sizes while requiring no additional computation during training. These contributions include proposing a method for reasoning about the current example in inference normalization statistics, fixing a training vs. inference discrepancy; recognizing and validating the powerful regularization effect of Ghost Batch Normalization for small and medium batch sizes; examining the effect of weight decay regularization on the scaling and shifting parameters γ and β; and identifying a new normalization algorithm for very small batch sizes by combining the strengths of Batch and Group Normalization. We validate our results empirically on six datasets: CIFAR-100, SVHN, Caltech-256, Oxford Flowers-102, CUB-2011, and ImageNet.

Published as a conference paper at ICLR 2020
FOUR THINGS EVERYONE SHOULD KNOW TO IMPROVE BATCH NORMALIZATION
Cecilia Summers
Michael J. Dinneen
Department of Computer Science University of Auckland ceci lia a summers. 07 gma i...i.... com com
Department of Computer Science
University of Auckland mjdē cs auckland ac.nz
ABSTRACT
A key component of most neural network architectures is the use of normalization layers, such as Batch Normalization. Despite its common use and large utility in optimizing deep architectures, it has been challenging both to generically im- prove upon Batch Normalization and to understand the circumstances that lend themselves other enhancements. In this paper, identify four improvements to the generic form of Batch Normalization and the circumstances under which they work, yielding performance gains across all batch sizes while requiring additional computation during training. These contributions include proposing method for reasoning about the current example in inference normalization statistics, fixing training vs. inference discrepancy; recognizing and validating the powerful regularization effect of Ghost Batch Normalization for small and medium batch sizes; examining the effect of weight decay regularization on the scaling and shifting parameters 7 and 8:; and identifying a new normalization al- gorithm for very small batch sizes by combining the strengths of Batch and Group Normalization. We validate our results empirically on six datasets: CIFAR-100, SVHN, Caltech-256, Oxford Flowers-103 CUB-2011, and ImageNet.
INTRODUCTION
Neural networks have transformed machine learning, forming the backbone of models for tasks in
computer vision, natural language processing, and robotics, among many other domains (Krizhevsky
& Hinton 2009 He et al.) 2017 Levine et al.) 2016, (Sutskever et al. 2014 Graves et al.) 2013). A
key component of many neural networks is the use of normalization layers such as Batch Normaliza- tion (Ioffe & Szegedy 2015), Group Normalization (Wu & He 2018), or Layer Normalization (Ba
ett al., 2016), with Batch Normalization the most commonly used for vision-basee tasks. While the true reason why these methods work is still active area of research (Santurkar et al.) 2018))
normalization techniques typically serve the purpose of making neural networks more amenable optimization, allowing the training of very deep networks without the use of careful initializa- tion schemes (Simonyan & Zisserman, 2015 Zhang et al., 2019), custom nonlincarities (Klambauer al. 2017), other more complicated techniques (Xiao al. 2018). Even in situations where Tramiig without normalization layers is possible, their usage can still aid generalization (Zhang al. 2019). In short, normalization layers make neural networks train faster and generalize better.
Despite this, it has been challenging to improve normalization layers. In the general case, new approach would need to be uniformly better than existing normalization methods, which has proven difficult. It has even been difficult to tackle a simpler task: characterizing when specific changes to common normalization approaches might yield benefits. In all, this has created an environment where approaches such Batch Normalization are still used as-is, unchanged since their creation.
In this work we identify four techniques that everyone should know to improve their usage of Batch Normalization, arguably the most common method for normalization in neural networks. Taken together, these techniques apply in all circumstances in which Batch Normalization is currently used, ranging from large to very small batch sizes, including one method which is useful when the batch size = 1, and for each technique we identify the circumstances under which it is expected to be of use. In summary, contributions are:

---

Published as a conference paper at ICLR 2020
1. A way to more effectively use the current example during inference, fixing discrepancy between training and inference that had been previously overlooked,
2. Identifying Ghost Batch Normalization, a technique designed for very large-batch multi- GPU training (Hoffer et al. 2017], as surprisingly effective even in the medium-batchh, single-GPU regime,
3. Recognizing weight decay of the scaling and centering variables y and | valuable source of regularization, an unstudied detail typically neglected, and
4. Proposing a generalization of Batch and Group Normalization in the small- batch setting, effectively making use of cross- eros-eexample information present in the minibatch even when such information is not enough for effective normalization on its own.
Experimentally, we study the most common use-case of Batch Normalization, image classification, which is fundamental to most visual problems in machine learning. In total, these four techniques can have a surprisingly large effect, improving accuracy by over 6% on one of our benchmark datasets while only changing the usage of Batch Normalization layers.
We have released code at atttos//ggtthacccom///eellaaaarr batch_norm
2 RELATED WORK/BACKGROUND ON NORMALIZATION METHODS
Most normalization approaches in neural networks, including Batch Normalization, have the general form of normalizing their inputs r, to have learnable mean and standard deviation:
T; P4
2
=B
(I)
Vo? +
where 7 and B are the learnable parameters, typically initialized to 1 and O, respectively. Where approaches typically differ is in how the mean μæ and variance of are calculated.
Batch Normalization (Toffe & Szegedy, 2015), the pioneering work in normalization layers, defined μ4 and of to be calculated for each channel or feature map separately across a minibatch of data. For example, in a convolutional layer, the mean and variance are computed across all spatial lo- cations and training examples in a minibatch. During inference, these statisties are replaced with an exponential moving average of the mean and variance, making inference behavior independent of inference batch statistics. The effectiveness of Batch Normalization undeniable, playing a key role in nearly all state-of-the-arr convolutional neural networks since its discovery (Szegedy (ct al. 2016 2017 He et al.) 2016a bj Zoph & Le 2017 Zoph et al. 2018 Hu et al. 2018 Howard ct al. 2017: Sandler al. (018). Despite this, there is still a lairly limited understanding of Batch while Batch Normalization 's original motivation was to reduce internal covariate shift during training (loffe & Szegedy 2015), recent work has instead proposed that its true effectiveness stems from making the optimization landscape smoother (Santurarr et al. 2018).
Normalization's 's efficacy
One weakness of Batch Normalization is its critical dependence having a reasonably large hatch size, due to the inherent approximation of estimating the mean and variance with a single batch of data. Several works propose methods without this limitation: Layer Normalization (Ba al. 2016), which has found use in many natural language processing tasks (Vaswani et al. 20117). tackles this by calculating μ4 and of over all channels, rather than normalizing each channel independently, but does not calculate statistics across examples in each batch. Instance Normalization (Ulyanov et al., 2016). in contrast, only calculates μ4 and of using the information present in each channel, rely- ing on the content of each channel at different spatial locations to provide effective normalization statistics. Group Normalization (Wu & He 2018) generalizes Layer and Instance Normalization, calculating statistics in "groups" of channels, allowing for stronger normalization power than In- stance Normalization, but still allowing for each channel to contribute significantly to the statistics used for its own normalization. The number of normalization groups per normalization layer is typically set to a global constant in group normalization, though alternatives such as specifying the number of channels per group have also been tried (Wu & He 2018).
Besides these most common approaches, many other forms of normalization also exist: Weight Nor- malization (Salimans & Kingma 2016) normalizes the weights of each layer instead of the inputs,
2

---

Published as a conference paper at ICLR 2020
parameterizing them in terms of vector giving the direction of the weights and an explicit scale, which must be initialized very carefully. Decorrelated Batch Normalization (Huang et al. 2018) per- forms ZCA whitening in its normalization layer, and Iterative Normalization (Huang et al. 2019) makes it more efficient via a Newton iteration approach. Cho & Lee (2017) analyze the weights in Batch Normalization from the perspective of Riemannian manifold, yielding new optimization and regularization methods that utilize the manifold's geometry.
Targeting the small batch problem, Batch Renormalization (Toffe, 2017) uses the moving average of batch statistics to normalize during training, parameterized in such a way that gradients still propa- gate through the minibatch mean and standard deviation, but introduces two new hyperparameters and still suffers somewhat diminished performance in the small-batch setting. [uo et al. (2018) tackle the small batch setting by aggregating normalization statistics over multiple forward passes.
Recently, Switchable Normalization (Luo et al. 2019) aims to learn more effective normalizer by calculating 14; and of as learned weighted combinations of the statistics computed from other normalization methods. While flexible, care must be taken for two reasons: First, the parame- ters are learned differentiably, they are fundamentally aimed at minimizing the training loss, rather than improved generalization, which typical hyperparameters are optimized for validation sets. Second, the choice of which normalizers to include in the weighted combination remains important, manifesting in Switchable Normalization's 's somewhat worse performance than Group Normalization for small batch sizes. Differentiable Dynamic Normalization (Luo et al., 2019) fixes the latter point, learning even more flexible normalization layer. Beyond these, there are many approaches omit for lack of space (Littwin & Wolf 2018 Deecke et al. 2019: Hoffer et al., 2018 Klambauer al. 2017 Xiao al. 2018 Zhang et al. 2019))
IMPROVING NORMALIZATION: WHAT EVERYONE SHOULD KNOW
In this section we detail four methods for improving Batch Normalization. We also refer readers to the Appendix for a discussion of methods which do not improve normalization layers (sometimes surprisingly so). For clarity, we choose to interleave descriptions of the methods with experimental results, which aids in understanding each of the approaches as they are presented. We experiment with four standard image-centric datasets in this section: CIFAR-100, SVHN, Caltech-256, and ImageNet, and report results on validation datasets in order to fully describe each approach without contaminating test-set results. We give results on test sets, and experimental details in Sec.
3.1 INFERENCE EXAMPLE WEIGHING
Batch Normalization has disparity in function between training and inference: As previously noted, Batch Normalization calculates its normalization statisties over each minibatch of data separately while training, but during inference a moving average of training statistics is used, simulating the expected value of the normalization statistics. Resolving this disparity is a common theme among methods that have sought to replace Batch Normalization (Ba et al., 2016 Ulyanov et al.) 2016 Salimans & Kingma| 2016 Wu & He, 2018 [offe 2017). Here we identify key component of this training versus inference disparity which can be fixed within the context of Batch Normalization itself, improving it in the general case: when using a moving average during inference, each example does not contribute to its own normalization statistics.
To give an example of the effect this has, we consider the output range of Batch Normalization. During training, due to the inclusion of each example in its own normalization statistics, it can be showr" that the minimum possible output of a Batch Normalization layer is:
min
To- Ms
+ 3 -VB 1+
(2)
Vo
with corresponding maximum value of vv BB where B is the batch size, and we assume for simplicity that Batch Norm is being applied non-convolutionally. In contrast, during inference the output range of Batch Normalization unbounded, creating a discrepancy. Morever, this actually happens for real networks: the output range of network with Batch Normalization is wider during inference than during training (see Sec. B in Appendix).
Proof in Appendix Sec.
3

---

Published as a conference paper at ICLR 2020
ResNet-112
MobileNet22
NasNet-A
92.
70.4
Top- Atc.
LOSS
14.0
$2.00
$0.00
0.50 77.
5:0 (1.7
5.0 0:31
336
D/36
60.75
7248
2.00:
5.04
5017 0006 0.52
06.10
32.
77000 DIU
Figure 1: Effect of the example-weighing hyperparameter on ImageNet for ResNet-152. 152, Mo- bileNetV2, and NASNet-A, measuring top-1 I and top-5 accuracies and the cross-entropy loss.
Fortunately, once this problem has been realized, it possible to fix - we need only figure out how to incorporate example statistics during inference. Denoting my as the moving average over r and m2a the corresponding moving average over x², we apply the following normalization:
#4 ab' [z + (1 a)mz
of [aE(2; (1 a)mma)) -1
(3)
& = 24 Mi +
Vo?
where a is the contribution of x; to the normalization statistics, and we have reparameterized the variance of = El E|x;]] E|x,1²
Given this formulation, a natural question is the choice of the parameter a, where a = corresponds to the classical inference setting of Batch Normalization and niques which do not use cross-image information in calculating normalization statistics. Intuitively, it would make sense for the optimal value be B. However, this turns out not be the case instead, ar is a hyperparameter best optimized on a validation set, whose optimal value may depend the model, dataset, and metric being optimized. While counterintuitive, this can be explained by the remaining set of differences between training and inference: for basic yet fundamental example, the fact that the model has been fit on the training set (also typically with data augmentation) may produce systematically different normalization statistics between training and inference.
replicates the setting of tech-
An advantage of this technique is that we can apply retroactively to any model trained with Batch Normalization, allowing us to verify its efficacy on wide variety of models. In Fig. [ we show the effect of a on the ImageNet VLS 2012 validation set (Russakovsky et al., 2015) for three diverse models: ResNet- 152 (He et al. 2016b), MobileNetV2 (Sandler et al. 2018). and NASNet-A Large (Zoph et al. 2018 On ResNet-152, for example, proper setting of a can increase accuracy by up to 0.6%. 5 accuracy by 0.16%, and loss by relative 4.7%, which are all quite significant given the simplicity of the approach, the competitiveness of ImageNet as a benchmark, and the fact that the improvement is essentially "free" it involves only modifying the inference behavior of Batch Normalization layers, and does not require any re-training. Across models, the optimal value for was largest for NASNet-A, the most memory-intensive (and therefore smallest batch size) model of the three. We refer the reader to the Appendix for additional plots with larger ranges of a.
Surprisingly, it turns out that this approach can have positive effects models trained without any cross-image normalization at all, such as models trained with Group Normalization (Wu & He 2018 We demonstrate this in Fig. 2 where we find that adding a tiny amount of information from the moving average statistics can actually result in small improvements, with relatively larger improvements in accuracy on C56 and cross entropy loss on CIFAR- -EFAR--10 and SVHN. This finding is extremely surprising, since adding in any information from the moving averages at all represents a clear difference from the training setting of Group Normalization. Similar to the unin- tuitive optimal value for a. we hypothesize that this effect is due to other differences in the settings of training and inference: for example, models are generally trained on images with the application of data augmentation, such random cropping. During inference, though, images appear unper- turbed, and it might be the case that incorporating information from the moving averages is way of influencing the model's intermediate activations to be more similar to those of data augmented
Models obtained from (Silherman & Guadarrama
4

---

Published as a conference paper at ICLR 2020
CIFAR-100 100 with Group Normalization,
SVHN with Group Normalzation
Caltech with Group Normalizaion
12.4
50.00
6.04
0.70
Figure 2: Effect of the example-weighing hyperparameter for models trained with Group Nor- malization on CIFAR-100, 100, SVHN, and Caltech-256.
images, which it has been trained on. This mysterious behavior may also point to more general approaches for resolving training-infee inference discrepancies, and worthy further study.
Last, also note very recent work (Singh & Shrivastava, 2019) which examines similar approach for incorporating the statistics of an example during inference time, using per-layer weights and op- timizing with more involved procedure that encourages similar outputs the training distribution.
Inference example weighing resolves one disparity between training and inference for Batch Normalization, is uniformly beneficial across all models and very easy to tune to metrics of interest, and can be used with any model trained with Batch Normalization, even retroactively.
Summary:
3.2 GHOST BATCH NORMALIZATION FOR MEDIUM BATCH SIZES
Ghost Batch Normalization, a technique originally developed for training with very large batch sizes across many accelerators (Hoffer al. 2017). consists of calculating normalization statisties on dis- joint subsets of each training batch. Concretely, with overall batch size of B and "ghost" batch size such that B' evenly divides B, the normalization statistics for example i are calculated as
B
iB'
iB'
Σ
2
B'
B
B
j=1
(4)
B
jB'
iB'
of B"
Σ 3=1
1
B
B
where is the Iverson bracket, with value if its argument is true and otherwise. Ghost Batch Normalization was previously found to be an important factor in reducing the generalization gap between large-batch and small-batch models (Hoffer al., 2017). and has since been used by sub- sequent research rigorously studying the large-batch regime (Shallue et al., 2018). Here, we show that it can also useful in the medium-batch setting
Why might Ghost Batch Normalization be useful? One reason is its power as regularizer: due to the stochasticity in normalization statistics caused by the random selection of minibatches during training. Batch Normalization causes the representation of a training example to randomly change every time appears different batch of data. Ghost Batch Normalization, by decreasing the number of examples that the normalization statistics calculated over, increases the strength of this stochasticity, thereby increasing the amount regularization. Based on this hypothesis, we would expect to see unimodal effect of the Ghost Batch Normalization size B' on model performance a large value of B' would offer somewhat diminished performance as a weaker regularizer, a very low value of B' would have excess regularization and lead to poor performance, and an intermediate value would offer the best tradeoff of regularization strength.
We confirm this intuition in Fig. Surprisingly, just using this simple technique was capable of improving performance by 5.8% on Caltech-256 and 0.84% on CIFAR-100, which is remarkable given has no additional cost during training. On SVHN, though, where baseline performance is already very high 98. 79% and models do not overfit much, usage of Ghost Batch Normalization
We experiment with hatch sizes up to 128 in this work.

---

Published as a conference paper at ICLR 2020
CIFAR-100
SVHN
50,000
57.5
Accuracy 15
06.750
Accuracy
72.
47.5
08.815
45.0
60.050
226
Figure 3: Accuracy vs. Ghost Batch Normalization size for CIFAR-100, SVHN, and Caltech-256.
CIFAR-100
Ghost Barch
Ghost Batch Size
Ghost Batch Size
Ciosst Batch Size
735
130
734
117 A11
Caltech-256
Chose Baich Soee
Ghadd
Chaed
313
633 114
322
SVHN
Dheek Buich Suot
Clood betch Suc
Clool Batch Siee
Chool Buich Sker 10
14:00
Figure 4: The complementary effects of Inference Example Weighing (Sec. 3.1 and Ghost Batch Normalization (Sec, 3.2) on CIFAR-100, SVHN, and Caltech-256.
did not result in improvement, giving evidence that least part of its effect is regularization in nature. In practice, B' may be treated as an additional hyperparameter to optimize.
As bonus, Ghost Batch Normalization has a synergistic effect with inference example weighing - it has the effect of making each example more important in calculating its own normalization statisties μ4 and of with greater effect the smaller B' is, precisely the setting that inference example weighing corrects for. We show these results in Fig. 4 where we find increasing gain from inference example weighing B' is made smaller. a gain that compounds from the benefits of Ghost Batch Normalization itself. Interestingly, these examples also demonstrate that accuracy and cross-entroopy the most commonly classification loss, are only partially correlated, with the optimal values for the inference example weight sometimes differing wildly between the two (e.g. for SVHN).
Summary: Ghost Batch Normalization is beneficial for all but the smallest of batch sizes, has no computational overhead, is straightforward to tune, and can be used in combination with inference example weighing to great effect.

---

Published as a conference paper at ICLR 2020
3.3 BATCH NORMALIZATION AND WEIGHT DECAY
Weight decay (Krogh & Hertz 1992) is a regularization technique that scales the weight of a neural network after each update step by a factor of ō, and has complex interaction with Batch Normalization. At first, it may even seem paradoxical that weight decay has any effect in a network trained with Batch Normalization, as scaling the weights immediately before normalization layer by any non-zero constant has mathematically almost no effect on the output of the normalization layer (and no effect at all when e = 0). However, weight decay actually has a subtle effect on the effective learning rate of networks trained with Batch Normalization weights in batch-normalized network grow to have large magnitudes, which has an inverse effect on the effective learning rate, hampering training (Hoffer et al. 2018 van Laarhoven 2017).
without weight decay, the
Here we turn our attention to the less studied scale and bias parameters common in most normal- ization methods, 7 and B. As far as
aware, the effect of regularization on y and 8 has not been studied to any great extent - -Wu & He (2018) briefly mention weight decay with these pa- rameters, where weight decay was used when training from scratch, but not fine-tuning, two other papers (Goyal et al. 2017; He et al. 2016a) have this form of weight decay explicitly turned off, and He et al. (2019) encourage disabling weight decay on y and $, but ultimately find diminished performance by doing so.
Unlike weight decay on weights in e.g. convolutional layers, which typically directly precede nor- malization layers, weight decay on y and |3 can have a regularization effect so long as there is a path in the network between the layer in question and the ultimate output of the network, if such paths do not pass through another normalization layer, then the weight decay is never "undone" by normalization. This structure is only common in certain types of architectures; for example, Resid- ual Networks (He et al.) 201 6a b) have such paths for many of their normalization layers due to the chaining of skip-connections. However, Inception-styl -stlle networks (Szegedy al., 2016) (017) have no residual connections. and despite the fact that each "Inception block" branches into multiple paths, every Batch Normalization layer other than those in the very last block do not have direct path to the network's 's output.
We evaluated the effects of weight decay on y and 3 on CIFAR- across 10 runs, where we found that incorporating it improved accuracy by a small but significant 0.3% (P = 0.002). Interestingly, even though y has a multiplicative effect, we did not find it mattered whether r was regularized to 0 or 1 (P = 0.46) what important whether it had weight decay applied at all.
We did the same comparison on Caltech-256 with Inception-v3 and ResNet-50 networks, where found evidence that the network architecture plays crucial effect: for Inception-v3, incorpo- rating weight decay on 7 and B actually hurt performance by 0. 0.13% (mean across 3 trials), while improved performance for the ResNet-50 network by 0.91%. supporting the hypothesis that the structure of paths between layers and the network's 's output are what matter in determining its utility.
On SVHN, where the baseline R8 already had a performance of 99.79%%. found a similar pattern as with Ghost Batch Normalization - introducing this regularization produced no change.
Summary:
Regularization in the form of weight decay on the normalization parameters y and I can be applied to any normalization layer, but is only effective in architectures with particular connectivity properties like ResNets and tasks for which models are already overfitting.
3.4 GENERALIZING BATCH AND GROUP NORMAL.ZZATION FOR SMALL BATCHES
While Batch Normalization is very effective in the medium to large-batch setting, it still suffers when not enough examples are available to calculate reliable normalization statistics. Although we have shown that techniques such Inference Example Weighing (Sec. 3.1) can help significantly with this, it is still only partial solution. At the same time, Group Normalization (Wu & He 2018 was designed for a batch size of B 1 or greater, but ignores all cross-image information.
In order to generalize Batch and Group Normalization in the batch size B > 1 case, we propose to expand the grouping mechanism of Group Normalization from being over only channels to being over both channels and examples- - that is, normalization statistics are calculated both within groups of channels of each example and across examples in groups within each batch
See submitted code for specific implementation details.
7

---

Published as a conference paper at ICLR 2020
In principle, this would appear to introduce additional hyperparameter top of the number of channel groups used by Group Normalization, both of which would need to be optimized by expensive end-to-end runs of model training. However, in this case we can actually take advantage of the fact that the target batch size is small: if the batch size B is ever large enough that having multiple groups in the example dimension is useful, then it is also large enough to eschew usage of the channel groups from Group Normalization, in a regime where either vanilla Batch Normalization or Ghost Batch Normalization more effective. Thus, when dealing with a small batch size, in practice we only need to optimize over the same set of hyperparameters as Group Normalization.
To demonstrate, we target the extreme setting of 2, and incorporate Inference Example Weigh- ing to all approaches. For CIFAR-I00, this approach improves validation set performance over a tuned Group Normalization by 0.69% top-1 accuracy (from 73.91% to 74.60% average over three runs), and on Caltech-256, performance dramatically improved by 5.0% (from 48.2% to 53.2%, av- erage over two runs). However, this approach has one downside: due to differences in feature statistics across examples, when using only two examples the variability in the normalization statis- ties can still be quite high, even when using multiple channels within each normalization group. As result, regularization effect can occur, which may be undesirable for tasks which models not overfitting much. As in Sec. [.2 and Sec. 3.3 we see this effect in SVHN where this approach is actually ever slightly worse than Group Normalization on the validation set (from 98. 78.75% to 98.73%). On such datasets and tasks, it may be more fruitful to invest in higher-capacity models.
Summary: Combining Group and Batch Normalization leads to more accurate models in the set- ting of batch sizes B > 1, and can have regularization effect due to Batch Normalization's vari- ability in statistics when calculated over small batch sizes.
ADDITIONAL EXPERIMENTS
4.1 EXPERIMENTAL DETAILS
All results Sec. were performed on the validation datasets of each respective dataset (this section examines test set performance after hyperparameters have been optimized). Of the six datasets experiment with, only ImageNet (Russakovsky et al. 2015) and Flowers- (Nilsback & Zisser- man 2008) have their own pre-defined validation split, so constructed validation splits for the other datasets as follows: for CIFAR-106 (Krizhevsky & Hinton, 2009), we randomly took 40,000 of the 50,000 training images for the training split, and the remaining 10,000 validation split. For SVHN (Netzer et al. 2011), we similarly split the 604,388 non-test images in a 80-20% split for training and validation. For Caltech-256, no canonical splits of any form defined, so we used 40 images of each of the 256 categories for training. 10 images for validation, and 30 for testing. For CUB-2011, we used 25% of the given training data as validation set.
The model used for CIFAR-100 and SVHN was ResNet-18 18 (He et al., 2016b a) with 64, 128, 256, and 512 filters across blocks. For Caltech-256, a much larger Inception- v3 (Szegedy et al., 2016) model was used, and we additionally experiment with ResNet- 152 (He et (016b)) on Flowers- 102 and CUB-2011 I in Sec. 4.3 All experiments were done on two Nvidia Geforce GTX 1080 Ti GPUs.
4.2 COMBINING ALL FOUR: IMPROVEMENTS ACROSS BATCH SIZES
Here we show the end-to-end effect of these four improvements on the test sets of each dataset, comparing against both Batch and Group Normalization, with a batch size B 128. We plot results for 100 and Caltech-256 in Fig. 5 (a), comparing against Group Normalization and an idealized Batch Normalization with constant performance across batch sizes (simulating the problematic dependence of Batch Norm on the batch size were completely solved). On CIFAR-100, we see improvements against the best available baseline across all batch sizes.
For medium to large batch sizes (B > 4), improvements are driven by the combination of Ghost Batch Normalization (Sec. 3.2). Inference Example Weighing (Sec. 3.I. and weight decay intro- duced y and / (Sec. 3.3) To aid in distinguishing between these effects,
also plot the impact of Ghost Batch Normalization alone, which we find particularly impactful as long as long as the batch size isn't too small (B > 2). Turning very small batch sizes, for B improvements
8

---

Published as a conference paper at ICLR 2020
CIFAR-100
Caltech-256
(a)
70.3
Accuracy
27.0
76.5
12
16.0
10-
15.3
15.0
Drspp Soceral oohee
14.
08x2 Maresiiattoon
Blard Morred otle
16.0
Batch
Batch
(b)
Flowers-102
CUB-2011
94-
Accuracy
02
20
70
08-
06
Boton Morralitter DAUE)
Broce Kerroirrtte
Broup Sccrudlateer
84
02
75
04
04
Batch size
Batch size
Total performance changes across batch sizes for CIFAR-100 and Caltech-256 (a) when training from scratch, incorporating all proposed improvements Batch Normalization. On the bot- tom (b) is the same on Flowers- 102 and CUB-2011 I, which employs transfer learning via fine-tuning from ImageNet. Also shown within each plot is the performance of Group Normalization, an ide- alized Batch Normalization that scales perfectly across batch sizes, and Ghost Batch Normalization (Sec. 3.2 by itself, for which the x-axis represents the Ghost Batch Size B'.
Figure 5:
are due to the introduced weight decay, and for B = 2 the generalization of Batch and Group Normalization leads to the improvement (Sec. 3.4.. with some additional effect from weight decay.
Improvements Caltech-256 follow the same trends, but to greater magnitude, with a total increase in performance of 6.5% over Batch Normalization and an increase 5.9% over Group Normaliza- tion for B 2.
4.3 TRANSFER LEARNING
We also show the applicability these approaches in the context of transfer learning, which we demonstrate the Flowers- 102 (Nilsback & Zisserman 2008) and CUB-2011 (Wah et al., 2011) datasets via fine-tuning a ResNet-152 model from ImageNet. These tasks presents several chal- lenges: 1) the Flowers-102 data only contains 10 images per category in the training set (and CUB- 2011 only 30 examples per class), 2) pre-training models on ImageNet is very strong form of prior knowledge, and despite the small dataset size may heavily reduce the regularization effects of some of the techniques, and 3) we examine the setting of pre-training with generic ImageNet mod- els trained without any of these modifications, which gives an advantage to both the generic Batch Normalization and Group Normalization, for which pre- pre-trained models exist.
We plot results in Fig. (b), where we find remarkable qualitative agreement of our non- non-transfer learning results to this setting, despite the challenges. In total, on Flowers- 102 our techniques were able to improve upon Batch Normalization by 2.4% (from 91.0% to 93.4% top-1 accuracy, a 27% relative reduction in error), and upon Group Normalization by 6.1% (from 87.3%, 48% relative reduction error). On CUB-201 I, which has more training data, we improved upon Batch Normal- ization by 1.4% (from 81.1% to 82.4%) and Group Normalization by 3.8% (from 78.6%).
We anticipate that even further improvements might arise by additionally pre- pre-trainin models with were able to large
some of these techniques (particularly Ghost Batch Normalization), impact (roughly 5%) on Group Normalization by pre- pre-training with a Group Normalization-bar model instead of Batch Normalization.

---

Published as a conference paper at ICLR 2020
Table 1: Accuracy on CIFAR-100 with non-i.i.d. minibatches. B' refers to the Ghost Batch Nor- malization size (equivalent to the batch size for Batch Normalization and Batch Renormalization) and "Batch Group Norm." refers to our approach Sec. 3.4 "Inf. Ex Weight: Off" refers to using only the moving averages for normalization statistics (i.e. = 0), while "On" refers to tuning based on the validation set.
Method
B
Inf. Ex. Weight: Off Inf. Ex. Weight: On
Batch Norm
128
40.1
62.2
Batch ReNorm
128
69.0
69.0
64
42.3
50.8
32
57.8
70.9
16
64.3
72.2
Ghost Batch Norm
8
68.7
72.0
4
70.4
71.5
2
68.4
71.4
Batch Group Norm.
2
75.2
76.1
4.4 NON-LLD. MINIBATCHES
implicit assumption in Batch Normalization is that training examples are sampled independently, so that minibatch normalization statistics all follow roughly the same distribution and training statis- ties are faithfully represented in moving averages. However, in applications where training batches
sampled i..dd,, such metric learning (Oh Song al. 2016; Movshovitz-Attin jet al. 2017) or hard negative mining (Shrivastava et al., 2016). violating this assumption may lead to undesired consequences in the model. Here, we test approaches in this challenging setting.
Following Batch Renormalization (Totee 2017). we study the case where examples in a minibatch sampled from small number of classes specifically, we consider CIFAR-100 10R, and study the extreme case where each minibatch (B = 128) is comprised of examples from only four random categories (sampled with replacement), each of which is represented with 32 examples in the mini- batch. We present results for Batch Normalization, Batch Renormalization, our generalization of Batch and Group Normalization from Sec. 3.4 ("Batch Group Norm."). and the full interaction of Ghost Batch Normalization and Inference Example Weighing in Table || In this challenging setting, Inference Example Weighing, Ghost Batch Normalization, and Batch Group Norm all have large fect, in many cases halving the error rate of Batch Normalization. For example, Inference Example Weighing was able to reduce the error rate by 20% without any retraining, and tuning Ghost Batch Normalization, even without any inference modifications, was just effective as Batch Renormal- ization, technique partially designed for the non-i.i.d. case. Even further, Batch Group Normaliza- tion was hardly affected at all by the non-i.i.d. training distribution (76.1 vs 76.2 for i...d). Last, it interesting to note that Inference Example Weighing had practically no effect on Batch Renormal- ization (improvement 0.). confirming Batch Renormalization's 's effect in making models more robust to the use of training vs moving average normalization statistics.
CONCLUSION
In this work, we have demonstrated four improvements to Batch Normalization that should be known by all who use it. These include: a method for leveraging the statistics of inference examples more effectively in normalization statistics, fixing discrepancy between training and inference with Batch Normalization; demonstrating the surprisingly powerful effect of Ghost Batch Normalization for improving generalization of models without requiring very large batch sizes; investigating the previously unstudied effect of weight decay on the scaling and shifting parameters 7 and B; and introducing a new approach for normalization in the small batch setting, generalizing and leveraging the strengths of both Batch and Group Normalization. In each case, have done best to not only demonstrate the effect of the method, but also provide guidance and evidence for precisely which cases in which it may be effective, which hope will aid in their applicability.
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv preprint arXiv:1607.000 0640000650, 2016.
Minhyung Cho and Jaehyung Lee. Riemannian approach to batch normalization. In Advances in Neural Information Processing Systems, pp. 5225-5235, 2017.
Lucas Deecke, lain Murray, and Hakan Bilen. Mode normalization. In International Conference on Learning Reprexentations, 2019.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint arXiv:1706. 02677, 2017.
Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recur- rent neural networks. In 2013 IEEE international conference on acoustics, speech and signal processing, pp. 6645-6649, IEEE, 2013.
Yong Guo, Qingyao Wu, Chaorui Deng, Jian Chen, and Mingkui Tan. Double forward propagation for memorized batch normalization. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016a.
Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European Conference on Computer Vision, pp. 630-645. Springer, 2016b.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of tricks for image classification with convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 558-567, 2019.
Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generaliza- tion gap in large batch training of neural networks. In Advances in Neural Information Processing Systems, pp. 1731-1741, 2017.
Elad Hoffer, Ron Banner, Itay Golan, and Daniel Soudry. Norm matters: efficient and accurate normalization schemes in deep networks. In Advances in Neural Information Processing Systems, pp. 2164-2174, 2018
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint
2017.
Jie Hu, Shen, and Gang Sun. mqueeze-and-excitatton networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018.
Lei Huang, Dawei Yang, Bo Lang, and Jia Deng. Decorrelated batch normalization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 791-800, 2018.
Lei Huang, Yi Zhou, Fan Zhu, Li Liu, and Ling Shao. Iterative normalization: Beyond standardiza- tion towards efficient whitening. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp.
2019.
Sergey loffe. Batch remormalization: Towards reducing minibatch dependence in batch- normalized models. In Advances in Neural Information Processing Systems, pp. 1945-1953, 2017.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In In Proceedings of The 32nd International Conference on Machine Learning, pp. 448-456, 2015.
11

---

Published as a conference paper at ICLR 2020
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. Advances in neural information processing systems, pp. 971-980, 2017.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical report, Citeseer, 2009.
Anders Krogh and John A Hertz. A simple weight decay improve generalization. In Advances in neural information processing systems, pp. 950-957, 1992.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuo- motor policies. The Journal of Machine Learning Research, 17(1):33341333, 2016.
Etai Littwin and Lior Wolf. Regularizing by the variance of the activations' sample-variances. In Advances in Neural Information Processing Systems, pp. 2119-2129,200
Ping Luo, Jiamin Ren, and Zhanglin Peng. Differentiable leaming-to-nnoraalize via switchable normalization. In International Conference on Learning Representations, 2019.
Yair Movshovitz-Attin Alexander Toshev, Thomas K Leung, Sergey loffe, and Saurabh Singh. No fuss distance metric learning using proxies. In Proceedings of the IEEE International Conference on Computer Vision, pp. 360-368, 2017.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
M-E. Nilsback and A. Zisserman. Automated flower classification over large number of classes. In Proceedings of the Indian Conference on Computer Vision, Graphics and Image Processing. Dec 2008.
Hyun Oh Song. Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4004-4012, 2016.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Scan Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211-553 2015.
Tim Salimans and Durk P Kingma. Weight normalization: simple reparaaeettntiinatiionnn to accelerate training of deep neural networks. In Advances in Neural Information Processing Systems, pp. 901-909, 2016.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chies Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4510 4510-4520, 2018
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch nor- malization help optimization? Advances in Neural Information Processing Systems, pp. 2488- 2498, 2018.
Christopher J Shallue, Jaehoon Lee, Joe Antognini, Jascha Sohl-Dickstein, Roy Frostig, and George E Dahl. Measuring the effects of data parallelism on neural network training. arXiv preprint arXiv: 1.03000, 2018
Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick. Training region-based object detectors with online hard example mining. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 761-769, 2016.
N. Silberman and S. Guadarrama. Tensorflow-slin image classification model library. https
//gi Bigithubb.cm wtt.:/cco/teeoeeee wtw.teeensorwweeeeeoooooeeeoetee ow/mode http:////////aaeaa///e/eerrr/a/a
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference Learning Representations, 2015.
12

---

Published as a conference paper at ICLR 2020
Saurabh Singh and Abhinav Shrivastava. Evalnorm: Estimating batch normalization statistics for evaluation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3633- (641,20199
Iya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence sequence learning with neural networks. In Advances in neural information processing systems, pp. 3104-3112, 2014.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818 2016.
Christian Szegedy, Sergey loffe, Vincent Vanhoucke, and Alexander A Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning. In Thirty-First AAAI Confer- on Artificial Intelligence, 2017.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing in- gredient for fast stylization. arXiv preprint arXiv:1607.08022, 2016.
Twan van Laarhoven. 1.2 regularization versus batch and weight normalization. arXiv preprint
arXiv: 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd
birds-200-201 dataset. 2011.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 3-19, 2018.
Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel S Schoenholz, and Jeffrey Penning- ton. Dynamical isometry and a mean field theory of cnns: How to train 10,000-layer vanilla convolutional neural networks. In International Conference Machine Learning. 2018.
Hongyi Zhang, Yann N. Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization. In International Conference on Learning Representations, 2019.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. In Interna- lional Conference on Learning Representations, 2017.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference computer vision and paltern recognition, pp. 8697-8710, 2018.
13

---

Published as a conference paper at ICLR 2020
A PROOF OF BATCH NORMALIZATION OUTPUT BOUNDS
Here we present a proof of Eq. We first prove the bound as an inequality and then show that is tight. Without loss of generality, we assume that xo is the minimum of {2:))=0 -- and that y 2 0. Then we want to show that
min
To M4 o -
- 3
(5)
Expanding μ4; and of (using the maximum likelihood estimator for of), and canceling the scaling and offset terms y and B, we want to show
B 1-1 B 2
min 20...... BB
-B- -
(6)
B Σド= 22,² +E
From here we assume without loss of generality that xy = since the output of Batch Normal- ization is invariant to an additive constant on all H,, we can subtract za from all I; and maintain the same value. We also assume that all x; 2 0, then frame the minimum as a bound
B-1 2i=0 T1
2
-VB-1
Zi=0 (xi
B-1
B 2 I3 4E
B-1
+E
R-1
B - B-1
B
Zj
+ E
(8)
B
B 1=0
B j=0
1=0
2
B-1
B-1 B-1
B-1
B-1
2r
34 2
Σ
1 +
Σ
Iz
+
(9)
B 1=
B
B
B
1=0
1=0
2
B-1
B-1
B-1
B-1
B 1
Σ
34 R
B - 1)-1)Bay -BB - 1)0;
I3 +
Σ T
(10)
B
j=0
j=0
B-1
B-1
B B-1
2
B
2
Σ I4 S
(B - 1)B Σ
2 - 2(B 1)
Σ 1=0
Iz
+ +(B - 1)
Σ
+
(11)
1=0
1=0
B-1
B-1 Σ
B-1
2
Σ
32 5
BB 1)B
(B -1)
Σ
+
(12)
1=0
1=0
1=0
B
2
B-1
B-1
2
Σ
<
(B- - 1)B
Σ
2
(B 11
Σ
Fi
+
(13)
1=0
1=
1=0
B-1
B-1 Σ
B
Σ
I2
(B 1)B
x
+
(14)
1=0
1=0
B
1
2
B-1 < 1) Σ
Σ
2
+
(15)
1=0
1=0
Using the fact that vo and é > 0, it suffices show
B 1
2
B 1
Σ
x
- 1)
Σ
x
(16)
i=1
14

---

Published as a conference paper at ICLR 2020
With change of variables, we have the more general
N
1
2
N
Σ
I2
(17)
0=
N
2
N
2
(18)
N
N
t=0
0=
E|x] 2 - B|x²)
(19)
E|²²] - E 2
(00)
which is simply an alternate form for the variance of which is always non-negative, completing the bound.
To show that the bound is tight, we can set zo = 0 and x. = for all i >> 0, where is non-negative constant:
B-1
B 2i=0 34
(21)
Enj )
V
B 2i= (0i
B 2===
1 Σ
B i== a
(22)
1
(B (B-1)² 1)2 B3
a² + Di=l (a - a)) te
B-1
B
B B-1a
(23)
1) B2
a²
+ (B- 1)a2 (1 (1- 2(B-1) (1-2(B-1)
(B-1)2)
B²
+
-B 1)a
(24)
B
(B1B
B-1 B2
+ 1 - 2 B-1 (B-1)²)
B2
+ c
-BB 1B-117 1)a
(25)
1 ce((B 1) (
B-1
+ B 22B + 2 + (B-1)2
+
B
-(B 1)a
(26)
Vol(B (1) [2 -B2-2B+1 B 1- B2+2B
-BB 1)a
(27)
(a 1) E
As + xo (or if 0), then this approaches
(B 1)a
(28)
a (
which is simply
VB
(29)
completing the proof.
15

---

Published as a conference paper at ICLR 2020
BN Output Range
-20
Bound during training
Observed during training
-00
Observed during testing
lBg
Figure 6: Range of output values obtained during inference the CIFAR-10 test set, compared with the range observed during training and the bound Eq. P See text for details.
B EMPIRICAL EVIDENCE OF BATCH NORMALIZATION OUTPUT BOUNDS
In Fig. 6 we show the observed output ranges for the last Batch Normalization layer in our CIFAR- 10 network (spatial resolution: 4 4), plotting both the range during training and at inference time on the CIFAR-10 test set. Different values of B were obtained by using different Ghost Batch Normalization sizes, keeping in mind that B is determined by the product of the batch size and spatial dimensions.
At large values of B, it is unlikely that any network obtains a value even close to the bound of Eq. 2 but as B gets smaller, the output range of the network during training becomes smaller in magnitude, eventually being nearly tight with our bound for example, for log = 5. the theoretical minimum is -5.57, while the network obtained minimum of -5.30. However, the maximum and minimum values obtained during inference on the test set show no clear pattern as B changes, and not subject to the training time bound, which is particularly noticeable for small values of B, where values fall outside the training-tine bounds.
C NEGATIVE RESULTS: APPROACHES THAT DIDN'T WORK.
Here detail handful of approaches which seemed intuitively promising but ultimately failed to produce positive results.
BATCH NORMALIZATION MOVING AVERAGES. In an attempt to resolve the other disparities Batch Normalization has between its training and inference behaviors, we experimented with handful of different approaches for modifying the moving averages used during inference. First, since examples at inference time do not have data augmentation applied to them, we tried comput- ing the moving averages over examples without data augmentation (implemented by training the model for few extra epochs over non-augmented examples with learning rate of 0. but while still updating the moving average variables). This decreased accuracy on CIFAR- 100 by roughly half a percent, though it did yield mild improvements to the test set cross-entropy loss.
Next, we experimented with calculating the moving averages over the test set, not making use of any of the test labels. Perhaps surprisingly, this behaved very similar to when moving averages were calculated over the training examples (within 0.1% in accuracy and within 1% in cross-entropy), with trends holding regardless of whether data augmentation was applied not.
ADDING BATCH NORMALIZATION-LIK STOCHASTICITY TO GROUP NORMALIZATION. One of the hypotheses for why Group Normalization generally performs slightly worse than Batch Nor- malization is the regularization effeet of Batch Normalization due random minibatches producing variability in the normalization statistics. Therefore, we tried introducing stochasticity to Group Normalization in variety of ways, none of which could get to work well: 1) Adding gaussian
16

---

Published as a conference paper at ICLR 2020
ResNet-112
MobileNeetv
NasNeet-A
$6.22
$4.
Top ACC
10
$9.25
LOSS
00.
60-
$4.50
665
500
0.33
19:9
0.60
0.1
2wW.SSOOO 10P.10017711100 178-14011
Figure 7: Effect of the example-weighing weighing hyperparameter on ImageNet; supplemental version of Fig. T
with a larger range of a.
CIFAR-100 Normalizztiin
SVHN Normalization
[alttcch-255 with Normalization
60
ACC
Lock
0.0
0.2
0.5
0.5
0.
0.2
3.4
0.6
0.0
0.
0.4
1.5
3.
Figure 8: Effect of the example-weighing hyperparameter for models trained with Group Nor- malization on CIFAR-100, SVHN, and Caltech-255 supplemental version of Fig. with a larger range of a.
noise the normalization statistics, where the noise is based on moving average of the normal- ization statistics, 2) Using random groupings of channels for calculating normalization statistics (optionally only doing randomization a fraction of the time), and 3) changing the number groups throughout the training procedure, either as increasing decreasing functions of training steps.
MORE PRINCIPLED GROUP SIZE COMPUTATION. As part of generalizing Batch and Group Nor- malization, examined whether was possible to determine the number of groups in each nor- malization layer in a more principled way that simply specifying it as a constant throughout the network. For example, one approach had mild success with was setting the number of elements per group (height x width x group size) to constant, making the number elements contributing the normalization statistics uniform across layers. However, we were unable get any of these ideas to work in a way that generalized properly across datasets. We also tried learning group sizes differentiable way with Switchable Normalization, but found that this made models overfit too much.
D SUPPLEMENTAL INFERENCE EXAMPLE WEIGHING PLOTS
In Figures and we present plots corresponding to Figures and of the main text, with larger ranges of the inference weight a. In the main text, we restricted the range of a to values which showed off the tradeoff of a versus performance at reasonably local scale, and these figures show a larger scale for completeness in characterizing model behavior. While this behavior can largely be extrapolated from the behavior for smaller range of a, there are some interesting trends.
ImageNet 7 we see that only small amount inference example weighing is necessary to get most of its benefit, and setting ar to larger values corresponds to a regime quite different than in training. smoothly decaying model performance as a becomes less and less appropriate. Similarly, when applying inference example weighing to Group Normalization (Fig. [ while performance intuitively decays as a moves farther and farther away from 1, a surprisingly large range of values Group Normalization, especially SVHN. Lastly, when
for result in similar performance comparing the effect of a on models trained with Ghost Batch Normalization (Fig. 9 we clearly see that the optimal value for a is decreasing with respect the Ghost Batch Normalization size, with the possible unusual exception of optimizing for loss on SVHN.
17

---

Published as a conference paper at ICLR 2020
CIFAR-100
Ghort
Ghoot Batrh Stre
Ghoet Sabri
Barch 16
J0C
033
7.4
Caltech-256
Chust Butch Siac
Ehost Butch Sue
Chast Butch Six
Sirest Satch Size
SVHN
60.20
Groet Sabch Saw
Gbodt Batce Strw
Choer Batch Size
Ghost faems Sow
63.07
$0.00
LOGS
Figure 9: The complementary effects of Inference Example Weighing and Ghost Batch Normaliza- tion on CIFAR-100, SVHN, and Caltech- 256; supplemental version of Fig. 4 with larger range of a.
18


Reviews:
[{"review_id": "r1lmLcNaYB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors discuss four techniques to improve Batch Normalization, including inference example weighing, medium batch size, weight decay, the combination of batch and group normalization. Equipped with the proposed techniques, the authors obtain promising results when training deep models with various batch sizes. However, the novelty of this paper seems very limited and more experiments are required.\n\nPlease see my detailed comments below.\n\nPositive points：\n1. The proposed inference example weighing method yields promising results and does not require any re-training.\n2. The combination of batch and group normalization makes it possible to train deep models with very small batch size.\n3. By combining all the techniques, the proposed method yields promising performance when training deep models with different batch sizes.\n\nNegative points:\n\n1. Some notations are very confusing. For example, the authors use B to represent the size of a minibatch. However, why do the authors only consider B-1 samples in Eq. (2), i.e., selecting the minimum possible output among x_1, …, x_{B-1}?\n\n2. The proposed inference example weighing method seems very similar to Batch Renormalization. Both methods seek to use a linear function to combine the batch statistics and the moving statistics. What is the essential difference between these methods?\n\n3. What model do the authors use in the experiment of Figure 2? Why do the authors conduct experiments on different datasets in Section 3.1 (including ImageNet) and Section 3.2 (excluding ImageNet)? It would be stronger to provide ImageNet results in Section 3.2.\n\n4. The authors draw different conclusions about the usage of weight decay from a recent work (He et al, CVPR2019). The CVPR paper reports that training \\gamma and \\beta without weight decay on ResNet-50 yields significant performance improvement. However, this paper shows that training ResNet-50 with weight decay improves the performance. Please comment on the differences in the conclusions.\n\nReference: \"Bag of tricks for image classification with convolutional neural networks.\" CVPR, 2019.\n\n5. The authors only report ImageNet results of the proposed inference example weighing method. However, all the experiments in Section 4 are performed on three small datasets. It is necessary and important to provide ImageNet results to show the effectiveness of the other three techniques in Section 4. \n\n6. Note that training deep models with non-i.i.d. minibatches is a typical case to evaluate normalization methods, e.g., Batch Renormalization. Specifically, examples in a minibatch are not sampled independently. What would happen if the authors apply the proposed techniques to the non-i.i.d. case?\n\n7. Some closely related work should be discussed in the paper, such as\n\n[1] \"Decorrelated Batch Normalization.\" CVPR, 2018.\n[2] \"Double Forward Propagation for Memorized Batch Normalization.\" AAAI, 2018.\n[3] \"Differentiable Dynamic Normalization for Learning Deep Representation.\" ICML, 2019.\n[4] \"Iterative Normalization: Beyond Standardization towards Efficient Whitening.\" CVPR, 2019.\n\nMinor issues:\n1. In Section 1, the third contribution is not a complete sentence.\n\n2. There are many typos in the paper.\n(1) In Section 2, “Layer Normalization, which has found use in many natural language processing tasks.” Should “which has found use” be “which has been used”?\n(2) In Section 3.1, “Batch Normalization has a disparity in function between training inference”. “between training inference” should be “between training and inference”.\n(3) In Section 3.1, “we need only figure out …” should be “we only need to figure out …”\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "r1glmJ2ptH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper performs an empirical study of four batch-normalization improvements and proposes a new normalization technique for small batch sizes, based on group and batch normalizations. Among others, the authors address the inconsistency between the train and the test stages and the problem of small batch sizes. The authors conducted an empirical ablation study of the four techniques and proposed an intuition when each method should be used.\n\nConcerns:\n(1) The comparison with baselines in section 4.2 seems to be unclear. Fig.5 shows the performance of normalization methods for different batch sizes. Batch Normalization, however, has the same performance for all batch sizes. The authors refer to this baseline as “idealized Batch Normalization”. Additional elaboration on what does this means is required. \n(2) It would also be beneficial to see the comparison with the original Ghost Batch Normalization in the final evaluation (section 4.2), since this method, according to section 3.2, was capable of the significant improvement for Caltech-256 dataset.\n(3) In section 3.1, the authors provide an intuition of why can the discrepancy between test and train phases hurts the performance of a model. The empirical evaluation of this effect is needed to justify this intuition.\n\nOverall, the newly proposed method is a minor update, and novelty is limited. However, the thorough empirical study of existing improvement techniques would be a good addition to the conference.\n\nMinor comments:\n1. share the y-axis in Fig.4 between different ghost batch sizes.\n\nI would also recommend authors to include the following papers to the related work section:\n1. Riemannian approach to batch normalization [https://arxiv.org/abs/1709.09603]\n\n----------\n\nRespond to the rebuttal. Clarification on the concern (3):\n\nI agree that, in general, a discrepancy between training and testing can hurt a model. The paper showed that the output of a batch normalization layer is theoretically unbounded during testing. However, it would be beneficial to see numerically if it indeed the case on a real test set (the output range is wider than the one during training).\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1lHRzu6YS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces four techniques to improve the deep network model through modifying Batch Normalization (BN). The inspirations are from the gaps between train&test and between batches in multi-gpu training, comparison to other normalization methods, and weight decay in regularizing convolution weights training. The paper studies each techniques with the support from experiments. The paper is easy to follow. The techniques seem effective.\n\nThe paper mentions \"theory\" multiple times, but lacks sufficient justification to support these \"theories\". So one suggestion is to replace \"theory\" with a soft word.\n\nExperimental evidence seems sufficient and there are some theoretical derivations, but it looks incremental that the paper presents some techniques in improving Batch Normalization only. In general, the paper is of values to the community.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes techniques to improve training with batch normalization. The paper establishes the benefits of these techniques experimentally using ablation studies. The reviewers found the results to be promising and of interest to the community. However, this paper is borderline due in part due to the writing (notation issues) and because it does not discuss related work enough. We encourage the authors to properly address these issues before the camera ready.
Decision: Accept (Poster)