Batch Normalization has Multiple Benefits: An Empirical Study on Residual Networks
Authors: Soham De
Samuel L Smith
Venue:  (2020)
Keywords: batch normalization
residual networks
initialization
batch size
learning rate
ImageNet

Abstract: Many state of the art models rely on two architectural innovations; skip connections and batch normalization. However batch normalization has a number of limitations. It breaks the independence between training examples within a batch, performs poorly when the batch size is too small, and significantly increases the cost of computing a parameter update in some models. This work identifies two practical benefits of batch normalization. First, it improves the final test accuracy. Second, it enables efficient training with larger batches and larger learning rates. However we demonstrate that the increase in the largest stable learning rate does not explain why the final test accuracy is increased under a finite epoch budget. Furthermore, we show that the gap in test accuracy between residual networks with and without batch normalization can be dramatically reduced by improving the initialization scheme. We introduce “ZeroInit”, which trains a 1000 layer deep Wide-ResNet without normalization to 94.3% test accuracy on CIFAR-10 in 200 epochs at batch size 64. This initialization scheme outperforms batch normalization when the batch size is very small, and is competitive with batch normalization for batch sizes that are not too large. We also show that ZeroInit matches the validation accuracy of batch normalization when training ResNet-50-V2 on ImageNet at batch size 1024.

Under review as a conference paper at ICLR 2020
BATCH NORMALIZATION HAS MULTIPLE BENEFITS: AN EMPIRICAL STUDY ON RESIDUAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Many state of the art models rely on two architectural innovations; skip connections and batch normalization. However batch normalization has a number of limitations. It breaks the independence between training examples within batch, performs poorly when the batch size is too small, and significantly increases the cost of computing parameter update in some models. This work identifies two practical benefits of batch normalization. First, it improves the final test accuracy. Second, it enables efficient training with larger batches and larger learning rates. However we demonstrate that the increase in the largest stable learning rate does not explain why the final test accuracy is increased under a finite epoch budget. Furthermore, we show that the gap in test accuracy between residual networks with and without batch normalization can be dramatically reduced by improving the initialization scheme. We introduce "Zerolnit", which trains a 1000 layer deep Wide-ResNer without normalization to 94.3% test accuracy on CIFAR- in 200 epochs at batch size 64. This initialization scheme outperforms batch normalization when the batch size is very small, and is competitive with batch normalization for batch sizes that are not too large. We also show that ZeroInit matches the validation accuracy of batch normalization when training ResNet-50-v2 on ImageNet at batch size 1024.
INTRODUCTION
Skip connections (He al.) 2016a) and batch normalization (Loffe Szegedy 2015) have contributed to the dramatic improvement in the performance of convolutional nets in recent years. However batch normalization also has a number of limitations. It breaks the independence between training examples within a batch, performs poorly with small batch sizes (Wu & He 2018), and increases the cost of computing an update (Wu et al., 2018). Many authors have tried to understand the role batch normalization plays during training (Balduzzi al. 2017 [Santurkar al. 2018 Bjorck et al. 2018
Sankararaman et al. 2019) Yang et al. 2019; Ghorbani et al. 20199), to replace batch normalization with cheaper alternatives (Ba et [016 Salimans & Kingma 2016 Ulyanov et 2016 to
remove normalization entirely (Mishkin & Matas 2015 Krähenbühl et al. 2015 Zhang et al. 2019).
Santurkar al. (2018) argued that batch normalization helps primarily because it improves the
conditioning of the loss landscape, enabling stable training with larger learning rates. [Bjorck et al.) (2018) argued that this not only explains why batch normalization helps during optimization, but also explains why it improves the final test accuracy, since large learning rates are thought to generalize well (Keskar et al. 2016, Smith& & Le 2017 Dastrzbbkki et al. 2017). Other authors have remarked that batch normalization introduces noise by computing the batch statistics over subset of the training data, helping generalization (Luo et al.) 2019). Meanwhile, both Balduzzi et al.) (2017) and Yang et al.) (019)) argued that the combination of batch normalization and skip connections preserves the correlation between the gradients of different training examples at initialization, and they speculated that this enables the efficient training of deep architectures. Mishkin & Matas (2015), Krähenbühl et al. (2015) and Zhang et al. (2019) have all demonstrated empirically that some of the benefits of batch normalization
be recovered by improving the initialization scheme.
In this work, we argue that one cannot build a complete picture of the benefits of batch normalization, unless studies the training loss and test accuracy achieved by deep networks at a wide range of batch sizes. When the batch size is large, one can train with large learning rates, and the role of batch normalization in enhancing the conditioning of the loss landscape becomes apparent. When the batch

---

Under review as a conference paper at ICLR 2020
size is smaller, one does not need large learning rates to train efficiently, and so can study the other benefits of batch normalization in isolation. Our results establish the following:
1. Batch normalization does enable training with significantly larger learning rates, and this enables practictioners to efficiently parallelize training over much larger minibatches.
2. However this is not the only benefit. Batch normalization also performs well on both the training loss and the test accuracy with small batch sizes and small learning rates.
3. We propose "ZeroInit", a simple change to the initialization scheme of residual networks, designed to preserve the correlation between gradients at the start of training. We show that this enables us to train very deep residual networks without normalization, and our method recovers most of the benefits of batch normalization when the batch size is not too large.
Our results suggest that batch normalization improves conditioning (Santurkar al. 2018 Bjorck et al. 2018), preserves the correlation between gradients (Balduzzi et al. 201 San Sarrmmaa et al. 2019 Yang al. 2019), and has additional regularization benefits (Luo et al.) 2019). It is easy to miss one or more of these benefits if one performs experiments at a single batch size. One should also be cautious in claiming to identify a complete replacement for batch normalization without first verifying that it matches the performance of batch normalization for wide range of batch sizes.
This paper is structured as follows. In section 2 we introduce residual networks, batch normalization, and Fixup initialization (Zhang et al. 2019). In section 3 we compare the test accuracy and training loss of residual networks trained with and without batch normalization at a range of batch sizes, with the aim of isolating the different benefits batch normalization brings. In section 4 we introduce "ZeroInit", a simple initialization scheme inspired by both Fixup and shattered gradients (Balduzzi et al. 2017). This scheme trains very deep residual networks to high accuracies without batch normalization, but it does not replicate the full benefits batch normalization when training with large batch sizes. In section 5 we evaluate the performance of Zerolnit and Fixup ImageNet.
2 BACKGROUND
Residual Networks (ResNets): First introduced by He et al. (2016b). these models are composed of a sequence of residual blocks, which contain both a "residual branch" containing a number of convolutions and activation layers, as well as a "skip connection", which is typically just the identity. These skip connections create pathways from the input to the output which have a shorter effective depth than the main path through all the residual branches, and these pathways enable the input signal to pass more easily from the input to the output (Balduzzi et al. [0117] Sankararaman et al. 2019). Crucially, the network still retains high degree of expressivity thanks to the main path through all the residual branches. Most of the experiments in this paper will follow the popular Wide Reseet architecture, introduced by Zagoruyko & Komodakis (2016). However also consider the original ResNet-V1 He et al. 2016a) and ResNet-V2 V2 (He et al.) 2016b) architectures in section 5.
Batch Normalization: As in most previous work, we apply batch normalization in convolutional layers. The inputs to and outputs from batch normalization layers are therefore 4-dimensional tensors, which we denote by 1o,2. and Op. Here b denotes the minibatch, e denotes the channels, and x and denote the two spatial dimensions. Batch normalization applies the same normalization to every input in the same channel (loffe & Szegedy 2013). This leads to the update:
Ob.,, = % I0,z.g. Pe + Bc.
Vo?+
Eo.2, Here, Me - = Eb.e,y Jb.z.g.c denotes the per-channel mean, and -² = Eo Ih. 3 denotes the per-channel variance of the inputs, and Z denotes the normalization constant, which summed over the minibatch and spatial dimensions and y. A small constant is generally added to the variance for numerical stability. The "scale" and "shift" parameters, To and B, respectively, are learnt during training and are added to preserve model expressivity after the normalization operation. Running averages of the mean je and variance of are also maintained during training. and these averages are used at test time to ensure the predictions are independent of other examples in the batch.
Batch normalization enables researchers to train significantly deeper residual networks (Sankararaman et al. 2019), and these networks have achieved fate-of-the-an performance number of tasks.
2

---

Under review as a conference paper at ICLR 2020
However batch normalization also has number of limitations. It breaks the independence between training samples in a minibatch, which makes it harder to apply in certain models (Girshick] (013)) and also contradicts the assumptions of most theoretical models of optimization (Polyak & Juditsky 1992. Mandt et al. 2017; Park et al. 2019). The normalization operation itself can constitute a significant fraction of the total cost of computing parameter update in some models (Wu et al..
2018). Batch normalization also performs poorly when the batch size is too small (Toffe) 2017]] Wu &
He 2018), which often limits the size of model that can be trained on single device.
Fixup: Zhang et al. (2019) proposed Fixup initialization which removes batch normalization entirely from residual networks. They demonstrated that this algorithm is able to train ResNet-50 competi- tive validation accuracies on ImageNet. while it also displayed promising results when training very deep (1000+ layers) Wide-ResNets on CIFAR-10. The Fixup algorithm has four components:
1. The classification layer and final convolution of each residual branch are initialized to zero.
2. The initial weights of the remaining convolutions are scaled down by L-1/(2m-2 2), where L is the number of residual branches and m the number of convolutions in each branch.
3. A scalar multiplier is introduced at the end of each residual branch, intialized one.
4. Scalar biases are introduced before every convolution, linear and element-win activation layer, initialized to zero.
It is not clear whether all of these components are required to enable efficient training without batch normalization, or a subset of these components would be sufficient. The authors claim that component 2 above crucial, however we will demonstrate below that is not required in practice. Additionally, the authors only provide results at a single batch size and do not tune the learning rate. We demonstrate below that a significantly simpler initialization scheme (ZeroInit) enables the efficient training of very deep residual networks without batch normalization when the batch size is not too large. ZeroInit and Fixup both achieve similar test accuracies to batch normalization when the batch size is small, but they underperform batch normalization when the batch size is very large.
3 THE BENEFITS OF BATCH NORMALIZATION
The primary goal of this paper is to identify the empirical benefits of batch normalization when training residual networks. We perform most of our experiments on CIFAR-10 (Krizhevsky et al. 2009), since this enables us to run thorough experiments, averaging across multiple training runs and tuning the learning rate at each batch size. To verify that our conclusions scale to larger datasets, we provide a limited set of experiments on ImageNet in section 5 (Russakovsky et al. 2015).
In figure 1, we provide the mean performance of 16-4 Wide-ResNe (Zagoruyko & Komodakis] 2016), trained on CIFAR-10 for 200 epochs at a range of batch sizes. At each batch size, we train the network 15 times for a range of learning rates on logarithmic grid, and we independently measure the mean and standard deviation of the best 12 runs on both the training loss and the test accuracy. This ensures that our results are not corrupted by outliers or failed runs. Throughout this paper, we train using SGD with heavy ball momentum, and fix the momentum coefficient m = 0.9. The optimal test accuracy is the mean performance at the learning rate whose mean test accuracy was highest, while the optimal training loss is the mean training loss at the learning rate whose mean training loss was lowest. We always verify that the optimal learning rates are not the boundary of our grid search. Although we tune the learning rate on the test set, we emphasize that our goal is not to achieve state of the art results. Our goal is to compare the performance of different training procedures. We apply the same experimental protocol in each case.
Experimental Setup:
We adopt the following simple learning rate schedule. We first hold the learning rate constant for 100 epochs, before dropping the learning rate by a factor of every 10 epochs. We find that this simple schedule outperforms the original sequence of 3 drops proposed by He et al. (2)1 We apply data augmentation including padding, random crops and left-right flips and we also use L2 regularization with coefficient of 5 x 10-4, We provide results both with and without batch normalization. When using batch normalization, we evaluate the batch statistics over a fixed ghost batch size of 64 (Hoffer et al. (017), before accumulating gradients to form larger batches. Evaluating the batch statistics over fixed number of training examples independent of the batch size preserves the benefits of noise when the batch size is large (Luo et al. 2019). and reduces communication overheads in distributed
3

---

Under review as a conference paper at ICLR 2020
0.95
1.25
Optimal test accuracy
Optimal training loss
w/ batch norm
0.94
1.00
w/o batch
Zerolnit
0.93
0.75
0.92
0.50
batch norm
0.91
w/o batch norm
0.25
Zerolnit
0.90
0.00
2
Batch size
Batch size
(a)
(b)
Figure 1: A 16-4 Wide-ResNet, trained on CIFAR-10 10 for 200 epochs at a range of batch sizes. We report the performance both with and without ghost batch normalization, and also provide the performance without ghost batch normalization using our modified initialization scheme "Zerolnit". We perform grid search to identify the optimal learning rate which maximizes the test accuracy at each batch size in figure la, and which minimizes the training loss at each batch size in figure 1b, and we report the mean performance of the best 12 of 15 runs a) Batch normalization has at least two benefits. It improves the final test accuracy, and it also substantially increases the largest batch size we can use efficiently during training. Our modified initialization scheme recovers most of the improvement in the test accuracy, however the largest efficient batch size remains an order of magnitude smaller. b) Batch normalization achieves smaller training losses at all batch sizes, while both methods without normalization achieve similar training losses when the batch size is small.
training (we provide experiments using the original formalism of batch normalization in figure 3). For completeness, also provide results here without normalization using simple modification to the initialization scheme, "ZeroInit". We will describe this modification in full in section 4.
Looking the optimal test accuracy in figure 1a, make three observations. First, the op- timal test accuracy is substantially higher with batch normalization than without batch normalization, even after we tune both the learning rate and the batch size. Second, both with and without batch normalization, the performance is independent of batch size in the small batch limit, before beginning fall as the batch size rises. Third, the largest batch size at which one can train efficiently with batch normalization is substantially larger than the corresponding batch size without batch normalization. Since the number of training epochs is fixed, this indicates that one can train Wise-ResNets with significantly fewer parameter updates when using batch normalization. Given access to sufficient hardware, this will enable practitioners to dramatically reduce the wallclock time of training (Goyal al. 2017). Zerolnit significantly improves the optimal test accuracy achieved without normalization, and is competitive with batch normalization (within one standard deviation) for batch sizes B S 256. However it underperforms batch normalization when the batch size is large. Now considering the optimal training accuracy in figure 1b, we find that batch normalization achieves lower optimal training losses all batch sizes, and continues perform well when the batch size is large. The optimal training loss without batch normalization is similar both with and without ZeroInit. However used, the optimal training loss degrades rapidly as the batch size is increased.
Results:
when ZeroInit
To help interpret these results, in figure 2 we provide the optimal learning rates at which both the training loss minimized and the test accuracy is maximized. From figure 2a, we conclude that for all three techniques, the optimal learning rate is initially proportional to the batch size, before saturating when the batch size is large (Krizhevsky) 2014 Goyal et al. 2017 Smith al. 2017 McCandlish et al., 2018 (Shallue et al. 2018). Furthermore, the optimal ratio of the learning rate to the batch size is similar for all three techniques. However, while the optimal learning rate saturates at a maximum value of 2-3 without batch normalization, it reaches larger value of 2 with ZeroInit, and an even larger value of 24 with batch normalization. This directly explains why batch normalization and Zerolnit perform better when the batch size is large; they have larger maximum stable learning rates which enables them to scale efficiently to larger batches (McCandlish et al., 2018). We observe very similar results when considering the optimal learning rates minimizing the training loss in figure 2b.
4

---

Under review as a conference paper at ICLR 2020
w/ batch norm batch Zerolnit
w/ batch norm batch Zerolnit
Optimal learning rate (for test accuracy)
Optimal learning rate (for training loss)
29 Batch (a)
27 Batch
(b)
Figure 2: 16-4 Wide-ResNet, trained on CIFAR-10 for 200 epochs at range of batch sizes. For all three methods, the learning rate which maximizes the test accuracy is proportional to the batch size when the batch size is small, before saturating when the batch size is large. Batch normalization achieves significantly larger optimal learning rates in the large batch limit, however all three methods have similar optimal learning rates at the same batch size when the batch size is small. This demonstrates that the stability of batch normalization at large learning rates is not the reason why batch normalization performs better on the test set when the batch size is small. b) We observe similar trends when we evaluate the optimal learning rates which minimize the training loss.
Discussion: Santurkar al. 2018) and Bjorck et al. (2018) argued that the empirical success of batch normalization is primarily explained by the increase in the maximum stable learning rate, which arises because batch normalization improves the conditioning of the loss landscape. However our results conclusively demonstrate that this is not the case. While a larger maximum stable learning rate is the primary reason why batch normalization performs well with large training batches, when the batch size is small the optimal learning rate both with and without batch normalization is also small. Nevertheless, the optimal test accuracy and the optimal training loss are significantly improved when batch normalization is used in the small batch limit as well as the large batch limit.
conclude that batch normalization increases the largest stable learning rate, and this enables us to reduce the number of parameter updates required to train the model when training with large batches. However batch normalization also improves the final training loss and test accuracy, even when the batch size and learning rate are small. Our results demonstrate that one cannot understand batch normalization without studying its performance at a range of batch sizes. ZeroInit recovers much of the performance gap the test set when the batch size is small, but it does not close the gap in the training loss, and it performs worse with large batches. It is difficult to measure exact wallclock times in our distributed training setup, but find that 16-4 wide residual networks typically complete 200 epochs roughly 40% faster at batch size 64 without batch normalization (Wu et al. 2018).
4 "ZEROINIT": A SIMPLE ALTERNATIVE TO BATCH NORMALIZATION
We established above that larger stable learning rates are not the only benefit of batch normalization, and that they do not explain why batch normalization achieves lower training losses and test accuracies under finite epoch budget when the batch size is small (Santurkar et al. 2018 Bjorck et al. 2018). We now identify the minimal components required to train very deep ResNets without normalization. Balduzzi et al. (017)) and Yang et al. (2019) both argued that batch normalization when used with skip connections preserves the corre latronations between gradients at initialization, and Balduzzi et al. (2017) argued that these correlations can also be preserved by initializing deep networks close to linear functions. Inspired by these results, we introduce a new initialization scheme, "ZeroInit", comprising three simple components
1. We introduce a scalar multiplier the end of each residual branch, initialized to zero.
2. We introduce biases to each convolution and the classification layer, initialized zero
3. We apply Dropout the classification layer. If not specified, the drop probability is 60%.
5

---

Under review as a conference paper at ICLR 2020
Table 1: The performance of range of 16-4 Wide-ResNets on CIFAR-10 without batch normalization at batch size 64. We use compute budget of 200 epochs and follow our standard learning rate schedule. We perform two independent grid searches identify the learning rate which minimizes the training loss and the learning rate which maximizes the test accuracy, and we provide the mean performance of the best 12 out of 15 runs. Introducing scalar multiplier initialized at zero to each residual branch is sufficient to achieve test accuracy of 94%, and this configuration also achieves the lowest training loss of 0.181. Meanwhile Zerolnit, which combines this scalar multiplier with biases and dropout, improves the optimal test accuracy but also increases the training loss.
Method
Optimal test accuracy (%)
Optimal training loss
16-4 wide ResNet w/o batch normalization
92.3 + 0.3
0.212 + 0.004
biases
91.9 + 0.2
0.205 + 0.003
+ dropout
93.2 + 0.1
0.226 + 0.001
scalar multiplier
94.0 + 0.1
0.181 0.001
+ biases & dropout
93.6 = 0.1
0.212 + 0.001
Zerolnit (scalar multiplier, biases & dropout)
94.4 + 0.1
0.189 0.001
Table 2:
The performance of d-2 Wide-ResNet CIFAR-I0 10 at batch size 64, for range of depths d. We use a compute budget of 200 epochs and follow our standard learning rate schedule. We perform grid search to identify the optimal learning rate, and provide the mean performance of the best 12 out of 15 runs. Both batch normalization and Zerolnit are able to train very deep networks up to (at least) one thousand layers. The optimal learning rate only weakly dependent on depth.
Batch Normalization
ZeroInit
Depth
Test accuracy (%)
Optimal learning rate
Test accuracy (%)
Optimal learning rate
16
93.5 = 0.1
2-1 2-1 (2 2-2---2 20)
93.3 + 0.1
2 2-
100
94.7 = 0.1
94.2 + 0.1
2-2
1000
94.6 = 0.1
2- (2-3 ----2 -20)
94.3 + 0.2
2- (2-3 -2-3-2-1)
We note that each of these modifications should not require changing more than a couple of lines of code. Modification makes the residual blocks linear at initialization, so long as there are no non-linear activations on the skip connection. This preserves the gradient correlations at initialization (Balduzzi et al. 2017), and we show below that this single change is sufficient to train very deep ResNets without normalization. Modification 2 is added to preserve the model expressivity, and Modification is added to recover some of the additional regularization benefits batch normalization provides (Luo et al., 2019). We note that neither
(Zagoruyko & Komodakis (016) nor Re (He et al. 201 have non-linear activations on the skip connection, while ResNet-Vi (He et a 6a) does. We discuss how modify ZeroInit for Re in section 5. We note that Fixup initialization also initializes the residual branches at zero, while Goyal et al.) (2017) proposed initializing the scalar multiplier in the final batch normalization layer of the residual branch to zero.
201
We provide an ablation study in table 1, which demonstrates that solely introducing scalar multiplier initialized to zero sufficient to train our 16-4 wide residual network 94% test accuracy. This configuration also achieves the lowest training loss. Introducing biases and dropout further increases the test accuracy to 94.4%. In table 2, we compare the performance of batch normalization and ZeroInit at range of depths between 16 and 1000 layers. To ensure each model fits a single GPU, we reduce the width factor from to which slightly reduces the test accuracy. While
Table The performance of d-2 Wide-ResNet on CIFAR-IO at batch size 64, for range of depths d. We train for 200 epochs. We perform a grid search to identify the optimal learning rate, and provide the mean performance of the best 12 out of 15 runs. L.2 regularization is not required to train very deep networks with ZeroInit, and we can also train very deep networks without biases or dropout, using solely a scalar multiplier at the end of each residual branch initialized to zero
ZeroInit without L2
Scalar Multiplier
Depth
Test accuracy (%)
Optimal learning rate
Test accuracy (%)
Optimal learning rate
16
91.0
2-3
93.1 + 0.1
2-2 (2-3 2-2)
100
92.2 + 0.1
2
94.1 0.1
2-2 (2-3 2-2)
1000
92.5 + 0.2
2-3
94.2 + 0.1
2-1 (2-2 2-

---

Under review as a conference paper at ICLR 2020
Optimal test accuracy
0.945
0.25
0.94
Optimal training
0.20
0.935
0.15
0.93
w/ batch norm
0.10
w/ batch norm
w/o batch norm
w/o batch norm
0.925
0.05
Zerolnit
Zerolnit
0.92
0.90
2
Batch size
Batch size
(a)
(b)
Figure 3: The performance of a 16-4 Wide-ResNee in the small batch limit, trained on CIFAR- 10 for 200 epochs. We perform a grid search over the learning rate and provide the mean performance the best 12 out of 15 runs. In contrast to the rest of this paper, we do not use ghost batch normalization, and evaluate the batch statistics over the whole batch. Batch normalization outperforms ZeroInit on the training loss. However underperforms Zerolnit on the test set when the batch size is very small.
batch normalization slightly outperforms ZeroInit for all depths, both techniques are able to train Wide-ResNets with one thousand layers to over 94% test accuracy within 200 epochs. Intriguingly, we also find that in both cases the optimal learning rate only depends weakly on the network depth.
When increasing the network depth, we found that the loss initialization becomes increasingly dominated by the L2 regularization loss, and we suspected that this may explain why Zerolnit is able to train very deep networks without rescaling the convolutional layers (as proposed by Zhang et al. (2019). Therefore in table 3, we also provide the test accuracy when training very deep wide residual networks using ZeroInit without 1.2 regularization. To our surprise, stable, and that the optimal learning rate remains independent of depth. We therefore conclude that rescaling is not necessary in practice. We also verify that is possible to train very deep networks solely by introducing the scalar multiplier in Zerolnit to each residual branch initialized to zero.
found that training still
Finally, we note that number of authors are interested in alternatives batch normalization for very small batch training (Wu & He 2018). In figure 3, we compare the performance of Zerolnit and batch normalization, when training a 16-4 Wide-ResNet for 200 epochs at range batch sizes 1 &&B B-= < 1024. In contrast to the rest of this paper, we do not apply ghost batch normalization here (Hoffer et al. (017). and instead evaluate the batch statistics over the training minibatch. expected, the final performance of Zerolnit is independent of the batch size in the small batch limit B S 256 (within one standard deviation), but the test accuracy achieved by batch normalization drops for batch sizes B S 64. Zerolnit may therefore provide alternative to batch normalization in small batch settings. Zerolnit still significantly underperforms batch normalization for large batch sizes B 2 256. We provide the corresponding optimal learning rates at each batch size in appendix A.
5 COMPARISONS ON IMAGENET
We now compare batch normalization, Fixup initialization and ZeroInit on ImageNet using ResNer-50. We compare performance on both the original ResNet-50-V V1 architecture (He et al. (0166)). and the modified ResNet-50-Vi 52- (He et al. 2016b). Like Wise-ResNets, ResNet-S0-VV contains an identity skip connection through all the residual blocks. Meanwhile ResNet-50-V 51 introduces a ReL.U at the end of every residual block after the residual branch and skip connection merge. While V2 is thought to perform better in most cases and is casier to train (He et al.) 2016b), VI is still oftenn used as benchmark. Since we wish to compare networks with and without an identity skip connection, we train using the original versions of ResNet-50-V V1 and ResNet-50- V2 (He et al. 2016a b). We do not use the modifications to ResNet-50-N V1 described by Goyal et al. (2017) ) although these are known to improve the validation accuracy by 1-2%. We use a ghost batch size of 256 (Hoffer al. 2017).
In table 4, we present the performance of batch normalization, Fixup and Zerolnit when training ResNet-500-2 (He al. 2016b) ImageNet range of batch sizes. We train for 90 epochs,
7

---

Under review as a conference paper at ICLR 2020
Table 4: The performance of ResNet-50-V2 on ImageNet. We use compute budget of 90 epochs and perform grid search to identify the optimal learning rate which maximizes the top-1 validation accuracy. We perform a single run at each learning rate and report both top-1 and top-5 accuracy scores. We use a drop probability of 0.2 for ZeroInit. Both Fixup and ZeroInit are competitive with batch normalization at small batch sizes, while batch normalization is better at larger batch sizes.
Batch Size
Batch Normalization
Fixup
ZeroInit without Dropout
ZeroInit
256
74.97 / 99. 74.93 / 92.13 74.88 / 91 .93
74.81 /9.84
74.93 / 91.89
75.56 / 92.35 75.46 / 92.53 72.65 / 90.70
1024
74.60 / 91.69
74.61 / 91.81 70.76 / 89.15
4096
73.02 / 90.59
Table 5: The performance of ResNet--0--1 on ImageNet. We use a compute budget of 90 epochs and perform grid search identify the optimal learning rate which maximizes the top-1 validation accuracy. We perform a single run at each learning rate and report both top-1 and top-5 accuracy scores. We use a drop probability of 0.2 for Zerolnit. Fixup performs well when the batch size is small, but significantly underperforms batch normalization when the batch size is large. Zerolnit performs poorly at all batch sizes, but its performance improves considerably if we add scalar bias before the final ReLU in each residual block (after the skip connection and residual branch merge).
Batch Size
Batch Normalization
Fixup
ZeroInit
ZeroInit + Scalar Bias
256
75.57 / 92.48
74.41 /9..65
70.05 89.16
75.23 99.37
1024
75.32 / 92.37 75.37 / 92.33
74.43 91.74
68.40 / 87.82
74.94 / 99.99
4096
72.37 / 90.28
68.19 / 87.92
70.82 89.60
and hold the learning rate constant for 45 epochs before decaying the learning rate by factor of 2 every 5 epochs. We find that both Fixup and Zerolnit are competitive with batch normalization when the batch size is small, but both underperform batch normalization when the batch size is large. These results match our earlier observations when training wide residual networks in section 3. The performance of Fixup is comparable to the performance of ZeroInit without dropout.
In table 5, we present the performance of batch normalization, Fixup and ZeroInit when training Resnet-50-VI (He et al. 2016a). As discussed above, and unlike both ResNet- V2 and Nese-ResNets (Zagoruyko & Komodakis 2016), this network introduces a ReLU at the end of the residual block after the skip connection and residual branch merge. We find that Fixup slightly underperforms batch normalization when the batch size is small, but considerably underperforms batch normalization when the batch size is large. Meanwhile ZeroInit significantly underperforms both batch normalization and Fixup at all batch sizes. This is not surprising, since designed Zerolnit for models which contain an identity skip connection through the residual blocks. We also provide additional results for modified version of ZeroInit, which additionally contains single scalar bias in each residual block, just before the final ReLU (after the skip connection and residual branch merge). We find that this scalar bias eliminates the gap in validation accuracy between Fixup and ZeroInit when the batch size is small. We conclude that only two components of Fixup are essential; initializing the residual branch at zero, and introducing a scalar bias after the skip connection and residual branch merge.
6 CONCLUSIONS
Batch normalization brings at least two practical benefits in residual networks. It increases the test accuracy, and it also substantially increases the largest stable learning rate which can be used during training. This second benefit enables efficient training with larger batch sizes, which can be exploited to reduce the wallclock time of training. However the increase in the largest stable learning rate cannot be the cause of the increase in the test accuracy, since we achieve similar test accuracies with smaller learning rates when the batch size is smaller. This demonstrates that empirical studies of batch normalization must consider a range of batch sizes. We also identify a simple modification to the initialization scheme, "ZeroInit", which allows us to train very deep (1000+ layer) residual networks without normalization. This scheme achieves similar test accuracies to batch normalization for reasonable batch sizes and outperforms batch normalization for very small batch sizes, but it underperforms batch normalization when the batch size is large. ZeroInit achieves similar validation accuracy to batch normalization when training ResNet-50--2 on ImageNet at batch size 1024.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 1xXiv:607..00 06400, 2016.
David Balduzzi, Marcus Frean, Lennox Leary, JP Lewis, Kurt Wan-Duo Ma, and Brian McWilliams. The shattered gradients problem: If resnets are the answer, then what the question? In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 342-350.
JMLR. org, 2017.
Nils Bjorck, Carla P Gomes, Bart Selman, and Kilian Q Weinberger. Understanding batch normaliza- tion. In Advances Neural Information Processing Systems, pp. 7694-7705, 2018.
Behrooz Ghorbani, Shankar Krishman, and Ying Xiao. An investigation into neural net optimization via hessian eigenvalue density. arXiv preprint 1XXv::991111115: 2019.
Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 1440-1448, 2015.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint arXiv:1706.008 02677, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645. Springer, 201 20.
Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generaliza- tion gap in large batch training of neural networks. In Advances in Neural Information Processing Systems, pp. 1731-1741. 2017.
Sergey loffe. Batch renormalization: Towards reducing minibatch dependence in batch-normalized models. In Advances in neural information processing systems, pp. 1945-1953, 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv: arXiv: 1502.03167 2015.
Stanislaw Jastrzebski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima in sgd. arXiv preprint arXiv:1711.04623 2017.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On turge-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609, 1609006609 2016.
Philipp Krähenbühl, Carl Doersch, Jeff Donahue, and Trevor Darrell. Data-dependent initializations of convolutional neural networks. arXiv preprint arXiv:1511.06855 2015.
Alex Krizhevsky, One weird trick for parallelizing convolutional neural networks. arXiv preprint arXiv: /4045597, 2014.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Ping Luo, Xinjiang Wang, Wenqi Shao, and Zhanglin Peng. Towards understanding regularization in batch normalization. In International Conference on Learning Representations, 2019. URL
https://openrevier net wwwp:eeeeeeeeeeeeeeeeeeeeee worummiddduHLKKSSS
Stephan Mandt, Matthew Hoffman, and David M Blei. Stochastic gradient descent as approximate bayesian inference. The Journal of Machine Learning Research, 18(1):4873-490 2017.

---

Under review as a conference paper at ICLR 2020
Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAl Dota Team. An empirical model of large-batch training. arXiv preprint arXiv:1812.06166 2018.
Dmytro Mishkin and Jiri Matas, All you need is a good init. arXiv preprint arXiv:1511.06422 2015.
Daniel S Park, Jascha Sohl-Dickstein. Quoc Le, and Samuel L Smith. The effect of network width on stochastic gradient descent and generalization: an empirical study. arXiv preprint arXiv:1905.03776, 2019.
Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization, 30(4):838-855, 1992.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision. 115(3):211-552, 2015.
Tim Salimans and Durk P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In Advances in Neural Information Prucessing Systems, pp- 901-909, 2016.
Karthik A Sankararaman, Soham De, Zheng Xu, W Ronny Huang, and Tom Goldstein. The impact of neural network overparameterization on gradient confusion and stochastic gradient descent. arXiv preprint arXiv:1904.000 2019.
Shibani Santurkar, Dimitris Tsipras, Andrew llyas, and Aleksander Madry. How does batch normal- ization help optimization? In Advances in Neural Information Processing Systems, pp. 2483-2493,
2018.
Christopher
Shallue, Jaehoon Lee, Joe Antognini, Jascha Sohl-Dickstein, Roy Frostig, and George Dahl. Measuring the effects of data parallelism neural network training. arXiv preprint arXiv:1811.03600 2018
Samuel L Smith and Quoe Le. bayesian perspective on generalization and stochastic gradient 2017.
descent. arXiv preprint
Samuel L Smith, Pieter-Jan Kindermans, Chris Ying, and Quoe Le. Don't decay the learning rate, increase the batch size. arXiv preprint arXiv:1711.00489, 2017.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv: 1607,00020 2016.
Shuang Wu, Guoqi Li, Lei Deng, Liu Liu, Dong Wu, Yuan Xie, and Luping Shi. L1-norm batch normalization for efficient training of deep neural networks. IEEE transactions neural networks and learning systems, 2018.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 3-19,20188 2018.
Greg Yang, Jeffrey Pennington, Vinay Rao, Jascha Sohl-Dickstein, and Samuel S Schoenholz. A mean field theory of batch normalization. arXiv preprint arXiv:1902.08122 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv: 160500600.071101 146, 2016.
Hongyi Zhang, Yann N Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization, arXiv preprint arXiv: Xii:11901.09321, 2019.
10

---

Under review as a conference paper at ICLR 2020
A THE OPTIMAL LEARNING RATES IN THE SMALL BATCH LIMIT
In figure 4, we provide the optimal learning rates when training a 16-4 Wide-ResNet for 200 epochs on CIFAR-I0 in the small batch size limit. We evaluate the batch statistics over the full training minibatch, which enables us to consider the limit B + 1 when using batch normalization. We note that the aceuracy and training loss each batch size was provided in the main text figure 3. Taken together, these two plots provide further evidence that larger stable learning rates do not explain why batch normalization performs well on the training set and the test set when the batch size is small.
w/ batch norm
w/ batch norm
Optimal learning rate (for test accuracy)
Optimal learning rate (for training loss)
batch
w/o batch
Zeroinit
Zerolnit
28
27
Batch size
Batch size
(a)
(b)
Figure 4: A 16-4 small batch limit. In order to study batch normalization the hatch size B 1, formulism batch normalization here, where the batch statistics are evaluated over the training minibatch. The learning rate which maximizes the test accuracy proportional the batch size when the batch size is small, and all three methods exhibit similar learning rates at the same batch size this limit. This demonstrates that the stability of batch normalization at large learning rates is not the reason why batch normalization performs better on the test set when the batch size is small. b) observe similar trends when we evaluate the optimal learning rates which minimize the training loss.
trained on CIFAR-10 10 for 200 epochs at range of batch sizes the
the original
11


Reviews:
[{"review_id": "rJePYYZWjr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper conducts extensive experiments to study batch normalization, a very popular technique for training a deep convolutional network and its relationship with learning rate and batch size. In addition, the authors also propose a new initialization scheme, “ZeroInit”, to train a deep ResNet for better test accuracy. This is a very empirical study and the authors also show extensive experimental results. However, I do not see any novel findings in this study. Mostly this paper confirms the results of previous studies. The experimental results do not show much advantage of ZeroInit either. Overall, it is unclear what is the major novel contribution in this paper.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "r1e_OsV6tS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper conducts extensive experiments to verify two practical benefits of batch normalization. i) It increases the final test accuracy and the largest stable learning rate; ii) it enables efficient training with larger batches and a larger learning rate. In addition, the authors propose a new initialization scheme, “ZeroInit”, to train a deep ResNet to improve the test accuracy. My detailed comments are as follows.\n\n\nPositive points:\n\n1. The experiments are sufficient. In this paper, the authors conduct extensive experiments to explore the benefits of batch normalization, and verifies the effectiveness of the proposed “ZeroInit”.\n\n2. The method is effective in some cases. Specifically, the proposed “ZeroInit” outperforms batch normalization when the batch size is small, and it is competitive with batch normalization when the batch size is not too large.\n\nNegative points:\n\n1. The importance and novelty of the empirical study should be emphasized. The practical benefits of batch normalization can be also found in other papers. For the first benefit, most studies (Bjorck et al. 2018) have found that batch normalization is able to improve the test accuracy. For the second benefit, batch normalization requires a large batch size and a large learning rate (Santurkar et al., 2018). Therefore, what is the difference between this paper and others? More critically, it is necessary to explain why batch normalization has these benefits. It would be better to provide empirical or theoretical justifications to support these. \n\n2. The motivation of the proposed “ZeroInit” is not clear. (Balduzzi et al., 2017) states that “the correlations can be preserved by initializing deep networks close to linear functions”. It is not clear how “ZeroInit” preserves the correlations? \n\n3. Why initialize the scalar multiplier and biases to zero? What are the benefits of the zero initialization? Actually, the scalar multiplier and biases can be randomly initialized. When they are randomly initialized, what is the performance of the initialization? It is an important baseline to justify the effectiveness of the proposed initialization method. \n\n4. The technical details of “ZeroInit” are not clear. It would be better to express the proposed initialization “ZeroInit” in the mathematical formulation.\n\n5. The proposed initialization “ZeroInit” is designed for deep ResNets. How to extend it to the other deep neural networks?\n\n6. This paper states that “the empirical success of batch normalization …improves the conditioning of the loss landscape. However, our results conclusively demonstrate that this is not the case”. Does it mean that batch normalization does not improve the conditioning of the loss landscape? However, the empirical results cannot justify this statement and explain the success of batch normalization.\n\n7. Some results of the figures are missing. In Figure 1, the experimental results of w/o batch norm with varying batch sizes (2^0 ~ 2^5) are missing. Similarly, Figure 2 also has missing results. Please provide more discussions about these missing results.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BylOW51AKr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The name \"ZeroInit\" is very confusing, because that is how FixUp was called initially https://openreview.net/forum?id=H1gsz30cKX , perhaps the authors should consider a different name. I will call it \"NewZeroInit\" in my review to avoid confusion.\n\nThe paper focuses on training image classification networks without batch normalization. The authors claim that effectiveness of batch normalization, and methods which attempt to eliminate it, should be tested on a wide range of learning rates. On experiments performed on CIFAR they find that batch normalization is able to achieve high accuracy even with very high learning rates, in line with Goyal et al. 2017. Based on this, they propose a simplification of FixUp for image classification, in which they remove the need in progressive scaling of initialization, and propose to remove weight decay regularization, while adding dropout on the last layer. This \"NewZeroInit\" is tested on ImageNet and compares favorably to batch normalization and FixUp.\n\nThe closest studies are FixUp and Goyal et al. 2017, with the difference that FixUp studies both image classification ResNet and seq2seq approaches in the absence of batch normalization, and Goyal et al. show a wide range of large scale experiments on full scale ImageNet, whereas \"NewZeroInit\" studies small scale CIFAR dataset. It is thus unclear if \"NewZeroInit\" transfers to seq2seq.\nThere is also \"Bag of Tricks for Image Classification with Convolutional Neural Networks\" by He et al 2018 (missing citation) which evaluates a similar set of tricks on ImageNet ResNet-50 with batch normalization. In particular, they show that removing weight decay from BN bias and setting scaling gamma to 0 initially significantly improves the results.\n\nOn page 4 the authors say \"Given access to sufficient hardware, this will enable practitioners to dramatically reduce wallclock time of training (Goyal et al.)\". It is not clear what they mean, since Goyal et al. already enabled the reduction by increasing learning rate and minibatch size on ImageNet, whereas the results authors show are on small CIFAR dataset.\n\nOn page 5 the authors mention that they introduce bias to each convolution and classification layer, which is surprising because it is a standard way to composing a convolutional network.\n\nOverall, the most significant contributions of the paper are:\n - a study of minibatch size on CIFAR\n - removing weight decay from FixUp on ImageNet\n\nAlso, I am interested in the following results:\n - clear comparison of FixUp with \"NewZeroInit\" for image classification\n - ImageNet ResNet-50 results with dropout regularization in the final layer\n - ImageNet ResNet-50 results with FixUp, dropout regularization and no weight decay.\n - (optionally) seq2seq with NewZeroInit instead of FixUp.\n\nWithout these results it hard to judge the novelty and contributions of the paper, so I propose reject.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is rejected based on unanimous reviews.
Decision: Reject