Deconstructing the Regularization of BatchNorm
Authors: Yann Dauphin
Ekin Dogus Cubuk
Venue: ICLR 2021 Poster (2021)
Keywords: deep learning
batch normalization
regularization
understanding neural networks

Abstract: Batch normalization (BatchNorm) has become a standard technique in deep learning. Its popularity is in no small part due to its often positive effect on generalization. Despite this success, the regularization effect of the technique is still poorly understood. This study aims to decompose BatchNorm into separate mechanisms that are much simpler. We identify three effects of BatchNorm and assess their impact directly with ablations and interventions. Our experiments show that preventing explosive growth at the final layer at initialization and during training can recover a large part of BatchNorm's generalization boost. This regularization mechanism can lift accuracy by $2.9\%$ for Resnet-50 on Imagenet without BatchNorm. We show it is linked to other methods like Dropout and recent initializations like Fixup. Surprisingly, this simple mechanism matches the improvement of $0.9\%$ of the more complex Dropout regularization for the state-of-the-art Efficientnet-B8 model on Imagenet. This demonstrates the underrated effectiveness of simple regularizations and sheds light on directions to further improve generalization for deep nets.

Published as a conference paper at ICLR 2021
DECONSTRUCTING THE REGULARIZATION OF BATCH- NORM
Yann N. Dauphin Google Research
Ekin D. Cubuk
Google Research
ynd@google e
cubuk @google.ccoooooooo com
ABSTRACT
Batch normalization (BatchNorm) has become standard technique in deep learn- ing. Its popularity is in no small part due to its often positive effect on generalization. Despite this success, the regularization effect of the technique is still poorly under- stood. This study aims to decompose BatchNorm into separate mechanisms that are much simpler. We identify three effects of BatchNorm and assess their impact directly with ablations and interventions. Our experiments show that preventing explosive growth at the final layer initialization and during training can recover a large part of BatchNorm's ' generalization boost. This regularization mechanism can lift accuracy by 2.9% for Resnet-50 on Imagenet without BatchNorm. We show it is linked to other methods like Dropout and recent initializations like Fixup- Surprisingly, this simple mechanism matches the improvement of 0.9% of the more complex Dropout regularization for the state-of-the-art Efficientnet-B8 model on Imagenet. This demonstrates the underrated effectiveness of simple regularizations and sheds light directions to further improve generalization for deep nets.
INTRODUCTION
Deep learning has made remarkable progress on a variety of domains in the last decade. While part of this progress relied on training larger models larger datasets, it also depended crucially the development new training methods. A prominent example of such a development is batch normalization (BatchNorm) (Toffe and Szegedy 2015), which has become standard component of training protocols. For example, state-of-the-art models in image recognition (Szegedy et al. 2017 7]
He al. 2016 Tan and Le 2019). object detection (He al., 2019 Du et al.] 2019)). and image segmentation (Chen et al. 2017) all use BatchNorm. Despite its prominence, the mechanisms behind
BatchNorm's effectiveness are not well-understood (Santurkar et al. 2018 Bjorck et 2018 Yang
et al. 2019).
Perhaps at the core of the confusion that BatchNorm has many effects. It has been correlated to reducing covariate shift (Toffe and Szegedy, 2015), enabling higher learning rates (Bjorck et al. 2018). improving initialization (Zhang et al. 2019), and improving conditioning (Desjardins et al. 2015). to name a few. These entangled effects make it difficult to properly study the technique. In this work, we deconstruct some of the effects of BatchNorm in search of much simpler components. The advantage of this approach compared to previous work is that it allows going beyond correlating these effects to BatchNorm by evaluating their impact separately. The mechanisms we consider in this work purposefully simple. These simpler mechanisms easier to understand and, surprisingly, they are competitive even at the level of the state-of-the-art.
Our contributions can be summarized as follows:
How does normalization help generalization? We isolate and quantify the benefits the different effects of BatchNorm using additive penalties and ablations. To our knowledge, we are the first to provide empirical evidence that BatchNorm's 's effect of regularizing against explosive growth at initialization and during training recover large part of its generalization boost. Replicating this effect with Fixup initialization and the proposed additive penalty improves accuracy by 2.9% for Resnet-50 without BatchNorm.

---

Published as a conference paper at ICLR 2021
2. Links to Fixup and Dropout. We draw novel connections between the regularization the final layer, Dropout regularization, Fixup initialization and BatchNorm.
3. Simplicity in regularization. The mechanism we identify can be useful as a standalone regularization. It produces 0.9% improvement on the Efficientnet B8 architecture, matching the more complex Dropout regularization.
DECOMPOSING THE REGULARIZATION EFFECTS OF BATCH NORMALIZATION
In this section, we break BatchNorm into different mechanisms that can be studied separately. BatchNorm (loffe and Szegedy) (013)) a technique that accelerates training by standardizing the intermediate activations of a deep network. It achieves this standardization by
Output
Fanddreell 12 }
Output Layer
using explicit normalization instead of relying ization. While BatchNorm has been correlated to many effects, it is still unclear which effect, if any, explains most of its generalization boost.
additive regular-
inbessing
Feeture Embedding
Embedding Network
Dalenform or Soucaid sing Loes
The effects we evaluate are the implicit regularizing effect on the norms at the final layer, and also its primary effect of standardizing the intermediate layers. In order to test these purposefully simple mechanisms we will rely on ablations and additive penalties. The use of additive penalties allows us disentangle these effects where we control for the positive effect of BatchNorm on initialization by using the recently proposed Fixup initializer (Zhang et al. 2019).
Input
Figure 1: Diagram of neu- ral network showing where the mechanisms operate.
2.1 REGULARIZING AGAINST EXPLOSIVE GROWTH IN THE FINAL LAYER
First, we characterize the implicit effect of normalization on the final layer. Consider a neural network of the form NN(x) = WEmb(x) with loss L where is the final weight matrix in the model, and Emb(x) R + R" is a feature embedding network with layers. Let us take the common case where Emb(() = Swi (hh(yBattttt chNo rm reemooo) where PreEmb x) is the output of residual network, Swi sx()) = xa (px) is the Swish activation (Ramachandran al. 2017 Elfwing al., (018)) with scalar parameter (typically denoted B), and BatchNorm parameters B. BatchNorm makes weight decay regularization y, approximately cquivalent to an additive penalty on the norm on the feature embedding
/ is the input of the network, W R
33)
[(NN(x)) 21 ||llee 2 ((NN())) + [[[[[mb(()) O(p)).
See the Appendix A for the derivation. It means that the norm of the BatchNorm parameters alone is enough to directly control the norm of the feature embedding. It is a guarantee that the norm of the feature embedding cannot grow explosively during training as long as these parameters are small, This regularization effect of BatchNorm can occur even without explicit weight decay due to the tendency of stochastic gradient descent to favor low norm parameters (Wilson al.) (017))
This equivalency does not hold without BatchNorm because the activation of the embedding network become an important factor in the norm of the feature embedding [[l>||2 + |
E [[[>PreEmb(x) + [|] in general). Indeed, (Balduzzi et al., 2017; Gehring et al. 2017 Zhang
ct al. 2019) have shown that the activations of residual networks without BatchNorm tend explode
exponentially in the depth of the network initialization. This results in an extremely large embed-
ding norm, even though the parameters are relatively small. We confirm experimentally in Section that networks without BatchNorm have much larger feature embedding norms.
Feature Embedding 1.2 (EI.2) We propose to assess the effect of this regularization mechanism by isolating as the following additive penalty
REL2(NN) " [[]]Emb(x)]11
2

---

Published as a conference paper at ICLR 2021
Adding this regularization the loss allows us test the impact of this mechanism independently on networks with and without BatchNorm. We dub this regularization embedding L2 for short in the following sections as it is L2 with a metric that is at the embedding network function level, with no additional penalties on the intermediate layers. is applied right before the classification layer, in other words right after the final average pooling layer for residual networks. We will with our experiments that this simple regularization can in large part recover the regularization boost of BatchNorm, has links several known methods and practically useful even the level of the state-of-the-arr
Functional L2 (FL2) Regularizing the feature embedding has an implicit effect on the final output norm (X]]]] I 1/2 [| # (NL)(NN)). In order to test the impact of this effect, we will also evaluate a direct penalty on the norm of the final output
RFL2(NN)= 1 (1
(3)
We dub this regularization functional L2 for short in the following sections as is L.2 with a metric that is at the full network function level. It is applied the logits of the model for classification. In Section 4.44 we investigate whether it is regularizing this norm or the feature embedding norm that is more closely correlated to better generalization.
2.2 STANDARDIZING THE INTERMEDIATE ACTIVATIONS OF THE MODEL
As baseline mechanism, we will also consider an additive penalty that encourages the normalization of every intermediate layer called standardization loss (Collins et al. 2019). This is a useful reference to gauge the effectiveness of the regularization on the embedding described in the previous sub-section. also helps disentangle the side-effects of normalization. The penalty is
DKL (P()))))XX(x 1)) = 2 Σ (n} + o² log of -1
(4)
where (4 is the mean and o; is the variance of intermediate layer of the network. These the same statistic that are computed by BatchNorm and a penalty is added to the loss for all the intermediate layers. This regularization has been considered in (Collins al. 2019). They found that it accelerated learning but fell significantly short of the generalization boost of batch normalization. However, they did not account for the positive effect of normalization on initialization. We correct for this in our experiments using the recently proposed Fixup initialization (Zhang et al. 2019).
3 DRAWING LINKS TO OTHER METHODS
In this section, we will draw connections between the mechanisms considered for BatchNorm and other methods.
3.1 DROPOUT REGULARIZATION
Dropout (Hinton et al. 2012 is regularization that prevents overfitting by randomly omitting subsets of features during training. Despite its early popularity, its use has declined with the rise of batch normalized convolutional networks. Ghiasi et al. (2018), and ZZagoryyko and Komodakis (2016), find that it produces comparatively much smaller improvements when applied to the intermediate layers of such networks. However, Tan and Le (2019) has shown that state-of-the-ard results can be obtained by applying Dropout only at the input of the final layer of the network. Interestingly, we can relate this particular use of Dropout to BatchNorm. |Wang and Manning (2013) have shown that Dropout with MSE loss can be isolated as an additive regularization when applied at the last layer
Drepout
Σ E (WDropott
=w)²
N
Σ WEmb(x)) - /
N
where the additive Dropout regularization is R (NN) = tr ddigg (W (E(Embx)) (x) (x)7) WT) and À is the strength of the penalty. In this formulation, we can that Dropout is related to the
3

---

Published as a conference paper at ICLR 2021
mechanisms in Section 2.1 as follows
[ H
K (NN) RDmpou (NN)
REL2(NN).
(5)
The approximate relationship functional L2 relies on the assumption that the features are relatively decorrelated. To our knowledge this close but simple relationship between the expected norm of the output and Dropout with MSE loss had not been noted before. The upper bound with embedding L2 gives us guarantee on Dropout robustness when training with BatchNorm and weight decay. In some sense, this means that networks with BatchNorm already incorporate a regularization effect similar to that conferred by Dropout. This can explain why networks with BatchNorm have tended to benefit relatively less from Dropout.
The approximate relationship to functional L2 can be extended to networks with cross-entropy by using Taylor expansion and assuming the network has low-confidence. This assumption is likely only correct at initialization. In comparison, a related upper-bound can be found for embedding L2 using Taylor expansion. We will see in Section [45 that embedding and functional L.2 can match or exceed Dropout for tfeteof-the-- art architectures Imagenet such Efficientnet (Tan and Le 2019). This is surprising because using an additive penalty at the final layer is much simpler than Dropout.
3.2 FIXUP INITIALIZATION
Fixup (Zhang et al. 2019) is a recent initialization method for the ubiquitous residual networks. Recall that residual networks without normalization have explosive dynamics with conventional initializations (Balduzzi et al. 2017 Gehring et al.) 2017] /Zhang et al., 2019). In fact, the output scale of the networks grows exponentially with their depth This means that the embedding and functional L2 at initialization are
REI2(NNNNOBN) RF122NNNOON E O((2)
(6)
where t is the training iteration. These explosive dynamics can be very detrimental to learning (Balduzzi et (017)). Fixup aims to stabilize the training of very deep residual networks without normalization. It does so by initializing the network so that the output does not change explosively in the first gradient step, that |iuN (x) NN (x)] E (1)) where 7) the learning rate. They show this stabilizes learning and the output scale. We find that Fixup initialization minimizes the initial penalties when compared to conventional initialization
REL2 (N Ry L2
(NN E (()
since it initializes residual branches to zero. Most importantly, Fixup ensures this does not grow explosively in the first few steps. Zhang al. (2019) observe that Fixup can improve generalization compared to conventional initialization, but requires strong regularization to be competitive to networks with normalization. In Section 4.3] will see that regularizing the functional L2 not just at initialization but throughout training remarkably improves generalization.
4 EXPERIMENTS
In this section, we evaluate the regularization mechanisms on set of architectures for two challenging benchmarks.
4.1 DATASETS
CIFAR-10 CIFAR-10 10 has 50k samples for training and 10k samples for test evaluation. We tune hyperparameters using 5k of the training samples validation set. We then train a final model using the whole training set of 50,000 samples with the best hyperparameters for 10 different seeds, and report the median test accuracy. Each model is trained single GPU. The robustness is reported as the accuracy on CIFAR- dataset (Hendrycks and Dietterich] 2019). averaged over all corruptions and their severities.
SVHN SVHN has 73,257 samples for training and 26,032 samples for testing (note that we do not consider the 531,131 "extra" samples). We tune hyperparameters using 3,257 of the training samples
4

---

Published as a conference paper at ICLR 2021
validation set. We then train final model using the whole training set of 73,257 samples with the best hyperparameters and report the median test accuracy for 10 different random initializations.
Imagenet Imagenet is large scale image recognition dataset with 1.2M examples and 1000 classes. Following the past work, we report classification accuracy on the validation set of 50k examples.
4.2 IMPLEMENTATION DETAILS
WideResnet We train WideResNet-28-10 architecture (Zaorrukoo and Komodakis 2016) CIFAR-I0, which is one of the most commonly used architectures for this dataset. We train the model for 200 epochs using cosine learning rate decay with batch size of 128. We use the standard data augmentation of horizontal flips and pad-and-crop. For models without BatchNorm, we use Fixup initialization. We use the activation function Swish (Ramachandran et al. 2017). with the p value initialized to O.
Resnet-50 Resnet-50 (He et al. 2016 has quickly become standard architecture evaluate Imagenet. Our implementation makes use of the improvements proposed by hy|Goyal et al. (2017). Only the networks trained without BatchNorm use Fixup initialization (Zhang et al.) 2019). In order to improve results for the standardizing loss, we found it was useful to use a special case of Fixup where the residual branches are not initalized to zero: when the standardization loss is used we initialize the last layer of each residual block inversely proportional to the square root of the depth. The models are trained for 90 epochs with a batch size 512 and precision of float32. All results are the average of two seeds. The coefficient for the standardizing loss is cross- validated in the range from 10-7 to 10-5 logarithmically. We found that higher coefficients led to divergence at high learning rates. The coefficient for embedding and functional L2 was found in the range from 0 to with increments of 0.1.
Efficientnet Efficientnet (Tan and Le 2019) is the state-of-the-art architecture on Imagenet the time writing. We will evaluate on the biggest version trained with RandAugmen (Cubuk et al. 2019b) data augmentation and without additional data, called the B8 (Xie et al. 2020). We follow the implementation of Tan and Le (2019) and the ayper-parameters they found optimal. We use early stopping based held-out validation set of 25022 images. All results reported averaged over two random seeds.
4.3 RESULTS: ABLATION OF BATCHNORM AND INTERVENTIONS
Table I: Accuracy on CIFAR-I0 dataset for Wide-Resnet-28-10.
Method
Accuracy (%)
Robustness (%)
Baseline (WRN-28- 10 without BatchNorm)
94.9 0.1
82.8 = 0.1
Baseline + BatchNorm
96.1 + 0.1
82.8 + 0.2
Baseline + Standardizing Loss (A = 1e-7)
95.0 + 0.2
83.4 + 0.2
Baseline + Functional 1.2 (A 1.0)
95.5 + 0.2
84.1 + 0.3
Baseline + Embedding L2 (A = 0.5)
95.5 + 0.2
84.1 + 0.2
Table 2: Accuracy on Imagenet dataset for Resnet-50. Functional 1.2 with proper initialization allows us to match the generalization of networks trained with batch normalization.
Method
Accuracy (%) 73.1 + 0.1
Baseline (Resnet-50 without BatchNorm)
Baseline + BatchNorm
75.7 + 0.0
Baseline Standardizing loss (A = le-5)
74.7 0.2
Baseline + Functional L.2 (A = 0.4)
76.0 0.1
Baseline Embedding L2 (A 1)
76.0 - 0.0
First, we measure the impact of the regularization mechanisms under consideration for networks where we have removed NarchNorm. We also include results for network trained with BatchNorm
5

---

Published as a conference paper at ICLR 2021
for reference. On CIFAR-10 (Table and Imagenet (Table 2) we observe significant gap between the baseline network and the network trained with BatchNorm. The baseline networks benefit from improved initialization with Fixup, which (Zhang al. 2019) find stabilizes training at higher learning rates and increases accuracy by up to 2.9% on Imagenet without BatchNorm. However, the baseline networks do not have extra regularization during training.
We find that the standardization regularization produces modest robustness improvements on CIFAR- 10 but not for the accuracy, and moderate improvements on Imagenet. We see that this regularization does not replicate BatchNorm's 's generalization boost, even with proper initialization. This matches the results Collins et al. (2019) which considered the use of this regularization without accounting for the initialization effects of BatchNorm. We do not believe this indicates that standardization is not an important effect but rather reflects a failure of this particular approach. There are a number of possible explanations, including that this type of constraint is difficult for the optimizer to balance successfully with the classification loss. Indeed, we see that only comparatively small regularization coefficients were found to produce stable training with high learning rates.
In comparison, we note that replicating standardization's 's implicit effect on the norms at the final layer produces better results on both CIFAR-10 and Imagenet. While it is enough to match the generalization boost of BatchNorm on Imagenet, this is not the case on CIFAR-10. 10. However, the scale of the improvement suggest that controlling the norm of the output through the norm of the feature embedding is an important benefit of standardization and as a result BatchNorm. In particular, embedding and functional L.2 produces improvement of close 3% Imagenet and 0.6% on CIFAR-10. 10. We analyze this results more closely in Section 4.4
To further test the effect of output norm regularization on small dataset, we experiment with the SVHN core set. The Wide Reiie-RRenet-8--. 10 model with BatchNorm achieves 96.9% test accuracy on this dataset, whereas the same model without BatchNorm (but with Fixup initialization) achieves 95.5%. If we utilize the Functional L2 or the Embedding L2 regularization, we reach 96.5% accuracy. Similar to the models on CIFAR- 10, output norm regularization offers significant improvement in generalization (1.0%) for model without BatchNorm.
Bjorek et al. 2018 have shown that BatchNorm allows using larger learning rates without divergence. It is thus interesting to take note of how the optimal learning rate changed when we removed BatchNorm. We observe that in our experimental setup on Imagenet the optimal base learning rate with BatchNorm of 0.1 remains optimal with embedding and functional L2. In our experimental setup on CIFAR-10, the optimal learning rates increased from 0.1 with BatchNorm to 0.2 with functional L2.
While BatchNorm is effective at improving accuracy, we find that it does not improve the robustness of the baseline model to distortions applied at test-time. We measure the robustness of the models by evaluating their accuracy on the 15 different corruptions described by |Hendrycks and Dietterich (2019). which include corruptions from noise, blue, weather, and digital categories. Models trained with standardizing loss have 1.1 % lower accuracy than the models trained with BatchNorm, but their robustness is 0.6% higher. This suggests that while BatchNorm might be improving robustness due to the standardization of the intermediate layers, it is reducing robustness by a larger amount due to its other effects (negative effects of BatchNorm on adversarial robustness have been discussed by Galloway et al. )2.1)). Functional L2 is able to improve both accuracy and robustness (by 1.3% over the baseline model and the BatchNorm model), by its ability to reduce the output norm.
4.4 ANALYSIS: HOW MUCH DOES THIS EXPLAIN THE REGULARIZATION OF BATCHNORM?
In the previous section, we show that the mechanisms considered are effective. This section investi- gates their relationship to BatchNorm.
How much does Batch regularize the norm at the last layer? Figure Figure column) shows learning curves for networks with and without BatchNorm. As the networks without BatchNorm use a suitable initialization, we observe that the networks converge at similar speeds at first. However, we see a generalization gap emerge later in training on both datasets. We see that this gap is associated with larger growth throughout training in the norm of the final output and feature embedding for networks without BatchNorm. This confirms that BatchNorm prevents explosive growth in the representation from cascading to the output during training. This also makes clear that these quantities

---

Published as a conference paper at ICLR 2021
103.0
97.5
BatchNorm
Without BatchNorm
12
BatchNorm
Without BatchNorm
Accuracy (%6)
05.0
Embedding Norn
1.4
Outputts Norm
10
60.5
90.0
83.0
0.6
Without BatchNorm
0.4
82.5
BatchNomm
80.0
50
100 Epachs
150
200
50
100 Epochs
150
200
50
100 Epochs
150
200
80
120
360
Without BatchNorm BatchNorm
70
100
140
Accuracy (%6
Embedding Norm
Outputs Norm
120-
60
Without BatchNorm
Without BatchWorm
BetchNorm
40
20
20
20
40
80
20
40
60
80
60
Epochs
Epochs
Epochs
Figure 2: Statistics of models with and without BatchNorm on CIFAR-10 (top) and Imagenet (bottom) showing the accuracy (left), feature embedding norm (middle), mean output norm (right). Without additional regularization models without BatchNorm display explosive growth in the feature embedding norm. This correlates with poor generalization.
must be controlled throughout training and not just at initialization. And given that these networks already trained with well-tuned parameter L2, it is evidence that parameter L2 is not sufficient to control these quantities without NorchNorm.
76.0
100
140
Embedding L2
75.5
Embadding Norm
120
Functional L2
BetchNorm
Accuracy (86)
Output Norm
Embedding L2
300
74.
60
Functional L2
BatchNorm
74.0
Embedding L2
735
Functional
BatchNorm
73.0
Penalty Coefficient
Penalty Confficient
Penalty Coefficient
Figure 3: Effect of increasing the penalty coefficient for Imagenet the accuracy (left), feature embedding norm (middle), and output norm (right). We see that embedding and function L2 recover most of BatchNorm's generalization boost when the embedding norm matches that of BatchNorm networks.
Does regularizing the norm to the same level BatchNorm improve results? In Figure 3 we see that the accuracy increases with the coefficient of functional or embedding L2. Both the output and feature embedding norm reduce as expected. We see that comparable output norms, networks with functional I.2 perform much worse than BatchNorm. Indeed, the coefficient of functional L. must be quite small to allow for such high norms. In contrast, we see that the methods perform comparably when the norm of the feature embedding is similar. These results indicate that the coefficient must be high enough to control the reduce the norm of the feature embedding as much NatchNorm to obtain good results. This is consistent with the interpretation that the methods prevent explosions in the residual representation of the network from cascading to the output. Thus controlling the norm of the feature embedding appears more tightly related generalization than the
7

---

Published as a conference paper at ICLR 2021
norm of the final output. Like other regularizations such classical L2, a single regularization does not tell the whole story and it matters how these quantities are minimized.
Can we disentangle BatchNorm's effeet initialization to its effect on the norm at the last layer? Can we train networks with only a single BatchNorm layer at the feature embedding level? Next, evaluate Resnet-50 with only single BatchNorm layer at the level of the final feature embedding on Imagenet (added before the average pooling layer). Let us first consider the case without Fixup initialization. This bad initialization leads an explosion at the level before the BatchNorm layer, which has a norm of 1.8 x 10 = 3.8 x 10 at initialization. Normally this network would not train at reasonable learning rates but thanks to the BatchNorm at that final layer trains to 73.8 + 0.1 accuracy. This a slightly higher accuracy than with Fixup only but lower than with BatchNorm in every layer. This shows that preventing explosion in the feature embedding is sufficient to allow training even without proper initialization. In combination with Fixup, the accuracy rises to 74.9 + 0.0. This network contains over 100x less normalization layers, but recovers nearly 70% of the generalization improvement with respect to Fixup. It is corroboration of the previous results with additive penalties that proper initialization and controlling the feature embedding norm recovers a large part of the boost of BatchNorm.
Table 3: Accuracy drop from 50% Dropout test time for Resnet-50 models trained without Dropout on Imagenet. Networks regularized with BatchNorm or embedding 1.2 are considerably more robust to Dropout than the baseline.
Method
Accuracy Drop (%)
Baseline
22.8 + 0.1
BatchNorm
-11.0 + 0.3
Embedding L2
-18.5 + 0.1
How much do these methods increase natural robustness to Dropout? Section 3.1 establishes link between BatchNorm, embedding L2 and Dropout, with link to functional 1.2 at initialization only. We showed that a low feature embedding norm should correlate to Dropout regularization. Table shows the robustness to Dropout at test time for networks that were not trained with Dropout. We see that networks with embedding L2 suffer considerably less from Dropout corruption at test compared to the baseline. Similarly, the use of BatchNorm also boost robustness to Dropout, While it had been observed that Dropout regularizing during training is less effective for networks with BatchNorm, this is the first time it has been shown that BatchNorm networks have a naturally higher robustness to Dropout.
4.5 REGULARIZZATON WITH BATCHNORM
In this section, we evaluate using the regularizations on the final layer with BatchNorm. As explained in Section 2.1 BatchNorm already reduces the embedding I.2 to some degree. However, BatchNorm and weight decay by themselves does not make it easy to control the strength of the regularization. On one hand, it would require using independent weight decay coefficients between parameters y and the other parameters, and on the other they only approximately reduce these quantities. Embedding L. allows us to minimize this quantity directly.
Table|Ishow our results for Efficient B8. This model produces state-of-the-arr results for Imagenet without additional data sources. We find that embedding and functional L2 boosts accuracy by 0.9% compared to Efficientnet B8 without Dropout. This improvement matches that of Dropout. is surprising that regularizing such simple additive penalty improves results for a state-of- the-art architecture at the same level as the more complicated Dropout regularization. This is even more surprising considering that the baseline already regularized by powerful methods like RandAugment (Cubuk et al. 2019b) and StophasticDepth (Huang et al. 2016). This suggests that further investigation of simple regularizations could be beneficial even for state-of -ftth--rrr models.
5 CONCLUSION
In this work, we have quantified a number of the regularization effects of BatchNorm. Surprisingly, empirical study shows that preventing explosion of the feature embedding norm at initialization
8

---

Published as a conference paper at ICLR 2021
Table 4: Accuracy Imagenet dataset for Efficienet-B8.
Method
Accuracy (%)
ResNet-50 (He et al.
2016
76.0
DenseNet-16y (Huang et al.
2017)
76.2
ResNet-1122 He et al. (0116)
77.8
Inception-v3 Szegedy et
2017
78.8
ResNeXt-100 (Xie et al.
201
80.9
NASNet-A (Zoph et al.) 2018
82.7
AmocbaNet-C (Cubuk et al.
2019a
83.5
GPipe (Huang al.,, 2019)
84.3
Efficientnet B8 without Dropout
84.3 0.0
Efficientnet B8 with Dropout at last layer Efficientnet B8 without Dropout Functional L2
85.1 +.0
85.1 + 0.1
Efficientnet B8 without Dropout + Embedding L2
85.2 + 0.1
and during training can recover a large part of the generalization improvement of BatchNorm. Further- more, this much simpler regularization produces improvements state-of-the-art architectures such as Efficienet-BSS We hope that these results will help guide the way to more effective regularization methods.
REFERENCES
Balduzzi, D., Frean, M., Leary, L., Lewis, J., Ma, K. W.-D., and McWilliams, B. (2017). The shattered gradients problem: If resnets are the answer, then what is the question? In Proceedings of the 34th International Conference on Machine Learning Volume 70, pages 342-350. JMLR. org.
Bjorck, N., Gomes, C. P., Selman, B., and Weinberger, Q. (2018). Understanding batch normaliza- tion. In Advances in Neural Information Processing Systems, pages 7694-7705.
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., and Yuille, A. (2017). Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4):834-848.
Collins, J., Balle, J., and Shlens, (2019). Accelerating training of deep neural networks with standardization loss. arXiv preprint arXiv: 009500
Cubuk, F. D., Zoph, B., Mane, D., Vasudevan, V., and Le, Q. V. (2019a). Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 113-123.
Cubuk, D., Zoph, B., Shlens, J., and Le, Q. V. (2019b). Randaugment: Practical data augmentation with no separate search. arXiv preprint arXiv: /909. /3719.
Desjardins, G., Simonyan, K., Pascanu, R., et al. (2015). Natural neural networks. In Advances in Neural Information Processing Systems, pages 2071 2071-2079.
Du, X., Lin, T.-Y., K, Jin, P., Ghiasi, G., Tan, M., Cui, Y., Le, Q. V., and Song. X. (2019). Spinenet: Learn ing permuted backbone for recognition and localization. arXiv preprint arXiv:
Elfwing, S., Uchibe, E., and Doya, K. (2018). Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural Nenworks, 107:3-11.
Galloway, A., Golubeva, A., Tanay, Moussa, M., and Taylor, G. W. (2019). Batch normalization is cause of adversarial vulnerability. arXiv preprint arXiv: 1905.0200...11.
Gehring, J., Auli, M., Grangier, D., Yarats, D., and Dauphin, Y. N. (2017). Convolutional sequence to sequence learning. arXiv preprint /00.03227
Ghiasi, G., Lin, T.-Y,, and Le, Q. V. (2018). Dropblock: A regularization method for convolutional networks. In Advances in Neural Information Processing Systems, pages 10727-10737.

---

Published as a conference paper at ICLR 2021
Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., and He, K. (2017). Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint 1rXiv:7706.06677.
He, K., Girshick, R., and Dollár, P. (2019). Rethinking imagenet pre-training. In Proceedings of the IEEE International Conference Computer Vision, pages 4918-4927.
He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778.
Hendrycks, D. and Dietterich, T. (2019). Benchmarking neural network robustness common corruptions and perturbations. arXiv preprint arXiv: 122661.
Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. R. (2012). Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0588
Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q. (2017). Densely connected con- volutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4700-4708.
Huang, G., Sun, Y., Liu, Z., Sedra, D., and Weinberger, K. Q. (2016). Deep networks with stochastic depth. In European conference on computer vision, pages 646-661. Springer.
Huang, Y., Cheng, Y., Bapna, A., Firat, O., Chen, D., Chen, M., Lee, H., Ngiam, J., Le, Q. V., Wu, Y., et al. (2019). Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Advances in Neural Information Processing Systems, pages 103-112.
loffe, S. and Szegedy, C. (2015), Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167.
Ramachandran, P., Zoph, B., and Le, Q. V. (2017). Searching for activation functions. arXiv preprint arXiv:1710.05941.
Santurkar, S., Tsipras, D., Ilyas, A., and Madry, A. (2018). How does batch normalization help optimization? In Advances Neural Information Processing Systems, pages 2483-2493.
Szegedy, C., loffe, S., Vanhoucke, V., and Alemi, A. A. (2017). Inception-v4, inception-resnel and the impact of residual connections on learning. In Thirty-firt AAAI conference on artificial intelligence.
Tan, M. and Le, Q. V. (2019). Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv preprint arXi::1905.11946
Wang, S. and Manning, C. (2013). Fast dropout training. In international conference machine learning, pages 118-126.
Wilson, A. C.. Roclofs, R.. Stern M., Srebro, N., and Recht, B. (2017). The marginal value of adaptive gradient methods in machine learning. Advances in Neural Information Processing Systems, pages 4148 4158.
Xie, C., Tan, M., Gong, B., Wang, J., Yuille, I.., and Le, Q. (2020). Adversarial examples improve image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 819-828.
Xie, S., Girshick, R., Dollár, P., Tu, 7.,, and He, K. (2017). Aggregated residual transformations for deep neural networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492-1500.
Yang, G., Pennington, J., Rao, V.. Sohl-Dickstein, and Schoenholz, S. (2019). mean field theory of batch normalization. arXiv preprint arXiv:1902.08129,
Zagorukko, S. and Komodakis, N. (2016). Wide residual networks. arXiv preprint 46
10

---

Published as a conference paper at ICLR 2021
Zhang, H., Dauphin, Y. N., and Ma, T. (2019). Fixup initialization: Residual learning without normalization. arXiv preprint XXi1:1901.09333
Zoph, B., Vasudevan, V., Shlens, J., and Le, Q. V. (2018). Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8697-8710.

---

Published as a conference paper at ICLR 2021
A
APPENDIX
Here we provide the derivation for Equation around p = 0
1 Consider the Taylor expansion this expression
1
[wwishx)]]]
+ ||||| O(Pl).
(8)
Putting this together with the mean squared norm of the feature embedding
Ell Emb (x) ||
E h/l.s///..l//aaaaccooo//rrraaara + 3)]1
(9)
have
[x]]]mb(x)] htt/://lllybaaccccmmm/rrmmmmmmm) + O(pp).
(10)
This further simplifies due to the normalizing effect of BatchNorm
(lll + =|1 = Σ E [C)B)tcchllomm(((reemm))))))) + +33)3 (11)
= =11>11 + 1131?
(12)
yielding the result
[x][Emb(x)]]]- 4 11711 11311 O(Pe)).
(13)
12


Reviews:
[{"review_id": "wTSdNASUFqm", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper empirically studies the regularization of BN. It proposes the point that the BN's effect is connected with the regularizing against explosive growth in the final layer. To motivate this point, it takes a single-layer case and shows the BN approximately penalizes on the norm of the feature embedding thereon. Two regularizations are proposed according to the point and are used to justify it. \n\nIn the experiments, it combines the proposed regularizations with Fixup and tests on CIFAR-10 and ImageNet along with three typical architectures including, WideResNet, ResNet-50, and Efficientnet. The observed phenomena support the claims and further find that prevent the explosion of the norm can recover most of the improvements of BN. Also, the proposed regularizations achieve great performance on benchmarks via properly setting the coefficient. \n\nThe studied direction is very important and would bring some new understanding to the ICLR community. Motivated by the simple case and connections with the previous techniques, like dropout, fixup, standardizing are really good points. The reviewer can see clear differences and contributions to the literature. Experiments are conducted on large-scale benchmarks and different architecture. So the results and conclusions tend to be general. Overall the paper is easy to follow.\n\n-----------------------------------------\nSome concerns and comments are listed below.\n\nThe experiment would run a grid search to select a coefficient for the L2 regularization. Table 1,2 only list the performance of the best choice. The reviewer wonders how sensitive would the coefficient affect accuracy, especially on ImageNet. (The reviewer guesses the experiments of Figure 3 is not conducted on ImageNet.) If it is significant, the method may not suitable for common use. \n\nIn Sec. 4.4, the author claims that the generalization gap is associated with the growth in the norm. But, the gap shows in the very late phase, while the norm growth starts at the very beginning. There's no strong correlation.\n\nIn all experiments, the proposed regularizations perform based on the Fixup initialization. The reviewer wonders how the performance change if you change an initialization (e.g. Xavier Init), though the concept of your regularization is the same as the Fixup. To see if the proposed regularization can help training even without proper initialization, like the last layer BN experiments did in Sec.4.4. Also, the paper currently does not prove the importance of the initialization via any ablations but directly use the Fixup.\n\nTo sum up, the reviewer thinks the paper finds some points about the regularization effects of BatchNorm, but not much principal, and would rate 6.\n\n\n-------------------after rebuttal----------\nI thank the author for your answer. Here're the response to your latest reply.\n\nFor point 1, I am aware of your performance and encourage you to add the discussion in the main paper.\n\nFor point 2, I still doubt it since I think your claim may only hold for the ImageNet experiment on the Outputs Norm term. According to figure 2, the network does not faster converge with the help of BN. The regularization gap happened in the late stage. But, the gap of two networks on the feature embedding norm and mean output norm happens at the very beginning and keeps increasing. (except  ImageNet experiment on the Outputs Norm term)\n\nOverall, I think the paper may bring some insights to understand the BN and would like to keep my original score.", "extra_scores": {}, "extra_text": {}}, {"review_id": "_NYavOHMTf", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Paper Summary\n\nThis paper aims to improve our understanding of BatchNorm's positive effects on training and generalization by identifying its constituent effects, and then replicating those effects with simpler techniques. Through a series of experiments using these simpler techniques, the authors show that perhaps the most important mechanism by which BatchNorm improves training, and especially generalization, is by preventing excessive growth of the activations of the layer before the output layer.\n\n## Strengths\n\nThe paper makes a particularly strong case for studying the relationship between the \"embedding norm\" -- the norm of inputs to the final layer of the network -- and successful training/generalization. It does this by\n\na) showing that use of BatchNorm can implicitly control this norm, especially if weight decay is applied to BatchNorm parameters.\n\nb) showing that in networks trained with BatchNorm, the value of this norm remains much lower than in those trained without BatchNorm.\n\nc) showing that using simple L2 regularization penalty on this norm, its value can be prevented from growing and, in combination with Fixup initialization, networks without BatchNorm can achieve similar generalization performance as those with BatchNorm.\n\nI think this constitutes a good amount of initial evidence that this technique may be useful for both researchers and practitioners.\n\n## Weaknesses\n\nI did not find major weaknesses, though it remains a little puzzling why all the clearly positive results with activation norm penalties matching or outperforming BatchNorm are only obtained on ImageNet. This indicates that perhaps more tasks/datasets should have been explored by the authors (the paper uses only two datasets).\n\n## Review Summary\n\nDespite the weakness above, I think the results are sufficiently interesting to be published and evaluated by the wider community. I recommend accepting this paper, though results on more diverse tasks and datasets will certainly improve this paper (and my rating).\n\n## Minor comments\n\n- Sec. 3.2: Authors say \"Zhang et al. show residual networks without normalization have explosive dynamics with conventional initializations. In particular, they show that the output scale of the networks grows exponentially with their depth $L$\". It is incorrect to attribute these to Zhang et al. instead of Balduzzi et al. Please correct this.\n- Sec. 4.3: I don't see \"modest improvements\" with standardizing loss on CIFAR-10 in Table 1. Given the standard errors, I would say that there is effectively no improvement.\n- Section 2.1 uses $\\rho$ in Swish instead of $\\beta$ which is mentioned in Sec. 4.2.\n- Sec. 3.1: \"Wang and Manning (2013) has\" should be \"Wang and Manning (2013) have\"\n- Table captions should be on top, not bottom of the tables, according to ICLR formatting guidelines.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "WPve6pev_SD", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "** Overview\nThis paper is generally well written and well organized. It is easy to read. It strives to disentangle the two possible effects of batchnorm in neural network in order to study its main contribution to network regularization. The proclaimed two possible effects are\n1. standardizing the intermediate activations\n2. regularizing against explosive growth in the final layer.\nBy dropping the batchnorm layer and replace it with one of the two penalty functions --  penalty on normalization or penalty on the l2 norm of the final layer -- the authors claimed that much of the performance gain from batchnorm is recovered from the norm regularization of the final layer. Although the paper is interesting, I do not find the results to be very convincing (please see \"questions\" section for clarification)\n\n** Pros\n1 Well written and well organized.\n2 Experiments are conducted to explain the disentangled effect of batchnorm in regularization.\n\n** Cons and Questions.\n1. Even though the batch norm layer has the effect of standardizing intermediate layers and limiting the size of the final layers, it is not necessarily true that their effects can be replicated by simply putting a penalty on the network loss during training. In particular, In table 1 and table 2, the authors use very small penalty coefficient lambda to replicate the \"standardizing effect\", and claimed that \"we found that higher coefficients led to divergence at high learning rate.\" This leads me to think that the results from directly penalizing the effects of batchnorm are unconvincing. \n\n2. One of the benefit of batchnorm is that it allows for a significantly larger learning rate. After dropping the batchnorm and putting the penalty on, does such benefit still exist? I would very much prefer to see that the maximal \"allowable\" learning rate does not decrease after dropping the batchnorm layer.\n\n3. The authors claimed a connection between dropout and penalization of the output feature in equation (5). However, this plausible connection is only valid when features are decorrelated -- which seems to be a very strong assumption.\n\n4. The authors also make a difference between \"feature embedding L2\" and \"Functional L2\", whose difference seems unnecessary when weight decay penalty is used because of equation (5). Is that correct?\n\n5. If I understand correctly, it is the explosion of gradient instead of the output feature that makes training method hard to converge. Can the authors elaborate on the merit of limiting the feature size? After all, even if the feature is large, a rescaled weight matrix in the final layer can easily bring everything back to normal size.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper analyzes BatchNorm through a series of experiments and shows that BatchNorm improves training and generalization by preventing excessive growth of the activations of the previous to the last layer. The paper received mixed reviews. Two reviewers find that the paper brings clear and important new contributions to the understanding of BatchNorm, and appreciate the experimental evaluation, though some suggestions for improving the experiments were also provided. The other reviewer appreciated the contribution of regularizing against explosive growth in the previous to the last layer, but did not find the results to be very convincing as they can limit the learning rate. The authors responded that, contrary to the reviewer's claim, the learning rate is not limited. Overall, while there are some aspects that need improvement, and the authors should address those in the final version, this is a solid paper that brings an interesting contribution to ICLR.
Decision: Accept (Poster)