Beyond BatchNorm: Towards a Unified Understanding of Normalization in Deep Learning
Authors: Ekdeep Singh Lubana
Robert P. Dick
Hidenori Tanaka
Venue: NeurIPS 2021 Poster (2021)
Keywords: Normalization layers
BatchNorm
Unified framework

Abstract: Inspired by BatchNorm, there has been an explosion of normalization layers in deep learning. Recent works have identified a multitude of beneficial properties in BatchNorm to explain its success. However, given the pursuit of alternative normalization layers, these properties need to be generalized so that any given layer's success/failure can be accurately predicted. In this work, we take a first step towards this goal by extending known properties of BatchNorm in randomly initialized deep neural networks (DNNs) to several recently proposed normalization layers. Our primary findings follow: (i) similar to BatchNorm, activations-based normalization layers can prevent exponential growth of activations in ResNets, but parametric techniques require explicit remedies; (ii) use of GroupNorm can ensure an informative forward propagation, with different samples being assigned dissimilar activations, but increasing group size results in increasingly indistinguishable activations for different samples, explaining slow convergence speed in models with LayerNorm; and (iii) small group sizes result in large gradient norm in earlier layers, hence explaining training instability issues in Instance Normalization and illustrating a speed-stability tradeoff in GroupNorm. Overall, our analysis reveals a unified set of mechanisms that underpin the success of normalization methods in deep learning, providing us with a compass to systematically explore the vast design space of DNN normalization layers.

Beyond BatchNorm: Towards a Unified
Understanding of Normalization in Deep Learning
Ekdeep Singh Lubana' Robert Dick', Hidenori Tanaka
EBCS Department, University of Michigan Department of Applied Physics, Stanford University Pphysics& Informatics Laboratories, NTT Research, Inc.
Abstract
Inspired by BatchNorm, there has been explosion of normalization layers in deep learning. Recent works have identified a multitude of beneficial properties in BatchNorm to explain its success. However, given the pursuit of alternative normalization layers, these properties need to be generalized so that any given layer's success/failure can be accurately predicted. In this work, we take first step towards this goal by extending known properties of BatchNorm in randomly initialized deep neural networks (DNNs) several recently proposed normalization layers. Our primary findings follow: (i) similar to BatchNorm, activations normalization layers can prevent exponential growth of activations in ResNets, but parametric techniques require explicit remedies; (ii) use of GroupNorm can ensure informative forward propagation, with different samples being assigned dissimi- lar activations, but increasing group size results in increasingly indistinguishable activations for different samples, explaining slow convergence speed in models with LayerNorm; and (iii) small group sizes result in large gradient norm in earlier layers, hence explaining training instability issues in Instance Normalization and illustrating speed-stability tradeoff in GroupNorm. Overall, our analysis reveals a unified set of mechanisms that underpin the success of normalization methods in deep learning, providing us with compass to systematically explore the vast design space DNN normalization layers.
Introduction
Normalization techniques are often necessary effectively train deep neural networks (DNNs) I 2
3]. Arguably, the most popular of these is BatchNorm [ whose success can be attributed to several beneficial properties that allow it stabilize a DNN's training dynamics: for example, ability to propagate informative activation patterns in deeper layers 4 5]: reduced dependence on initializa- tion 6 faster convergence via removal of outlier eigenvalues 9 10 ; auto-tuning of learning rates equivalent to modern adaptive optimizers 112
; and smoothing of loss landscape [13 [4] However, depending on the application scenario, BatchNorm's use can be of limited benefit or even hindrance: for example, BatchNorm struggles when training with small batch-sizes
3 15 in
settings with train-test distribution shifts, BatchNorm can undermine model's accuracy [16] in
meta-learning, it can lead to transductive inference [18]: and in adversarial training, it can hamper
accuracy on both clean and adversarial examples by estimating incorrect statistics [9 20).
To either address specific shortcomings or to replace BatchNorm in general, several recent works propose alternative normalization layers (interchangeably called normalizers in this paper). For example, Brock et al. 23 propose to match BatchNorm's 's forward propagation behavior in Residual
Email: (eslubana, dickrp) @ummich.eee edu, and hidenori. hidemori.tanaka@@nii manaka@ntt-essarchh
*ork partially performed during an internship at Physics & Informatics Laboratorics, NTT Research.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

BN
EvoBO
Evo5O
FRN
UN
GN
IN
VN
SWS
WN
256,
Bsire: 16, LR:: 11
Bsize: 256, UR: D-
Bsize: 16, LRt 1//16
00-
100
Train Acc.
80-
60-
Train Acc.
80
60-
60
40
40
40-
40
20
20
50
10
10
50
Epoch
Epoch
Epoch
Epoch
(a) Non-Residual CNN with 10 layers
(b) Non-Residual CNN with 20 layers
Brize: 256, 0.1
Bsizo: 16, LR: /11
Bsibe: 256, LR: 0.1
Bribe: 16, LR: 0
100
100
Train Acc.
80-
Acc.
60-
40-
40
20-
20-
20
10
Epoch
Epoch
Epoch
Epoch
(c) ResNet-56 (without SkipInit (6)
(d) ResNet-56 (with SkipInit (6)
Figure 1: Each normalization method has its own success and failure modes. We plot training curves (3 seeds) for different combinations of normalizer (see Table l}. network architecture, and batch-size at largest stable initial learming rate on CIFAR-I00 1-00. Learning rate is scaled linearly with batch-size [2T]. Layers for which loss reaches infinity are not plotted. Test curves and several other settings are provided in the appendix. The plots show that all methods, including BatchNorm (BN), have their respective success and failure modes: e.g., LayerNorm (LN) 2 often converges slowly and Instance Normalization (IN) 22 can have unstable training with large depth or small batch-sizes.
networks 24 by replacing it with Scaled Weight Standardization 225 26). Wu and He [3] design GroupNorm, batch-independent method that groups multiple channels in a layer to perform normalization. Liu et al. (27] use an evolutionary algorithm to search for both normalizers and activation layers. Given the right training configuration, these works show their proposed normalizers often achieve similar test accuracy to BatchNorm and even outperform it on some benchmarks. This begs the question, for models defined using different combinations of normalizer, network architecture, batch size, and learning rate on CIFAR-100. As shown in Figure clear trends begin emerge. For example, we see LayerNorm 2 often converges at a relatively slower speed; Weight Normalization (28 cannot be trained at all for ResNets (with and without SkipInit (6);; Instance Normalization 22 results in unstable training in deeper non-residual networks, especially with small batch-sizes. Overall, evaluating hundreds of models in different settings, we see evident success/failure modes exist for all normalization techniques, including BatchNorm.
we ready replace BatchNorm? To probe this question, we plot training curves
As noted before, prior works have established several properties to help explain such suc- cess/failure modes for the specific case of BatchNorm. However, given the pursuit of alternative normalizers in recent works, these properties need to be generalized so that one can accurately determine how normalization techniques beyond BatchNorm affect DNN training. In this work, we take first step towards this goal by extending known properties of BatchNorm at initialization to several alternative normalization techniques. As we show, these properties are highly predictive of a normalizer's 's influence on DNN training and can help ascertain exactly when an alternative technique is capable of serving as a replacement for BatchNorm. Our contributions follow.
Stable Forward Propagation: In Section 3 we show activations hased normalizers are provably able prevent exploding variance of activations in ResNets, similar BatchNorm 55 6) Paramet- ric normalizers like Weight Normalization (28) do not share this property; however, we explain why architectural modifications proposed in recent works 6 can resolve this limitation.
Informative Forward Propagation: In Section 4] we first show the ability of a normalizer to generate dissimilar activations for different inputs is a strong predictor of optimization speed. We then extend a known result for BatchNorm to demonstrate the rank of representations in the deepest layer of Groupznormalized 3 model is at least 2(1 wiithh/Group Size). This helps us illustrate how use of GroupNorm can prevent high similarity of activations for different inputs if the group size is small, i.c., the number of groups is large. This suggests Instance Normalization [2] (viz.,
2

---

GroupNorm with group size equal to 1) is most likely and LayerNorm [2] (viz., GroupNorm with group size equal layer width) is least likely to produce informative activations.
Stable Backward Propagation: In |Section 5, we show normalization techniques that rely on
individual sample and/or channel statistics (c.g., Instance Normalization [22]) suffer from an exacerbated case of gradient explosion [29], often witnessing unstable backward propagation. We show this behavior is mitigated by grouping of channels in GroupNorm, thus demonstrating trade-off characterized by group size.
Related Work: Due to its ubiquity, past work has generally focused on understanding BatchNorm 5 4 6 9
T0
7 13 29 30 31. few works have studied LayerNorm [32] [3], due to its relevance in
natural language processing. In contrast, we try analyze normalization methods in deep learning in general manner. As we show, we can identify properties in BatchNorm that readily generalize to other normalizers and are often predictive of the normalizer's 's impact training. Our analysis is inspired
by a rich body of work focused on understanding randomly initialized DNNs 34 35 36 37 38). Most related to is the contemporary work by Labatie al. [39], who analyze the impact of different normalization layers on expressivity of activations and conclude LayerNorm leads to high similarity of activations in deeper layers. As we discuss, this result is in fact special case of our Claim
2 Preliminaries: Normalization Layers for DNNs
We first clarify the notations and operations used by the normalizers discussed this work. Specifically, we de- fine operators μay (T) and (d) (T), which calculate the mean and standard deviation of a tensor T along the di- mensions specified by set {d). T] denotes the &g norm of T. RMS (7) denotes the root mean square of T along dimensions specified by set {d]. For example, for VE. ve/n. We
Activations Based Layers
Pgay Maaa (A): (aa) (4)
BN
A-1(5a)
IN
(c.x)
IN
"(x)
GN
A-M(o/na)
vector v E IR,, we have (1) assume the outputs of these operators broadcast as per requirements. p(.) denotes the sigmoid function. We de- fine symbols b. C, to denote the batch, channel, and spatial dimensions. For feature maps in a CNN, will include both the height and the width dimensions. The notation c/g denotes division of e neurons (or channels) into groups of size g. When grouping is performed, each group is normalized independently.
((ofgi))
FRN
RMB
VN 4
EvoBO 27
max (b.x) (x))
EvoSO 2
Aptuaa: (c/sa))
Parametric Layers
M(d) Mqay (W): (M)) (()) (d)
WN 28
TWTT
Normalization Layers: We analyze ten normalization layers in this work. These layers were chosen to cover broad range ideas: e.g., activations-basse layers 40).
SWS 23
P(c.h.w)
"{o.a)
Table 1: Operations performed by dif- ferent normalizers. AA denotes layer in- put: W denotes incoming neuron weights to a neuron.
parametric layers 23 28 hand land-engineered layers
3
AutoMI. designed layers (27, and layers 22 that
form building blocks of recent techniques 41
Inctaation--Bssele Layers: BatchNorm (BN) Layer-
Norm (LN) (2). Instance Normalization (IN) 22 GroupNorm (GN) 3 Filter Response Normaliza-
tion (FRN) 400, Variance Normalization (VN) ExoNormBO 27 and EvoNoRMSO (27) fall in this category. These layers function in the activation space. Note that Variance Normalization is ab- lation of BatchNorm that does not use the mean-centering operation. Typically, given activations As,
layer T., these layers an operation of the form Anom = (ra])AA](AA (1AL-1 -maa (At)) + B))-
Here, y and B are learned affine parameters used for controlling quantities affected by the normaliza- tion operations (such as mean, standard deviation, and RMS) and a non-linearity, such as ReI.U. The exact operations for these layers, minus the affine parameters, are shown in Table 1
Parametric Layers: Weight Normalization (WN) 28 and Scaled Weight Standardization (SWS) 23 fall in this category. Table I shows the exact operations. These layers function in the parameter space and act on a filter's weights (W) to generate normalized weights (VVnomn) The normalized weights W used for processing the input: AL+1 6(W Ar).
3

---

3 avoids exponential growth of variance in 3 Stable Forward Propagation
Stable forward propagation is necessary condition for successful DNN training [3). In this section, we identify and demystify the role of normalization layers in preventing the problem of exploding or vanishing activations during forward propagation. These problems can result training
instability due to exploding or vanishing gradients during backward propagation [36] 38). Building on a previous study on BatchNorm, we first show that ietrrationsbbbeed normalizers provably avoid exponential growth of variance in ResNet[] ensuring training stability. Thereafter, we show parametric normalizers do not share this property and ensuring stable training requires explicit remedies.
3.1 Activations-Baser Normalizers and Exponential Variance in Residual Networks
Hanin and Rolnick (38 show that for stable forward propagation in ResNets, the average variance of activations should not grow exponentially (i.e., should not explode). Interestingly, FFiuure I shoows that all activations- normalizers are able to train the standard ResNet (24] architecture stably. For BatchNorm, this behavior is provably expected. Specifically, De and Smith [6] find that to ensure variance along the batch-dimension is 1, BatchNorm rescales the 7,th layer's residual path output by a factor of O (1/V.). This causes the growth of variance in a Batch-Normalized ResNet to be linear in depth, hence avoiding exponential growth of variance in and ensuring effective training. We now show this result can be extended to other normalization techniques too.
Claim 1. Similar to BatchNorm 10). GroupNorm ResNets, ensuring stable training.
Proof. We follow the same setup De and Smith 6). Assume the 7 residual path (fr) is processed by a normalization layer N, after which it combines with the skip connection to generate the next output: yz YL-1 (V(f((L-1)) The covariance of layer input and Residual path's output is assumed to be zero. Hence, the output's variance is: Var(y,) = Var(yr..1) + Var(N (y r))). Now, assume GroupNorm with group size G is used for nor- malizing the D-dimensional activation signal, i.e. .N' = GN(.). This implies for the gtd group, 1. Then, for batch of size N, denoting the 7th sample activations as YL and using (y to index the activations, we note the residual output's variance averaged 1. EN (GN( (4) ())))) = = 1. Overall, this implies (Var(yr.)) = (Var(y,.-)) 1. Recursively applying this relationship for bounded variance input, we see average variance the T layer is in O(L). Thus, similar to Normh use of GroupNorm will enable stable forward propagation in ResNets by ensuring signal variance grows linearly with depth.
ge (GN(
along the spatial dimension is: (Var(W(fe(yL
I IE + E (GN(SL(Y)-)))))) = + Σズ 101100 (o
(GN(J(VV)))))))) (
N
L
To understand the relevance of the above result, note that for G = 1, GroupNorm is equal to Instance Normaliza- tion 22 and for G = D, GroupNorm is equal to Lay- erNorm 2 Further, since the mean of the signal is as- sumed to be zero, the average variance along the spatial dimension is equal to the RMS operation used by Filter Response Normalization [40]. Thus, by proving the above result for GroupNorm, we are able to show alternative activations-bant normalizers listed in Table Iaass avoid the exponential growth of activation variance in ResNets.
BN
EvoSO
LN
IN VN
EvoBO
- FRN
GN
Activ. Var.
6
14
28
42
56
Layer
Figure 2: Activations-based normaliz- ensure linear and stable forward propagation, verifying [Clamm 1 Ac- tivation Variance (Activ. Var.) as a function of layer number in a ResNet- 56 24 processing CIFAR- samples.
We show empirical demonstrations of Claim in Fig ure 2 where the average activation variance is plotted for a ResNet-56. As can be seen, for all activations based nor- malizers, the growth of variance is linear in the number of layers. At the end of a Residual module, which spatially downsamples the signal, the variance plummets. However, the remaining layers follow a pattern of linear growth, as expected by our result. We note our
'The case of non-residual networks is discussed in appendix. In brief, most normalizers help avoid explod- ing/vanishing activations by enforcing unit activation variance in the batch, channel, or spatial dimensions.

---

Roadyd
M0
NY
Cans
NE!
Activ. Var.
10
SWS
Activ. Var.
10
Activ. Var.
10
10"
105
10
10
28
56
Layer
Layer
Layer
(a) Standard ResNet
(b) SkipInit
(c) Non-Lincarity on Residual Path
Figure 3: Parametric normalizers witness exponentially growing variance, verifying Claim 2 but we can stabilize by modifying the -sihual-path. We plot log activation variance as function of layer number in randomly initialized ResNet-56 (24), using CIFAR- 100 samples, with Scaled Weight Standardization (SWS) (23) and Weight Normalization (WN) (28) for different architectures (simplified illustrations provided top). (a) Standard ResNet: Both SWS and WN witness variance explosion in a standard ResNet model, as claimed in Claim 2 (b) Skiplnit: SkipInit [6] multiplies the residual signal with scalar initialized zero, thus preventing variance explosion in SWS model at initialization. Meanwhile, by scaling the non- non-linearity after addition, a WN model continues witness exploding variance. (c) Non- Linearity Residual Puth: Shifting the non-linearity the residual path prevents variance explosion in both WN and SWS models.
theory does not apply EvoNorms, which designed via AutoML. However, empirically, see EvoNorms also avoid exponential growth of variance in ResNets. Thus, our analysis shows, all activations-bused normalizers in Table share the beneficial property of stabilizing forward propagation in ResNets, similar to BatchNorm.
3.2 Parametric Normalizers and Exploding Variance in Residual Networks
By default, parametric normalizers such as Weight Normalization 28 and Scaled Weight Standard- ization (23) do not preserve the variance of signal during forward propagation, often witnessing vanishing activations. To address this limitation, properly designed output scale and bias corrections are needed. Specifically, for Weight Normalization and ReLU non-linearity, Arpit et al. 42] show
the output should be modified as follows: AL-1 = 22=/m-12 /27/n-1(0( 10lh- worm * AL) / 1/2m). ). For Scaled
Weight Standardization, only output scaling is needed 23 AL+1 25/2 -1W Worm AA)))
In Figure ResNet training curves for Weight Normalization 28 and Scaled Weight Standardiza- tion 23 were not reported as the loss diverges infinity. As we explain in the following, this is result of using correction factors designed to enable variance preservation non-residual networks.
Claim 2. Unlike BatchNorm (6], Weight Normalization [28] and Scaled Weight Standardization [23
witness unstable training due to exponential growth of variance in standard ResNets (24%.
Proof. Using the correction factors above, both Weight Normalization and Scaled Weight Standardiza- tion will ensure signal variance is preserved on the residual path: Var(N ((()))))))) = Var(yz.-1).
Thus, using these methods, the output variance at layer T. becomes: Var(yz) Var(yı-1) Var(A(((yyzz)))
2 -ar(y Recursively applying this relationship for bounded variance
input, we see signal variance at the 1h layer is in O(2²). Thus, Weight Normalization and Scaled Weight Standardization witness exponential growth variance.
More generally, the above result shows if the residual path is variance preserving, ResNets will witness exploding variance with growing depth. Prior works [43 5 44 have noted this result in the context of designing effective ResNet initializations. Here, we extended this result to show why Weight Normalized and Scaled Weight Standardized ResNets undergo unstable forward propagation. Empirical demonstration is provided in Figure 3al
In their work introducing Scaled Weight Standardization [23], Brock et al. are able to circumvent exponential growth variance by using SkipInit (6) Specifically, inspired by the fact that BatchNorm biases Residual paths to identity functions, De and Smith (6) propose Skiplnit, which multiplies the output the residual path by learned scalar a that is initialized zero. This suppresses the Residual path's 's contribution, hence avoiding exponential growth in variance (see Figure 3b). Interestingly, even
5

---

Batchhorm
Weight Normalization
Scaled Weight Standardization
Train
Test
100
Bsize: 256, LR: 0.1
100
Brize: 256, LR: 0.02
Bsize: 16, LR: 0.1/16
60
Bsize: 16, LR: 0.01 0.01/26
Accuracy
80-
60-
60
40-
40
20-
20-
20-
20-
Epoch
Epoch
Epoch
Epoch
Figure 4: Modified residual-path allows for successful training with parametric layers. We plot train/test accuracy (over 3 seeds) for ResNet-56 architecture on CIFAR-100 100 with non-linearity located on the residual path. We see parametric normalizers can train effectively if scaled non-linearities are not located after the addition operation in ResNet.
after using SkipInit, find Weight Normalized ResNets witness variance explosion (see Figure 3b). To explain this behavior, we note that in the standard ResNet architecture, the non-linearity is located after the addition operation of skip connection and the residual path's 's signals (see Figuure 3a). Thus, even if Skiplnit is used to suppress the residual path, the non-linearity will still be applied to the skip connections. Since the scale correction for Weight Normalization (v (v<=/=-1) is greater than 1, this implies the signal output is amplified at every layer to preserve signal variance; however, since convolutions are absent on the skip path, signal variance never decays. Consequently, variance is only amplified, causing the variance to increase exponentially in the number of layers (see (Figur 3b).
Training ResNets with Weight Normalization: The above discussion shows that for Weight Nor- malization, since the output has to be scaled-up preserve signal variance, standard ResNets [24] witness exploding activations. This also hints at a solution: place the non-linearity on the Residual path. This modification (see Figure 3cD in fact results in one of the architectures proposed by He et al. in their original work on ResNets [45]. We verify the effectiveness of this modification in Figure 3c As can be seen, the signal variance in a Weight Normalized ResNet stays essentially constant for this architecture. Furthermore, we show in Figure 4| that these models are able to match BatchNorm in performance for several training configurations. In general, our discussion here explains the exact reasons why architectures with non-linearity on residual path are better suited for parametric normalizers. Finally, we note that another ResNet architecture which boasts non-linearity on residual paths is pre-activation ResNets [45.. In their experimental setup for designing Scaled Weight Standardization (23), Brock al. specifically focused on pre-activatee activation ResNets [45]. This another reason why the problem of exploding activations does not surface in their work.
Informative Forward Propagation
Proper magnitude of activations is necessary, but sufficient, condition for successful training. Here, we study another failure mode for forward propagation, rank collapse, where activations for different input samples become indistinguishably similar in deeper layers. This can significantly slow training as the gradient updates no longer reflect information about the input data [4]. To understand this problem's 's relevance, we first show why the ability to generate dissimilar activations useful in the context of normalization methods for deep learning. Specifically, given a randomly initialized network that uses a specific normalizer, we relate its average cosine similarity of activations at the penultimate
layer (i.e., layer before the linear classifier) with its mean training accuracy (= Eta Train Acc.fil a * of epochs measure of optimization speed [46).. ). Results for three different architectures (Non-Residus CNN with 10 layers and 20 layers as well as ResNet-56 without Skiplnit) are shown in Figure 5) As can be seen, the correlation between mean training accuracy and the average cosine similarity of activations is high. In fact, for any given network architecture, fastest convergence without even training the model. This shows normalizers which result in more dissimilar representations at initialization are likely to be more useful for training DNNs.
can predict which normalizer will enable the
LayerNorm results in highest similarity of activations for any given architecture. To explain this, we again revisit known properties of Batch- Norm. As shown by Daneshmand et al. 4 47 BatchNorm provably ensures activations generated by randomly initialized network have high rank, i.e., different samples have sufficiently different activations. To derive this result, the authors consider activations for N samples at the penultimate layer, Y E W and define the covariance matrix YYT, whose rank is equal to that of the
We now note another interesting pattern in Figure 5)

---

Piain
Group Szze Group Size m
Group Size - InstanceNorm
GroupNorm
LayerNorm
7
(20 Layers)
ResNet 56
Cos. Simi arity
Stable Rank
0.9
0.7
7
1
14
20
1
14
28 Layer
42
56
V (width/ Size)
Layer
(a) Stable rank vs. group size.
(b) Layer-wise Cosine Similarity.
Figure 6: The smaller the group size, the higher the rank of the activations, verifying Claim (a) We plot stable rank of activations at the penultimate layer for random Gaussian inputs. As proposed in Claim 3| find a perfect linear fit between stable rank and values of / /Widih/Crouup for different group sizes. (b) Implications Claim on CIFAR- 100 sampels: by increasing group size (constant across layers), see similarity of features at any given layer increases. This shows LayerNorm 2 cannot generate informative features, thus witnessing slow convergence (see Figure 5).
similarity matrix Y"Y. The authors then show that in a zero-mean, randomly initialized network with BatchNorm layers, the covariance matrix will have rank at least as large 2 ((y/w. That is, there are at least Q2 width) distinct directions that form the basis of the similarity matrix, hence indicating the model is capable of extracting informative activations. In the following, we propose a claim that extends this result to activations- based normalizers beyond BatchNorm.
Claim 3. For a zero-mean, randomly initialized network with GroupNorm (3) layers, the penultimate layer activations have a runk of at least O /widthh/ Group Size), where width denotes the layer-widih (e.g, number of channels in CNN).
The intuition behind the above claim based on the proof
BN
LN
WN
by Daneshmand et al. [4. In their work, the authors extend a prior result from random matrix theory which suggests multi-
EvoBO EvoSO
Layer
VN
20 Layer CNN
FRN
SWS
ResNet-56 56
plication of several zero-mean, randomly initialized gaussian matrices will result BatchNorm ensures that on multiplication with randomly initialized weight matrix, the values of on-diagonal elements of the covariance matrix YYT are preserved, while the off- diagonal elements are suppressed. This leads to lower bound
rank-one matrix [T]. The use of
Mean Train. Acc.
85
0.67
75
0.88
of the order of S2( / wwidth) on the stable rank (48) of the co-
variance matrix. Now, if one directly considers the similarity matrix Y and uses GroupNorm instead of BatchNorm, then similar preservation and suppression of on- and off- diagonal matrix blocks should occur. Here, the block size will be equal to the Group size used for GroupNorm. This indicates the lower bound is in 12( width/Groop Size).
55
0.4
0.6
0.8
1.0
Cosine Similarity
Figure 5: Informative forward propagation results in faster opti- mization. We plot mean training of Train Acc.lil, on accuracy (= #f epochs
We provide demonstration of this claim in Figure 6a We use similar setup as Daneshmand et al. [4] randomly initializ- ing a CNN with constant layer-width (64) and 30 layers. GroupNorm layer is placed before every ReLU layer and the group size is sweeped from 1 to 64. As seen in Figure 6a| we find a perfect linear fit between the stable rank and the value of Grooth/Grrupppp Size, validating our claim empirically well.
CIFAR- 100 vs. average cosine simi- larity at initialization. As shown, nor- malizers which induce dissimilar ac- tivations converge faster. Instance Normalization was removed due to training instability (see Section 5).
To understand the significance of Claim 3) note that the result shows if the group size is large, then use of GroupNorm cannot prevent collapse of representations (i.e., cannot result in informative activations). To demonstrate this effect, we calculate the mean cosine similarity of activations between different samples of randomly initialized network that uses GroupNorm. We sweep the group size from layer-width to 1, thus covering the spectrum from LayerNorm (Group Size
7

---

layer-width) to Instance Normalization (Group Size = 1). We analyze both non-residual CNN with 20 layers and ResNet-56. Results are shown in Figure and confirm our claim that by grouping the entire layer for normalization, LayerNorm results in highly similar activations. This explains the slow convergence behavior of LayerNorm in [Fiuure 5] Meanwhile, if we reduce the group size, similarity of representations decreases as well, indicating generation of informative activations. This shows use of GroupNorm with group size greater than layer-width can help prevent a collapse of features onto a single representation. Importantly, this result helps explain why GroupNorm can serve as successful replacement for BatchNorm in similarity based self-supervised learning frameworks [49], which often witness representation collapse issues [50). Similar to BatchNorm, GroupNorm helps discriminate between representations of different inputs, helping avoid a collapse of representations.
5 Stable Backward Propagation
Taking the results of Section 4o the extreme should imply Instance Normalization (i.e., Group Size 1) is the best configuration for GroupNorm, but as we noted in Figure I Instance Normalization witnesses unstable training. To explain this, we describe a "speed-stability" trade-off in GroupNorm in the next section by extending the property of gradient explosion in BatchNorm alternative normalization layers. Specifically, Yang et al. (29] recently show that gradient norm in earlier layers of a -inittaliinn BatchNorm network increases exponentially with increasing model depth (see Figure 8). This shows the maximum depth of a model trainable with BatchNorm is finite. The theory leading to this result is quite involved, but much simpler analysis can not only explain this phenomenon accurately, but also illustrate the existence of gradient explosion in alternative layers.
Gradient explosion in BatchNorm: Following Luther [51], we analyze the origin of gradient explosion based on the expression of gradient backpropagated through a BatchNorm layer. We calculate the gradient of loss w.r.t. activations at layer I., denoted Yz, E R XN We define two sets of intermediate variables: (i) pre-activations, generated by weight multiplication, X = WLY-1 and (ii) normalized pre-activations, generated by BatchNorm, X = (N)(x)) = [x,) (X p4n; (Xx)) + 3. Under these notations, the gradient backpropagated from layer to layer L - is (see appendix for derivation): Vv = (X)) WE (V (J)]. Here P is composition
of two projection operators: p[Z]- = -ii, (P Ob(8L/UM) [Z]] The operator Po POM(L//VN) [Z] = Z
N diag (ZX &r subtracts its input's component that is inline with the BatchNorm outputs via projection onto the Oblique manifold diag( + & diag(1). Similarly, PH Z(I
N 1x mean-centers its input along the batch dimension via projection onto 1N E R.N.
Notice that at initialization, the gradient is unlikely to have a large component along specific directions such as the all-ones vector (1) or the oblique manifold defined by Xz. Thus, the gra- dient norm will remain essentially unchanged when propagating through the projection operation (P). However, the next opera-
0.9
Av. Corr.
0.6
0.3
0.0
tion, multiplication with [x2] (= [x] at initialization)
100
200
300
400
will rc-seale the gradient norm according to the standard devia- tion of pre-activations along the batch dimension. As shown by Luther [5T]. for a standard, zero-mean Gaussian initialization, the ireaactvvtioon have standard deviation equal to /*-1/= < 1. Thus, at initialization, the division by standard deviation operation amplifies the gradient during backward propagation. For each BatchNorm layer in the model, such an amplification of the gra- dient will take place, hence resulting in an exponential increase in the gradient norm at earlier layers. Overall, our analysis exposes an interesting tradeoff in Batch- Norm: Divison by standard deviation during forward propagation, which is important for generating dissimilar activations [4]. results in gradient explosion during backward propagation, critically limiting the maximum trainable model depth! Empirically, the above analysis is quite accurate near initialization. For example, in in(iiuure 7] we show that the correlation between the norm of the gradient at layer (I) (VVY,(()) (J) and the inverse product of standard deviation of the pre-activations of layers ahead of it (T (TI4+1 1/ (X2)) remains very high (0.6-0.9) over the first few hundred iterations in a 10-layer CNN trained on CIFAR-100.
Iteration
Figure 7: Gradient norm vs. pre-activation statisties. We see high correlation between gra- dient norm and inverse product of layer-wise pre-activation std. deviations.
8

---

BotchNorm
Instance Normalization
Group Size
Group Size
Group Size
Layenoorm
Gradlent Norm
0.24
Layers: 10; Bsize: 256
Layers: Bsize: 256
Layers: 10;
12-
Layers: 20; Bsize: 16
9.16-
1.8
0.6
0.08
0.3
0.0
0.0
15 20
10
10 Layer
15
20
Layer
Layer
Layer
Figure 8: Small group size increases gradient explosion, verifying Claim 4 We use CIFAR- 100 samples and plot layer-wise gradient norm for different models and batch sizes. As shown, Instance Normalization [22] undergoes highest gradient explosion, followed by BatchNorm [. GroupNorm (3, and LayerNorm in all settings.
Gradient Explosion in Other Normalizers: We now extend the phenomenon of gradient explosion to other normalizers. The primary idea is that since all activation-based normalizers have a gradient expression similar to BatchNorm (i.e., projection followed by division by standard deviation), they all re-scale the gradient norm during backprop. However, the statistic used for normalization varies across normalizers, resulting different severity of gradient explosion.
Claim 4. For given set of pre-activations, the backpropagated gradient undergoes higher average amplification through an Instance Normalization layer (22] than through a BatchNorm layer A7
Further, GroupNorm [3] witnesses lesser gradient explosion than both these layers.
Proof. The gradient backpropagated through the gth group in a GroupNorm layer with group-size G' is expressed as: V Vvi._, (J) (X3] WT.
P
IV
& (J) (see appendix for derivation). Here, P
defined as: P(Z] = PT(PP 8(xe/ve) [Z]]. where PL PS 8(&L/V [Z] = (I - & yZ. That is, the component of gradient inline with the normalized pre-activations will be removed via projection onto the spherical manifold defined by ||
VG. As can be seen, the gradient expressions for GroupNorm and BatchNorm are very similar. Hence, the discussion for gradient explosion in BatchNorm directly applies to GroupNorm as well. This implies, when Instance Normalization is used in a CNN, the gradient norm for a given channel e and the 7th sample amplified by the factor ((x) (Xx))) (inverse of spatial standard deviation). Then, over N samples, using the arithmetic-men mean 2 harmonic- mean inequality, we see the average gradient amplification in Instance Normalization is greater than gradient amplification in BatchNorm: N E: (X (X),))
N
Similarly applying arithmetic- mean 2 harmonic-mest mean for a given sample and the 9th group, we
average gradient amplification in Instance Normalization is greater than gradient amplification
in GroupNorm: & Lc (x,)(x]))
of ()) Extending this last inequality by
averaging over N samples, we see average gradient amplification in GroupNorm is lower than that in BatchNorm. This implies grouping of neurons in GroupNorm helps reduce gradient explosion.
We show empirical verification of Claim 4 in Figure 8 As can be seen, the gradient norm in earlier layers follows the order Instance Normaliation > BatchNorm 2 GroupNorm >
Test Acc.
LayerNorm, as proved in Claim 4 Further, since increasing depth implies more normalization operations, we see gradi- ent explosion increases as depth increases. Similarly, since reducing batch-size increases gradient noise, we find gradient explosion increases with decrease in batch-size as well.
Grad. Exponent
1.6
0.98
Cos. Smi Smilarity
-2.-
0.61
Speed-stability trade-off in GroupNorm: Combined with Section 4] our discussion in this section helps identify a speed- stability trade-off in GroupNorm. Specifically, we find that while GroupNorm with group size equal to 1 (viz., Instance Normalization) results in more diverse features (see Claim 3). it is also more susceptible to gradient explosion and hence sees training instability for small batch-sizes/large model depth (see Figure 1). Meanwhile, when group size is equal to layer- width (viz., LayerNorm), gradient explosion can be avoided, but the model is unable to generate informative activations and thus witnesses slower optimization. Combining these
16
Group Size
Figure 9: Speed-Stability trade- off in GroupNorm using 20-layer CNNs with batch-size 256 on CIFAR-100. We see increasing group size decreases gradient explo- sion (improved training stability) at the expense of high activation simi- larity (reduced optimization speed).

---

results demonstrates that the group size in GroupNorm ensues trade-off between high similarity of activations (influences training speed) and gradient explosion (influences training stability). To illustrate this trade-off, we can estimate training instability by fitting an exponential curve to layerwise gradient norms (measures degree of gradient explosion) and estimate training speed by calculating cosine similarity of activations at the penultimate layer at initialization (highly correlated with training speed; see Figurre 6). Results are shown in Figure 9] We see increasing group size clearly trades-off' the two properties related to training speed and stability, with moderately large group size resulting in best performance. In fact, we see test accuracy is highest exactly at this point of intersection in the trade-off. This explains the success of channel grouping in GroupNorm and other successful aatch-independent normalization layers like EvoNormSO [27]. Interestingly, these results also help explain why in comparison to BatchNorm, which suffers from gradient explosion and exacerbates the problem of high gradient variance in non-III Federated learning setups [52 [33, use of GroupNorm
with properly tuned group-size helps achieve better performance 52 54
Discussion and Limitations
Discussion; As the number of deep learning architectures continues to explode, the use of normaliza- tion layers is becoming increasingly common. However, past works provide minimal insight into what makes normalization layers beyond BatchNorm (un)successful. Our work acts as a starting point to bridge this gap. Specifically, we extend known results on benefits/lmiitaions of BatchNorm recently proposed normalization layers and provide thorough characterization of their behavior at initialization. This generalized analysis provides a compass that can help systematically infer which normalization layer most appealing under the constraints imposed by a given application. reducing reliance on empirical benchmarking. Moreover, since our results show phenomenon used to explain BatchNorm's success exist in alternative normalizers as well, we argue the success of BatchNorm requires further characterization. Our work also opens avenues for several new fronts of research. For example, in [Section we demonstrated that normalization layer's impact on similarity of activations accurately predicts resulting optimization speed. As shown in a contemporary work by Boopathy and Fiete (55), the weight update dynamics of a neural network are in fact guided by the matrix defining similarity of activations. Beyond providing grounding our observation, their results indicate that relating design choices in neural network development with similarity of activations can help optimize their values. Indeed, recent method for neural architecture search directly utilizes the similarity of activations to design "good" architectures [56].
Limitations: In this work, we limit our focus to discriminative vision applications. We highlight that nine out of ten normalizers studied in this work were specifically designed for this setting and we indeed find that all our analyzed properties show predictive control over the final performance of model in discriminative vision tasks, generalizing across multiple network architectures. However, extending our work to develop similar analyses in other contexts such as NLP will be very useful. The primary hurdle that for different data modalities, the standard architecture families and their corresponding optimization difficulties vary widely. For example, in both LSTM and transformer architectures, an often noted training difficulty arises from large gradient norms, which can result in divergent training training restarts (57] (58). In fact, optimizers in existing NLP frameworks have gradient clipping enabled by default to avoid this problem [59]. Beyond large gradients, unbalanced gradients are also known to be training difficulty in NLP architectures [60]. We think thorough treatment of the role of normalization layers in addressing these problems will be very valuable and leave it for future work.
Acknowledgements
We thank Hadi Daneshmand and anonymous reviewers for several helpful discussions that helped improve this paper. This work was partly supported by NSF under award CNS-2008151.
References
[1] Sergey loffe and Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In Proc. Int. Conf. on Machine Learning (ICMI., Jul 2015.
10

---

[2] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer Normalization. arXiv, abs/1607 100002227.802,, 2016.
[3] Yuxin Wu and Kaiming He. Group Normalization. In Proc. European Conf. on Computer Vision (ECCV), 2018.
[4] Hadi Daneshmand, Jonas Kohler, Francis Bach, Thomas Hofmann, and Aurelien Lucchi. Batch normalization provably avoids ranks collapse for randomly initialised deep networks. In Proc. Adv. in Neural Information Processing Systems (NeurIPS). 2020.
[5] David Balduzzi, Marcus Frean, Lennox Leary, J. P. Lewis, Kurt Wan-Duo Ma, and Brian McWilliams, The Shattered Gradients Problem: If resnets are the answer, then what is the question? In Proc. Int. Conf. on Machine Leurning (ICML), Aug 2017.
[6] Soham De and Samuel L. Smith. Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks. In Proc. Adv. in Neural Information Processing Systems (NeurIPS). 2020.
[7] Jie Shao, Kai Hu, Changhu Wang, Xiangyang Xue, and Bhiksha Raj. Is normalization indis- pensable for training deep neural network? In Proc. Ady. in Neural Information Processing Systems (NeurIPS), 2020.
[8] Hongyi Zhang, Yann N. Dauphin, and Tengyu Ma. Residual Learning Without Normalization via Better Initialization. In Proc. Int. Conf. on Learning Representations (ICLR), 2019.
[9] Ryo Karakida, Shotaro Akaho, and Shun-ichi Amari. The normalization method for alleviating pathological sharpness in wide neural networks. In Proc. Adv. in Neural Information Processing Systems (NeurIPS), 2019.
[10] Nils Bjorck, Carla Gomes, Bart Selman, and Kilian Q Weinberger. Understanding Batch Normalization. In Proc. Adv. in Neural Information Processing Systems (NeurIPS), 2018.
[11] Sanjeev Arora, Zhiyuan Li, and Kaifeng Lyu. Theoretical Analysis of Auto Rate-Tuning by Batch Normalization. In Proc. Int. Conf. on Learning Representations (ICLR), 2019.
[12] Hidenori Tanaka and Daniel Kunin. Noether's Learning Dynamics: The Role of Kinetic Symmetry Breaking Deep Learning. arXiv preprint arXiv:2105.027166 2021.
[13] Shibani Santurkar, Dimitris Tsipras, Andrew llyas, and Aleksander Mądry. How Does Batch Normalization Help Optimization? In Proc. Adv. in Neural Information Processing Systems (NeurIPS), 2018
[14] Hongwei Yong, Jianqiang Huang, Xiansheng Hua, and Lei Zhang. Gradient Centralization: A New Optimization Technique for Deep Neural Networks. In Proc. European Conf. on Computer Vision (ECCV), 2020.
[15] Sergey loffe. Batch Renormalization: Towards Reducing Minibatch Dependence in Batch- Normalized Models. In Proc. Adv. in Neural Information Processing Systems (NeurIPS), 2017.
[16) Cecilia Summers and Michael J. Dinneen. Four Things Everyone Should Know to Improve Batch Normalization. Proc. Int. Conf. on Learning Representations (ICLR), 2020.
[17] Ximei Wang, Ying Jin, Mingsheng Long, Jianmin Wang, and Michael I. Jordan. Transferable Normalization: Towards Improving Transferability of Deep Neural Networks. In Proc. Adv. in Neural Information Processing Systems (NeurIPS), 2020.
[18] John Bronskill, Jonathan Gordon, James Requeima, Sebastian Nowozin, and Richard E. Turner TaskNorm: Rethinking Batch Normalization for Meta-Learning, In Proc. Int Conf. on Machine Learning (ICML), Jul 2020.
[19] Angus Galloway, Anna Golubeva, Thomas Tanay, Medhat Moussa, and Graham W. Taylor. Batch Normalization is Cause of Adversarial Vulnerability. arXiv, abs/1905.02166. 2019.

---

[20] Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan L. Yuille, and Quoc V. Le. Adversarial Examples Improve Image Recognition. In Proc. Int. Conf. on Computer Vision and Puttern Recognition (CVPR), 2020.
[21] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, Large Minibatch SGD: Training ImageNet in I Hour. In Proc. Int. Conf. on Computer Vision and Pattern Recognition (CVPR). 2018.
[22] Dmitry Ulyanov, Andrea Vedaldi, and Victor S. Lempitsky. Instance Normalization: The Missing Ingredient for Fast Stylization. arXiv, tbs/1607.08022, 2016.
[23] Andrew Brock, Soham De, and Samuel L Smith. Characterizing signal propagation to close the performance gap unnormalized ResNets. In Proc. Int. Conf. on Learning Representations (ICLR), 2021.
[24] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In Proc. Int. Conf. on Computer Vision and Pattern Recognition (CVPR), 2016.
[25] Siyuan Qiao, Huiyu Wang, Chenxi Liu, Wei Shen, and Alan Yuille. Micro-Batch Training with Batch-Channel Normalization and Weight Standardization. arXiv, abs/1903. 10520, 2019.
[26] Lei Huang, Xianglong Liu, Yang Liu, Bo Lang, and Dacheng Tao. Centered Weight Normaliza- tion in Accelerating Training of Deep Neural Networks. In Proc. Int. Conference on Computer Vision (ICCV), 2017.
[27] Hanxiao Liu, Andrew Brock, Karen Simonyan, and Quoe V. Le. Evolving Normalization- Activation Layers. In Proc. Adv. in Neural Information Processing Systems (NeurIPS), 2020.
[28] Tim Salimans and Diederik P. Kingma. Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks. In Proc. Int. Conf. on Learning Representations (ICLR), 2016
[29] Greg Yang, Jeffrey Pennington, Vinay Rao, Jascha Sohl-Dickstein, and Samuel S. Schoenholz. A Mean Field Theory of Batch Normalization. In Int. Conf. Learning Representations (ICLR), 2019.
[30] Yuxin Wu and Justin Johnson. Rethinking "Batch" in BatchNorm. arXiv preprint arXiv:2105.07576, 2021.
[31] Vinay Rao and Jascha Sohl-Dickstein Is Batch Norm unique? An empirical investigation and prescription to emulate the best properties of common normalizers without batch dependence. arXiv,, abs/2010. 106877 2020.
[32] Jingjing Xu, Xu Sun, Zhiyuan Zhang, Guangxiang Zhao, and Junyang Lin. Understanding and Improving Layer Normalization. In Proc. Adv. in Neural Information Processing Systems (NeurIPS), 2019.
[33] Sheng Shen, Zhewei Yao, Amir Gholami, Michael Mahoney, and Kurt Keutzer, Powernorm: Rethinking batch normalization in transformers. In ICML, 2020.
[34] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proc. Int. Conf. on Artificial Intelligence and Statistics (AISTATS), 2010.
[35] Jeffrey Pennington, Samuel S. Schoenholz, and Surya Ganguli. Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice. In Proc. Adv. in Neural Information Processing Systems (NeurIPS), 2017.
[36] Samuel S. Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein, Deep Infor- mation Propagation. In Proc. Int. Conf. Learning Representations (ICLR), 2017.
[37] Antoine Labatie. Characterizing Well-Behaved vs. Pathological Deep Neural Networks. In Proc. Int. Conf. on Machine Learning (ICML), Jul 2019.
12

---

[38] Boris Hanin and David Rolnick. How to Start Training: The Effect of Initialization and Architecture. In Proc. Adv. in Neural Information Processing Systems (NeurIPS), 2018.
[39] Antoine Labatie, Dominic Masters, Zach Eaton-Rosen, and Carlo Luschi. Activations to Match Batch Normalization while Removing Batch Dependence. arXiv preprint MxKiv:2006.037433 2021.
[40] Saurabh Singh and Shankar Krishnan. Filter Response Normalization Layer: Eliminating Batch Dependence in the Training of Deep Neural Networks, In Proc. Int. Conf. Computer Vision and Pattern Recognition (CVPR), 2019.
[41] Ping Luo, Peng Zhanglin, Shao Wenqi. Zhang Ruimao, Ren Jiamin, and Wu Lingyun. Dif- ferentiable Dynamic Normalization for Learning Deep Representation, In Proc. Int. Conf. on Machine Learning (ICML), June 2019.
[42] Devansh Arpit, Yingbo Zhou, Bhargava Kota, and Venu Govindaraju. Normalization Propa- gation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networks. In Proc. Int. Conf. on Machine Learning (ICML), Jul 2016.
[43] Masato Taki. Deep Residual Networks and Weight Initialization. arXiv, abs// toss///9999966, 2017.
[44] Haozhi Qi, Chong You, Xiaolong Wang, Yi Ma, and Jitendra Malik. Deep Isometric Learning for Visual Recognition. In Proc. Int. Conf. on Machine Learning (ICML), Jul 2020.
[45] Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity Mappings in Deep Residual Networks. In Proc. European Conf. on Computer Vision (ECCV), 2016.
[46] Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning? In Proc. Adv. in Neural Information Processing Systems (NeurIPS), 2020.
[47] Hadi Daneshmand, Amir Joudaki, and Francis Bach. Batch Normalization Orthogonalizes Representations in Deep Random Networks. arXiv preprini
2021.
[48] Joel A. Tropp. An Introduction Matrix Concentration Inequalities. arXiv, Mtss/1501.01711, 2015.
[49] Pierre Richemond, Brock, Samuel Smith, Soham De, Razvan Pascanu, Bilal Piot, and Michal Valko. BYOL works even without batch statistics. arXiv preprint arXiv:2010. 2020.
Grill, Florent Altché, Corentin Tallec, Florian Strub, Andrew
[50] Xinlei Chen and Kaiming He. Exploring Simple Siamese Representation Learning. arXiv preprint arXiv:201 1. 10566,, 2020.
[51] Kyle Luther. Why Batch Norm Causes Exploding Gradients. 1o/2020/02/188 http///aaaaccoomeeeplllooooo
gradients .ataantttsmmmm..m..mmmmmmmmmmmtm 2020. Blogpost.
[52] Zhengming Zhang, Yaoqing Yang. Zhewei Yao, Yujun Yan, Joseph Gonzalez, and Michael Mahoney. Improving Semi-supervised Federated Learning by Reducing the Gradient Diversity of Models. arXiv preprint arXiv:2008. / 1364, 2020.
[53] Jianyu Wang, Zachary Charles, Zheng Xu, Gauri Joshi, H. Brendan McMahan, Blaise Aguera y Arcas, Maruan Al-Shedivat, Galen Andrew, Salman Avestimehr, Katharine Daly, Deepesh Data, Suhas Diggavi, Hubert Eichner, Advait Gadhikar, Zachary Garrett Antonious M. Girgis, Filip Hanzely, Andrew Hard, Chaoyang He, Samuel Horvath, Zhouyuan Huo, Alex Ingerman, Martin Jaggi, Tara Javidi, Peter Kairouz, Satyen Kale, Sai Praneeth Karimireddy, Jakub Konecny, Sanmi Koyejo. Tian Li, Luyang Liu, Mehryar Mohri, Hang Qi, Sashank J. Reddi, Peter Richtarik, Karan Singhal, Virginia Smith, Mahdi Soltanolkotabi, Weikang Song, Ananda Theertha Suresh, Sebastian U. Stich, Ameet Talwalkar, Hongyi Wang. Blake Woodworth, Shanshan Wu, Felix X. Yu, Honglin Yuan, Manzil Zaheer, Mi Zhang, Tong Zhang, Chunxiang Zheng, Chen Zhu, and Wennan Zhu. A Field Guide to Federated Optimization. arXiv preprint arXiv:2107.06917, 2021.
[54] Kevin Hsieh, Amar Phanishayee, Onur Mutlu, and Phillip Gibbons. The Non-IIID Data Quagmire of Decentralized Machine Learning. In Proc. Int. Conf. Machine Learning (ICML), Jul 2020.
13

---

[55] Akhilan Boopathy and Ila Fiete. Gradient-trained Weights in Wide Neural Networks Align Layerwise to Error-scaled Input Correlations. arXiv preprint arXiv:2106.08453, 2021.
[56] Joseph Mellor, Jack Turner, Amos Storkey, and Elliot Crowley. Neural Architecture Search without Training. In Proc. Int. Conf. on Machine Learning (ICML), Jul 2021.
[57] Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie. Why gradient clipping accel- erates training: theoretical justification for adaptivity. In Proc. Int. Conf. on Learning Representations (ICLR), 2020.
[58] Xinlei Chen, Saining Xie, and Kaiming He. An Empirical Study of Training Suff--uuervieed Vision Transformers, In Proc. Int. Conf. on Computer Vision (ICCV), 2021.
[59] HuggingFace. ing by default?
Why is
grad
norm
clipping done during train-
https://discuss.
htsp::///tieeusssshgggggg.g.... huggingface. co/taa.eeco///
why-is-grad-a wrad-normmclippinge done http://oooooooooooaaaaoaaoaaaaar brammigg-by-.effuu////B Discussion forum.
2020.
[60] Liyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han. Understanding the Difficulty of Training Transformers. In Proc. Conf. Empirical Methods in Natural Language Processing (EMNLP), April 2020.
14


Reviews:
[{"review_id": "hJomMejQDXh", "reviewer": "Reviewer_gpva", "summary": "This paper summarizes multiple aspects of a variety of normalization schemes and what makes them work or not work. Specifically, the authors analyze properties that prior works have found to be critical in BatchNormalso in the context of other normalizers (like GroupNorm), thus arguing for a more unified understanding and insight towards these normalization components. The discussion and thus contribution can be broken into three parts: stable activation, discriminative representation and gradient explosion analysis.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I find the paper a very interesting read. The paper is not completely novel as all of the properties studied here have already been discussed in prior works (e.g., rank collapse, gradient explosion, stable activation), but the authors manage to extend these discussions to other sorts of activation-based normalizers and even parametric normalizers. \n\nPros:\n- Clear explanation and intuition; the paper is well-written overall.\n- The proof is correct (upon a cursory look) and relatively straightforward.\n- Good set of empirical evidence supporting most of the claims and conjectures of the authors.\n- Thorough examination of the multiple aspects of the normalization components and the conclusion is a good step towards further unifying these normalizers.\n\nCons:\n- Relatively limited context in terms of the discussion (e.g., a major portion of this paper still focuses on activation-based normalizers, in particular BN, GN (and its variant IN); learnable parameter not discussed). \n- While the unified perspective is appealing and with good empirical support, I'm dubious of whether the property claims can generalize to other contexts and domains (i.e., are they necessary? are they sufficient?). See my comments below.\n\n--------------------------------\nI have more questions/comments/feedbacks for the authors:\n\n1. The first major concern I have over the analysis presented by the authors is the absence of analysis of the role learnable parameters play in this context. For example, with different layers having different $(\\gamma, \\beta)$, wouldn't the activation variance growth of a deep ResNet (e.g., Fig. 2) be more unstable? Wouldn't the gradient explosion derivation also have to include a $\\gamma$ factor in Eq. (1)? Were the experiments run with or without the affine parameters?\n\n2. The second major question is whether these properties are necessary/sufficient for building \"good normalizers\". For example, the analysis in the paper overall does not explain why LayerNorm is so indispensable in the NLP context (e.g., in Transformers). According to the analysis of the authors, LayerNorm yields similar representations and seems generally not preferred over GroupNorm (in fact, it is generally not a good idea to use BN in NLP tasks, I think). Can the authors comment more on this?\n\n3. What is the reason for vanishing variance of SWS in Figure 5? With the correction scaling, wouldn't one expect its activation variance to be relatively stable?\n\n4. This is more of a question of curiosity that I'd like to gather the authors' thoughts on: in the case where the parameters are shared across the residual blocks (i.e., $f_L = f_{L-1} = \\dots = f_1$), how will the behavior of the normalization components change? I'm mainly thinking of the formulation of Neural ODE and other implicit models, which could also take the form $\\mathbf{y}_L = \\mathbf{y}_{L-1} + \\mathcal{N}(f(\\mathbf{y}_{L-1}))$ (written in a differential, forward Euler form), but have infinite layers. Obviously, their activations did not explode, and neither did their gradient.\n\n5. Where is the correlation analysis of $\\|\\nabla_{Y_L}(J)\\|$ vs. $\\prod_{l=10}^{L+1} \\sigma_N$ mentioned in L340-341? I didn't find it in the appendix.\n\n-----------------------------\n\n**Minor points**:\n\n1. L302-303: Isn't the takeaway that smaller group size yield more discriminative features?\n2. L319: from layer $L$ to layer $L-1$.\n3. L341: $\\sigma_N(X_l)$.\n4. L594: $\\mathbf{W}_L \\in \\mathbb{R}^{D_L \\times D_{L-1}}$.\n5. L606: You missed a $\\frac{1}{N}$ in $\\text{diag}$ in the expression of $\\sigma(\\mathbf{X}_L)$.\n\n----------------\n----------------\n#### Post-rebuttal\n\nI've read the authors' response and the other reviews. I generally agree with the other reviewers that there are still quite a few observations revealed in the paper that are not yet fully explained (even empirically), but I also acknowledge that this would be interesting for future work. I'm overall satisfied with the authors' response and am inclined to keep my score.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I would indeed prefer that the authors expand upon the limitation of their analysis further (e.g., the LayerNorm and NLP point I mentioned above). There is no immediate societal impact that I think this paper should address.", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "WpSuKUJm7B", "reviewer": "Reviewer_dHUE", "summary": "This work focuses on the comparison of different normalization techniques for deep learning, using both theoretical and empirical means.  Therefore, the contributions are in the form of novel insights, rather than a novel methodology as such.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The focus of this paper is perfectly sound and relevant, and well motivated.  The approach to analysis is convincing, the discussion and arguments are insightful, and the empirical evidence is appropriate.  The quality of writing is also good.  My only concerns as regards its acceptance has to do with novelty which I am not sure is significant enough for NIPS.  This is a subjective judgement of course and a relative call (relative to the quality of the submission I expect to be considered, or from the point of view of Chairs who have better visibility, that are considered).  I also think that the authors' observation regarding rank collapse and the activations for different input samples becoming sufficiently similar to each other that they eventually become indistinguishable in deeper layers, which is correct, should be contextualized better.  In particular, the authors should highlight the alternatives, namely GhostNorm and  SeqNorm ((https://arxiv.org/abs/2007.08554) that combines GhostNorm and GroupNorm, seems to be a promising options for avoiding rank collapse while simultaneously conferring the optimization benefits that come with BatchNorm.\n\nPost rebuttal:\n\nI find the authors' responses to all feedback convincing.  As regards those aimed at my initial review specifically, they do present a good case  for a greater significance of their work than I originally recognized, so I am happy to upgrade my recommendation somewhat.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I do not see this field as being applicable in this instance.", "review_text": "", "extra_scores": {"ethical_concerns": "n/a", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "LyLGKV5TIha", "reviewer": "Reviewer_vUUS", "summary": "Normalization methods are effective optimization hacks that allow training deep neural networks. In this paper, different normalization methods for training neural networks are collectively studied. The focus of the paper is mostly on designing experiments that show how normalization methods allow faster training for deep neural networks.  In particular, they are studied from two main aspects: \n1. The change of gradient norm (at the random initialization) when the network grows in depth. \n2. The similarity of hidden representations when weights are initialized randomly. \n\n\n(1) It is experimentally and theoretically shown that the normalization method can be classified into two groups: one suffering from the exponential gradient vanishing, while the other group is more robust against in terms of the norm of the gradient (in presence/absence of residual connections). \n\n\n(2) First it is shown that the similarity of hidden representation (measured by pair-wise cosine similarity of inputs) is well correlated with the optimization speed. Therefore, one can predict the performance of optimization methods by this metric. Recent results confirm that BN provides distinctive features when the weights are initialized randomly. Here it is experimentally shown that this results in more normalization techniques such as group normalization. \nFinally, the issue of the decay of norm of gradient is more carefully studied where it is shown that the norm of gradient although do not exponentially decay (or explodes), yet may decay through the layers which decay rate determines the optimization speed of normalization methods. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and the results are insightful for the optimization of deep neural networks. \nI have a few comments regarding the results: \n1. It seems that two main factors determine the optimization performance of normalization methods: the similarity and the norm of gradient. I wondered whether the authors consider combining these two factors and reach unified predictive statistics for the optimization performance (i.e. reproducing figure 6 by also considering the norm of gradients?)\n2. Is figure 6 specific to normalization methods. One can replicate the result of Fig. 6 for a deep vanilla network without any normalization layers such that we can control the cosine similarity and the norm of gradient, then check whether we can predict the performance of gradient descent from the similarity and the norm of gradient. \n3. A recent paper theoretically studies the cosine similarity measure that is used in plot 6 (see \"Batch Normalization Orthogonalizes Representations in Deep Random Networks\"). \n4. I could not fully understand the reason for the gradient norm change in section 5. Since the induction over equation 1 is complicated, I can not see why this formula implies the decay/increase in the gradient norm. \n5. Does these different normalizations exhibit different sensitivity to the learning rate or the size of mini-batch for stochastic optimization (not the size of the normalization batch which is studied in the paper)? \n\n**Post Rebuttal**\n\nI read the authors' responses. They were very detailed. Based on my understanding of the result and also other reviews and responses, I keep my score 7. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper performs extensive empirical scans, comparing various aspects of normalization methods. The reviews are all positive, but the discussion raised a focused on novelty and clarity issues. Regarding novelty, some results seem to be less surprising and seem like a validation of existing understanding (sections 3 and 5), while some results seem to be more interesting (section 4). Regarding clarity, the paper had a lot of confusing typos (especially in the more interesting section 4), but those were cleared in the discussion. Still, I feel some parts are missing details, and I recommend to author to improve it (e.g., I couldn't follow the "proof idea" of conjecture 1, or the parts with the harmonic mean in section 5).
Decision: Accept (Poster)