Beyond BatchNorm: Towards a Unified Understanding of Normalization in Deep Learning
Year: 2021

Abstract:
Inspired by BatchNorm, there has been an explosion of normalization layers in deep learning. Recent works have identified a multitude of beneficial properties in BatchNorm to explain its success. However, given the pursuit of alternative normalization layers, these properties need to be generalized so that any given layer’s success/failure can be accurately predicted. In this work, we take a first step towards this goal by extending known properties of BatchNorm in randomly initialized deep neural networks (DNNs) to several recently proposed normalization layers. Our primary findings follow: (i) similar to BatchNorm, activations-based normalization layers can prevent exponential growth of activations in ResNets, but parametric techniques require explicit remedies; (ii) use of GroupNorm can ensure an informative forward propagation, with different samples being assigned dissimilar activations, but increasing group size results in increasingly indistinguishable activations for different samples, explaining slow convergence speed in models with LayerNorm; and (iii) small group sizes result in large gradient norm in earlier layers, hence explaining training instability issues in Instance Normalization and illustrating a speed-stability tradeoff in GroupNorm. Overall, our analysis reveals a unified set of mechanisms that underpin the success of normalization methods in deep learning, providing us with a compass to systematically explore the vast design space of DNN normalization layers.

heading: 1 Introduction
text: Normalization techniques are often necessary to effectively train deep neural networks (DNNs) [1, 2, 3]. Arguably, the most popular of these is BatchNorm [1], whose success can be attributed to several beneficial properties that allow it to stabilize a DNN’s training dynamics: for example, ability to propagate informative activation patterns in deeper layers [4, 5]; reduced dependence on initialization [6, 7, 8]; faster convergence via removal of outlier eigenvalues [9, 10]; auto-tuning of learning rates [11], equivalent to modern adaptive optimizers [12]; and smoothing of loss landscape [13, 14]. However, depending on the application scenario, BatchNorm’s use can be of limited benefit or even a hindrance: for example, BatchNorm struggles when training with small batch-sizes [3, 15]; in settings with train-test distribution shifts, BatchNorm can undermine a model’s accuracy [16, 17]; in meta-learning, it can lead to transductive inference [18]; and in adversarial training, it can hamper accuracy on both clean and adversarial examples by estimating incorrect statistics [19, 20].
To either address specific shortcomings or to replace BatchNorm in general, several recent works propose alternative normalization layers (interchangeably called normalizers in this paper). For example, Brock et al. [23] propose to match BatchNorm’s forward propagation behavior in Residual
Email: {eslubana, dickrp}@umich.edu, and hidenori.tanaka@ntt-research.com *Work partially performed during an internship at Physics & Informatics Laboratories, NTT Research.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
networks [24] by replacing it with Scaled Weight Standardization [25, 26]. Wu and He [3] design GroupNorm, a batch-independent method that groups multiple channels in a layer to perform normalization. Liu et al. [27] use an evolutionary algorithm to search for both normalizers and activation layers. Given the right training configuration, these works show their proposed normalizers often achieve similar test accuracy to BatchNorm and even outperform it on some benchmarks. This begs the question, are we ready to replace BatchNorm? To probe this question, we plot training curves for models defined using different combinations of normalizer, network architecture, batch size, and learning rate on CIFAR-100. As shown in Figure 1, clear trends begin to emerge. For example, we see LayerNorm [2] often converges at a relatively slower speed; Weight Normalization [28] cannot be trained at all for ResNets (with and without SkipInit [6]); Instance Normalization [22] results in unstable training in deeper non-residual networks, especially with small batch-sizes. Overall, evaluating hundreds of models in different settings, we see evident success/failure modes exist for all normalization techniques, including BatchNorm.
As we noted before, prior works have established several properties to help explain such success/failure modes for the specific case of BatchNorm. However, given the pursuit of alternative normalizers in recent works, these properties need to be generalized so that one can accurately determine how normalization techniques beyond BatchNorm affect DNN training. In this work, we take a first step towards this goal by extending known properties of BatchNorm at initialization to several alternative normalization techniques. As we show, these properties are highly predictive of a normalizer’s influence on DNN training and can help ascertain exactly when an alternative technique is capable of serving as a replacement for BatchNorm. Our contributions follow.
• Stable Forward Propagation: In Section 3, we show activations-based normalizers are provably able to prevent exploding variance of activations in ResNets, similar to BatchNorm [5, 6]. Parametric normalizers like Weight Normalization [28] do not share this property; however, we explain why architectural modifications proposed in recent works [6, 7] can resolve this limitation.
• Informative Forward Propagation: In Section 4, we first show the ability of a normalizer to generate dissimilar activations for different inputs is a strong predictor of optimization speed. We then extend a known result for BatchNorm to demonstrate the rank of representations in the deepest layer of a Group-normalized [3] model is at least Ω( √ width/Group Size). This helps us illustrate
how use of GroupNorm can prevent high similarity of activations for different inputs if the group size is small, i.e., the number of groups is large. This suggests Instance Normalization [22] (viz.,
GroupNorm with group size equal to 1) is most likely and LayerNorm [2] (viz., GroupNorm with group size equal to layer width) is least likely to produce informative activations.
• Stable Backward Propagation: In Section 5, we show normalization techniques that rely on individual sample and/or channel statistics (e.g., Instance Normalization [22]) suffer from an exacerbated case of gradient explosion [29], often witnessing unstable backward propagation. We show this behavior is mitigated by grouping of channels in GroupNorm, thus demonstrating a speed–stability trade-off characterized by group size.
Related Work: Due to its ubiquity, past work has generally focused on understanding BatchNorm [5, 4, 6, 9, 10, 7, 13, 29, 30, 31]. A few works have studied LayerNorm [32, 33], due to its relevance in natural language processing. In contrast, we try to analyze normalization methods in deep learning in a general manner. As we show, we can identify properties in BatchNorm that readily generalize to other normalizers and are often predictive of the normalizer’s impact on training. Our analysis is inspired by a rich body of work focused on understanding randomly initialized DNNs [34, 35, 36, 37, 38]. Most related to us is the contemporary work by Labatie et al. [39], who analyze the impact of different normalization layers on expressivity of activations and conclude LayerNorm leads to high similarity of activations in deeper layers. As we discuss, this result is in fact a special case of our Claim 3.

heading: 2 Preliminaries: Normalization Layers for DNNs
text: We first clarify the notations and operations used by the normalizers discussed in this work. Specifically, we define operators µ{d}(T ) and σ{d}(T ), which calculate the mean and standard deviation of a tensor T along the dimensions specified by set {d}. ‖T ‖ denotes the `2 norm of T . RMS{d}(T ) denotes the root mean square of T along dimensions specified by set {d}. For example, for a vector v ∈ Rn, we have RMS{1}(v) = √∑ i v 2 i/n. We assume the outputs of these operators broadcast as per requirements. ρ(.) denotes the sigmoid function. We define symbols b, c, x to denote the batch, channel, and spatial dimensions. For feature maps in a CNN, x will include both the height and the width dimensions. The notation c/g denotes division of c neurons (or channels) into groups of size g. When grouping is performed, each group is normalized independently.
Normalization Layers: We analyze ten normalization layers in this work. These layers were chosen to cover a broad range of ideas: e.g., activations-based layers [1, 40], parametric layers [23, 28], hand-engineered layers [3], AutoML designed layers [27], and layers [22, 2, 4] that form building blocks of recent techniques [41]. 1. Activations-Based Layers: BatchNorm (BN) [1], Layer-
Norm (LN) [2], Instance Normalization (IN) [22], GroupNorm (GN) [3], Filter Response Normalization (FRN) [40], Variance Normalization (VN) [4], EvoNormBO [27], and EvoNoRMSO [27] fall in this category. These layers function in the activation space. Note that Variance Normalization is an ablation of BatchNorm that does not use the mean-centering operation. Typically, given activations AL at layer L, these layers use an operation of the form Anorm = φ ( γ
σ{d}(AL) (AL − µ{d}(AL)) + β
) ).
Here, γ and β are learned affine parameters used for controlling quantities affected by the normalization operations (such as mean, standard deviation, and RMS) and φ is a non-linearity, such as ReLU. The exact operations for these layers, minus the affine parameters, are shown in Table 1. 2. Parametric Layers: Weight Normalization (WN) [28] and Scaled Weight Standardization (SWS) [23] fall in this category. Table 1 shows the exact operations. These layers function in the parameter space and act on a filter’s weights (W) to generate normalized weights (Wnorm). The normalized weightsWnorm are used for processing the input: AL+1 = φ(Wnorm ∗ AL).

heading: 3 Stable Forward Propagation
text: Stable forward propagation is a necessary condition for successful DNN training [36]. In this section, we identify and demystify the role of normalization layers in preventing the problem of exploding or vanishing activations during forward propagation. These problems can result in training instability due to exploding or vanishing gradients during backward propagation [36, 38]. Building on a previous study on BatchNorm, we first show that activations-based normalizers provably avoid exponential growth of variance in ResNets1, ensuring training stability. Thereafter, we show parametric normalizers do not share this property and ensuring stable training requires explicit remedies.

heading: 3.1 Activations-Based Normalizers and Exponential Variance in Residual Networks
text: Hanin and Rolnick [38] show that for stable forward propagation in ResNets, the average variance of activations should not grow exponentially (i.e., should not explode). Interestingly, Figure 1 shows that all activations-based normalizers are able to train the standard ResNet [24] architecture stably. For BatchNorm, this behavior is provably expected. Specifically, De and Smith [6] find that to ensure variance along the batch-dimension is 1, BatchNorm rescales the Lth layer’s residual path output by a factor of O (1/√L). This causes the growth of variance in a Batch-Normalized ResNet to be linear in depth, hence avoiding exponential growth of variance in and ensuring effective training. We now show this result can be extended to other normalization techniques too. Claim 1. Similar to BatchNorm [6], GroupNorm [3] avoids exponential growth of variance in ResNets, ensuring stable training.
Proof. We follow the same setup as De and Smith [6]. Assume the Lth residual path (fL) is processed by a normalization layer N , after which it combines with the skip connection to generate the next output: yL = yL−1 + N (fL(yL−1)). The covariance of layer input and Residual path’s output is assumed to be zero. Hence, the output’s variance is: Var(yL) = Var(yL−1) + Var(N (fL(yL−1))). Now, assume GroupNorm with group size G is used for normalizing the D-dimensional activation signal, i.e., N = GN(.). This implies for the gth group, σg,x(GN(fL(yL−1))) = 1. Then, for a batch of size N , denoting the ith sample activations as y (i) L , and using (y (i) L )
j to index the activations, we note the residual output’s variance averaged along the spatial dimension is: 〈Var(N (fL(yL−1))〉 = 1D ∑D j=1( 1 N ∑N i=1(GN(fL(y (i) L−1))
j)2) = 1 N ∑N i=1( 1 D ∑D j=1(GN(fL(yL−1) (i))j)2) = 1N ∑N i=1 G D ( ∑D/G g=1(σg,x(GN(fL(yL−1)
(i))))2) = 1. Overall, this implies 〈Var(yL)〉 = 〈Var(yL−1)〉 + 1. Recursively applying this relationship for a bounded variance input, we see average variance at the Lth layer is in O(L). Thus, similar to BatchNorm, use of GroupNorm will enable stable forward propagation in ResNets by ensuring signal variance grows linearly with depth.
We show empirical demonstrations of Claim 1 in Figure 2, where the average activation variance is plotted for a ResNet-56. As can be seen, for all activations-based normalizers, the growth of variance is linear in the number of layers. At the end of a Residual module, which spatially downsamples the signal, the variance plummets. However,
the remaining layers follow a pattern of linear growth, as expected by our result. We note our 1The case of non-residual networks is discussed in appendix. In brief, most normalizers help avoid exploding/vanishing activations by enforcing unit activation variance in the batch, channel, or spatial dimensions.
theory does not apply to EvoNorms, which are designed via AutoML. However, empirically, we see EvoNorms also avoid exponential growth of variance in ResNets. Thus, our analysis shows, all activations-based normalizers in Table 1 share the beneficial property of stabilizing forward propagation in ResNets, similar to BatchNorm.

heading: 3.2 Parametric Normalizers and Exploding Variance in Residual Networks
text: By default, parametric normalizers such as Weight Normalization [28] and Scaled Weight Standardization [23] do not preserve the variance of a signal during forward propagation, often witnessing vanishing activations. To address this limitation, properly designed output scale and bias corrections are needed. Specifically, for Weight Normalization and ReLU non-linearity, Arpit et al. [42] show the output should be modified as follows: AL+1 = √ 2π/π−1(φ(Wnorm ∗ AL)− √ 1/2π). For Scaled
Weight Standardization, only output scaling is needed [23]: AL+1 = φ( √
2π/π−1Wnorm ∗ AL). In Figure 1, ResNet training curves for Weight Normalization [28] and Scaled Weight Standardization [23] were not reported as the loss diverges to infinity. As we explain in the following, this is a result of using correction factors designed to enable variance preservation in non-residual networks. Claim 2. Unlike BatchNorm [6], Weight Normalization [28] and Scaled Weight Standardization [23] witness unstable training due to exponential growth of variance in standard ResNets [24].
Proof. Using the correction factors above, both Weight Normalization and Scaled Weight Standardization will ensure signal variance is preserved on the residual path: Var(N (f(yL−1))) = Var(yL−1). Thus, using these methods, the output variance at layer L becomes: Var(yL) = Var(yL−1) + Var(N (f(yL−1))) = 2 Var(yL−1). Recursively applying this relationship for a bounded variance input, we see signal variance at the Lth layer is in O(2L). Thus, Weight Normalization and Scaled Weight Standardization witness exponential growth in variance.
More generally, the above result shows if the residual path is variance preserving, ResNets will witness exploding variance with growing depth. Prior works [43, 5, 8, 6, 7, 44] have noted this result in the context of designing effective ResNet initializations. Here, we extended this result to show why Weight Normalized and Scaled Weight Standardized ResNets undergo unstable forward propagation. Empirical demonstration is provided in Figure 3a.
In their work introducing Scaled Weight Standardization [23], Brock et al. are able to circumvent exponential growth in variance by using SkipInit [6]. Specifically, inspired by the fact that BatchNorm biases Residual paths to identity functions, De and Smith [6] propose SkipInit, which multiplies the output of the residual path by a learned scalar α that is initialized to zero. This suppresses the Residual path’s contribution, hence avoiding exponential growth in variance (see Figure 3b). Interestingly, even
after using SkipInit, we find Weight Normalized ResNets witness variance explosion (see Figure 3b). To explain this behavior, we note that in the standard ResNet architecture, the non-linearity is located after the addition operation of skip connection and the residual path’s signals (see Figure 3a). Thus, even if SkipInit is used to suppress the residual path, the non-linearity will still be applied to the skip connections. Since the scale correction for Weight Normalization ( √ 2π/π−1) is greater than 1, this implies the signal output is amplified at every layer to preserve signal variance; however, since convolutions are absent on the skip path, signal variance never decays. Consequently, variance is only amplified, causing the variance to increase exponentially in the number of layers (see Figure 3b).
Training ResNets with Weight Normalization: The above discussion shows that for Weight Normalization, since the output has to be scaled-up to preserve signal variance, standard ResNets [24] witness exploding activations. This also hints at a solution: place the non-linearity on the Residual path. This modification (see Figure 3c) in fact results in one of the architectures proposed by He et al. in their original work on ResNets [45]. We verify the effectiveness of this modification in Figure 3c. As can be seen, the signal variance in a Weight Normalized ResNet stays essentially constant for this architecture. Furthermore, we show in Figure 4 that these models are able to match BatchNorm in performance for several training configurations. In general, our discussion here explains the exact reasons why architectures with non-linearity on residual path are better suited for parametric normalizers. Finally, we note that another ResNet architecture which boasts non-linearity on residual paths is pre-activation ResNets [45]. In their experimental setup for designing Scaled Weight Standardization [23], Brock et al. specifically focused on pre-activation ResNets [45]. This is another reason why the problem of exploding activations does not surface in their work.

heading: 4 Informative Forward Propagation
text: Proper magnitude of activations is a necessary, but not sufficient, condition for successful training. Here, we study another failure mode for forward propagation, rank collapse, where activations for different input samples become indistinguishably similar in deeper layers. This can significantly slow training as the gradient updates no longer reflect information about the input data [4]. To understand this problem’s relevance, we first show why the ability to generate dissimilar activations is useful in the context of normalization methods for deep learning. Specifically, given a randomly initialized network that uses a specific normalizer, we relate its average cosine similarity of activations at the penultimate
layer (i.e., layer before the linear classifier) with its mean training accuracy (= ∑# of epochs i=1 Train Acc.[i]
# of epochs , a measure of optimization speed [46]). Results for three different architectures (Non-Residual CNN with 10 layers and 20 layers as well as ResNet-56 without SkipInit) are shown in Figure 5. As can be seen, the correlation between mean training accuracy and the average cosine similarity of activations is high. In fact, for any given network architecture, one can predict which normalizer will enable the fastest convergence without even training the model. This shows normalizers which result in more dissimilar representations at initialization are likely to be more useful for training DNNs.
We now note another interesting pattern in Figure 5: LayerNorm results in highest similarity of activations for any given architecture. To explain this, we again revisit known properties of BatchNorm. As shown by Daneshmand et al. [4, 47], BatchNorm provably ensures activations generated by a randomly initialized network have high rank, i.e., different samples have sufficiently different activations. To derive this result, the authors consider activations for N samples at the penultimate layer, Y ∈ Rwidth×N , and define the covariance matrix Y Y T , whose rank is equal to that of the
similarity matrix Y TY . The authors then show that in a zero-mean, randomly initialized network with BatchNorm layers, the covariance matrix will have a rank at least as large as Ω( √ width). That is, there are at least Ω( √
width) distinct directions that form the basis of the similarity matrix, hence indicating the model is capable of extracting informative activations. In the following, we propose a claim that extends this result to activations-based normalizers beyond BatchNorm.
Claim 3. For a zero-mean, randomly initialized network with GroupNorm [3] layers, the penultimate layer activations have a rank of at least Ω( √ width/Group Size), where width denotes the layer-width (e.g., number of channels in a CNN).
accuracy (=
∑# of epochs
i=1 Train Acc.[i]
# of epochs ) on CIFAR-100 vs. average cosine similarity at initialization. As shown, normalizers which induce dissimilar activations converge faster. Instance Normalization was removed due to training instability (see Section 5).
The intuition behind the above claim is based on the proof by Daneshmand et al. [4]. In their work, the authors extend a prior result from random matrix theory which suggests multiplication of several zero-mean, randomly initialized gaussian matrices will result in a rank-one matrix [10]. The use of BatchNorm ensures that on multiplication with a randomly initialized weight matrix, the values of on-diagonal elements of the covariance matrix Y Y T are preserved, while the offdiagonal elements are suppressed. This leads to a lower bound of the order of Ω( √ width) on the stable rank [48] of the covariance matrix. Now, if one directly considers the similarity matrix Y TY and uses GroupNorm instead of BatchNorm, then a similar preservation and suppression of on- and offdiagonal matrix blocks should occur. Here, the block size will be equal to the Group size used for GroupNorm. This indicates the lower bound is in Ω( √ width/Group Size).
We provide demonstration of this claim in Figure 6a. We use a similar setup as Daneshmand et al. [4], randomly initializing a CNN with constant layer-width (64) and 30 layers. A GroupNorm layer is placed before every ReLU layer and the group size is sweeped from 1 to 64. As seen in Figure 6a, we find a perfect linear fit between the stable rank and the value of √ width/Group Size, validating our claim empirically as well.
To understand the significance of Claim 3, note that the result shows if the group size is large, then use of GroupNorm cannot prevent collapse of representations (i.e., cannot result in informative activations). To demonstrate this effect, we calculate the mean cosine similarity of activations between different samples of a randomly initialized network that uses GroupNorm. We sweep the group size from layer-width to 1, thus covering the spectrum from LayerNorm (Group Size
= layer-width) to Instance Normalization (Group Size = 1). We analyze both a non-residual CNN with 20 layers and a ResNet-56. Results are shown in Figure 6b and confirm our claim that by grouping the entire layer for normalization, LayerNorm results in highly similar activations. This explains the slow convergence behavior of LayerNorm in Figure 5. Meanwhile, if we reduce the group size, similarity of representations decreases as well, indicating generation of informative activations. This shows use of GroupNorm with group size greater than layer-width can help prevent a collapse of features onto a single representation. Importantly, this result helps explain why GroupNorm can serve as a successful replacement for BatchNorm in similarity based self-supervised learning frameworks [49], which often witness representation collapse issues [50]. Similar to BatchNorm, GroupNorm helps discriminate between representations of different inputs, helping avoid a collapse of representations.

heading: 5 Stable Backward Propagation
text: Taking the results of Section 4 to the extreme should imply Instance Normalization (i.e., Group Size = 1) is the best configuration for GroupNorm, but as we noted in Figure 1, Instance Normalization witnesses unstable training. To explain this, we describe a “speed-stability” trade-off in GroupNorm in the next section by extending the property of gradient explosion in BatchNorm to alternative normalization layers. Specifically, Yang et al. [29] recently show that gradient norm in earlier layers of a randomly-initialized BatchNorm network increases exponentially with increasing model depth (see Figure 8). This shows the maximum depth of a model trainable with BatchNorm is finite. The theory leading to this result is quite involved, but a much simpler analysis can not only explain this phenomenon accurately, but also illustrate the existence of gradient explosion in alternative layers.
Gradient explosion in BatchNorm: Following Luther [51], we analyze the origin of gradient explosion based on the expression of gradient backpropagated through a BatchNorm layer. We calculate the gradient of loss w.r.t. activations at layer L, denoted as YL ∈ RdL×N . We define two sets of intermediate variables: (i) pre-activations, generated by weight multiplication,XL = WLYL−1 and (ii) normalized pre-activations, generated by BatchNorm, X̂L = BN(XL) = γσ{N}(XL) (XL − µ{N}(XL)) + β. Under these notations, the gradient backpropagated from layer L to layer L− 1 is (see appendix for derivation): ∇YL−1(J) = γ σ{N}(XL)
WTL P [∇X̂L(J)]. Here P is a composition of two projection operators: P[Z] = P⊥1N [P ⊥ Ob(X̂L/ √ N)
[Z]]. The operator P⊥Ob(X̂L/√N)[Z] = Z − 1 N diag(ZX̂ T L)X̂L subtracts its input’s component that is inline with the BatchNorm outputs via projection onto the Oblique manifold diag( 1N X̂LX̂ T L) = diag(1). Similarly, P⊥1 [Z] = Z(I − 1 N 1N1 T N ) mean-centers its input along the batch dimension via projection onto 1N ∈ RN .
Notice that at initialization, the gradient is unlikely to have a large component along specific directions such as the all-ones vector (1) or the oblique manifold defined by X̂L. Thus, the gradient norm will remain essentially unchanged when propagating through the projection operation (P). However, the next operation, multiplication with γσ{N}(XL) (= 1 σ{N}(XL)
at initialization) will re-scale the gradient norm according to the standard deviation of pre-activations along the batch dimension. As shown by Luther [51], for a standard, zero-mean Gaussian initialization, the pre-activations have a standard deviation equal to √ π−1/π < 1. Thus, at initialization, the division by standard deviation operation amplifies the gradient during backward propagation. For each BatchNorm layer in the model, such an amplification of the gradient will take place, hence resulting in an exponential increase
in the gradient norm at earlier layers. Overall, our analysis exposes an interesting tradeoff in BatchNorm: Divison by standard deviation during forward propagation, which is important for generating dissimilar activations [4], results in gradient explosion during backward propagation, critically limiting the maximum trainable model depth! Empirically, the above analysis is quite accurate near initialization. For example, in Figure 7, we show that the correlation between the norm of the gradient at a layer (‖∇YL(J)‖) and the inverse product of standard deviation of the pre-activations of layers ahead of it (ΠL+1l=101/σ{N}(XL)) remains very high (0.6–0.9) over the first few hundred iterations in a 10-layer CNN trained on CIFAR-100.
Gradient Explosion in Other Normalizers: We now extend the phenomenon of gradient explosion to other normalizers. The primary idea is that since all activation-based normalizers have a gradient expression similar to BatchNorm (i.e., projection followed by division by standard deviation), they all re-scale the gradient norm during backprop. However, the statistic used for normalization varies across normalizers, resulting in different severity of gradient explosion.
Claim 4. For a given set of pre-activations, the backpropagated gradient undergoes higher average amplification through an Instance Normalization layer [22] than through a BatchNorm layer [1]. Further, GroupNorm [3] witnesses lesser gradient explosion than both these layers.
Proof. The gradient backpropagated through the gth group in a GroupNorm layer with group-size G is expressed as: ∇YgL−1(J) = γ σ{g}(X g L) WTL P [∇X̂gL(J)] (see appendix for derivation). Here, P is defined as: P[Z] = P⊥1 [P⊥S(X̂L/√G) [Z]], where P ⊥ S(X̂L/√G)[Z] = (I − 1 GX̂ g LX̂ g T L )Z. That is, the component of gradient inline with the normalized pre-activations will be removed via projection onto the spherical manifold defined by ||X̂gL|| = √ G. As can be seen, the gradient expressions for GroupNorm and BatchNorm are very similar. Hence, the discussion for gradient explosion in BatchNorm directly applies to GroupNorm as well. This implies, when Instance Normalization is used in a CNN, the gradient norm for a given channel c and the ith sample is amplified by the factor
1 σ{x}(X c L,i) (inverse of spatial standard deviation). Then, over N samples, using the arithmetic-mean ≥ harmonic-mean inequality, we see the average gradient amplification in Instance Normalization is greater than gradient amplification in BatchNorm: 1N ∑ i
1 σ2{x}(X c L,i) ≥ N∑ i σ 2 {x}(X c L,i) = 1 σ2{N}(XL) .
Similarly applying arithmetic-mean ≥ harmonic-mean for a given sample and the gth group, we see average gradient amplification in Instance Normalization is greater than gradient amplification in GroupNorm: 1G ∑ c
1 σ2{x}(X g,c L ) ≥ G∑ c σ 2 {x}(X g,c L ) = 1 σ2{g}(XL) . Extending this last inequality by
averaging over N samples, we see average gradient amplification in GroupNorm is lower than that in BatchNorm. This implies grouping of neurons in GroupNorm helps reduce gradient explosion.
We show empirical verification of Claim 4 in Figure 8. As can be seen, the gradient norm in earlier layers follows the order Instance Normaliation ≥ BatchNorm ≥ GroupNorm ≥ LayerNorm, as proved in Claim 4. Further, since increasing depth implies more normalization operations, we see gradient explosion increases as depth increases. Similarly, since reducing batch-size increases gradient noise, we find gradient explosion increases with decrease in batch-size as well.
Speed–stability trade-off in GroupNorm: Combined with Section 4, our discussion in this section helps identify a speed– stability trade-off in GroupNorm. Specifically, we find that while GroupNorm with group size equal to 1 (viz., Instance Normalization) results in more diverse features (see Claim 3), it is also more susceptible to gradient explosion and hence sees training instability for small batch-sizes/large model depth (see Figure 1). Meanwhile, when group size is equal to layerwidth (viz., LayerNorm), gradient explosion can be avoided, but the model is unable to generate informative activations and thus witnesses slower optimization. Combining these
results demonstrates that the group size in GroupNorm ensues a trade-off between high similarity of activations (influences training speed) and gradient explosion (influences training stability). To illustrate this trade-off, we can estimate training instability by fitting an exponential curve to layerwise gradient norms (measures degree of gradient explosion) and estimate training speed by calculating cosine similarity of activations at the penultimate layer at initialization (highly correlated with training speed; see Figure 6). Results are shown in Figure 9. We see increasing group size clearly trades-off the two properties related to training speed and stability, with a moderately large group size resulting in best performance. In fact, we see test accuracy is highest exactly at this point of intersection in the trade-off. This explains the success of channel grouping in GroupNorm and other successful batch-independent normalization layers like EvoNormSO [27]. Interestingly, these results also help explain why in comparison to BatchNorm, which suffers from gradient explosion and exacerbates the problem of high gradient variance in non-IID Federated learning setups [52, 53], use of GroupNorm with a properly tuned group-size helps achieve better performance [52, 54].

heading: 6 Discussion and Limitations
text: Discussion: As the number of deep learning architectures continues to explode, the use of normalization layers is becoming increasingly common. However, past works provide minimal insight into what makes normalization layers beyond BatchNorm (un)successful. Our work acts as a starting point to bridge this gap. Specifically, we extend known results on benefits/limitations of BatchNorm to recently proposed normalization layers and provide a thorough characterization of their behavior at initialization. This generalized analysis provides a compass that can help systematically infer which normalization layer is most appealing under the constraints imposed by a given application, reducing reliance on empirical benchmarking. Moreover, since our results show phenomenon used to explain BatchNorm’s success exist in alternative normalizers as well, we argue the success of BatchNorm requires further characterization. Our work also opens avenues for several new fronts of research. For example, in Section 4 we demonstrated that a normalization layer’s impact on similarity of activations accurately predicts resulting optimization speed. As shown in a contemporary work by Boopathy and Fiete [55], the weight update dynamics of a neural network are in fact guided by the matrix defining similarity of activations. Beyond providing grounding to our observation, their results indicate that relating design choices in neural network development with similarity of activations can help optimize their values. Indeed, a recent method for neural architecture search directly utilizes the similarity of activations to design “good” architectures [56].
Limitations: In this work, we limit our focus to discriminative vision applications. We highlight that nine out of ten normalizers studied in this work were specifically designed for this setting and we indeed find that all our analyzed properties show predictive control over the final performance of a model in discriminative vision tasks, generalizing across multiple network architectures. However, extending our work to develop similar analyses in other contexts such as NLP will be very useful. The primary hurdle is that for different data modalities, the standard architecture families and their corresponding optimization difficulties vary widely. For example, in both LSTM and transformer architectures, an often noted training difficulty arises from large gradient norms, which can result in divergent training or training restarts [57, 58]. In fact, optimizers in existing NLP frameworks have gradient clipping enabled by default to avoid this problem [59]. Beyond large gradients, unbalanced gradients are also known to be a training difficulty in NLP architectures [60]. We think a thorough treatment of the role of normalization layers in addressing these problems will be very valuable and leave it for future work.

heading: Acknowledgements
text: We thank Hadi Daneshmand and anonymous reviewers for several helpful discussions that helped improve this paper. This work was partly supported by NSF under award CNS-2008151.

Review summaries:
This paper summarizes multiple aspects of a variety of normalization schemes and what makes them work or not work. Specifically, the authors analyze properties that prior works have found to be critical in BatchNormalso in the context of other normalizers (like GroupNorm), thus arguing for a more unified understanding and insight towards these normalization components. The discussion and thus contribution can be broken into three parts: stable activation, discriminative representation and gradient explosion analysis.

Normalization methods are effective optimization hacks that allow training deep neural networks. In this paper, different normalization methods for training neural networks are collectively studied. The focus of the paper is mostly on designing experiments that show how normalization methods allow faster training for deep neural networks.  In particular, they are studied from two main aspects:  1. The change of gradient norm (at the random initialization) when the network grows in depth.  2. The similarity of hidden representations when weights are initialized randomly.    (1) It is experimentally and theoretically shown that the normalization method can be classified into two groups: one suffering from the exponential gradient vanishing, while the other group is more robust against in terms of the norm of the gradient (in presence/absence of residual connections).    (2) First it is shown that the similarity of hidden representation (measured by pair-wise cosine similarity of inputs) is well correlated with the optimization speed. Therefore, one can predict the performance of optimization methods by this metric. Recent results confirm that BN provides distinctive features when the weights are initialized randomly. Here it is experimentally shown that this results in more normalization techniques such as group normalization.  Finally, the issue of the decay of norm of gradient is more carefully studied where it is shown that the norm of gradient although do not exponentially decay (or explodes), yet may decay through the layers which decay rate determines the optimization speed of normalization methods.