Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependence
Year: 2022

Abstract:
We investigate the reasons for the performance degradation incurred with batchindependent normalization. We find that the prototypical techniques of layer normalization and instance normalization both induce the appearance of failure modes in the neural network’s pre-activations: (i) layer normalization induces a collapse towards channel-wise constant functions; (ii) instance normalization induces a lack of variability in instance statistics, symptomatic of an alteration of the expressivity. To alleviate failure mode (i) without aggravating failure mode (ii), we introduce the technique “Proxy Normalization” that normalizes post-activations using a proxy distribution. When combined with layer normalization or group normalization, this batch-independent normalization emulates batch normalization’s behavior and consistently matches or exceeds its performance.

heading: 1 Introduction
text: Normalization plays a critical role in deep learning as it allows successful scaling to large and deep models. In vision tasks, the most well-established normalization technique is Batch Normalization (BN) [1]. At every layer in the network, BN normalizes the intermediate activations to have zero mean and unit variance in each channel. While indisputably successful when training with large batch size, BN incurs a performance degradation in the regime of small batch size [2, 3, 4, 5, 6, 7]. This performance degradation is commonly attributed to an excessive or simply unwanted regularization stemming from the noise in the approximation of full-batch statistics by mini-batch statistics.
Many techniques have been proposed to avoid this issue, while at the same time retaining BN’s benefits. Some techniques mimic BN’s principle while decoupling the computational batch from the normalization batch [2, 8, 7]. Other techniques are “batch-independent” in that they operate independently of the batch in various modalities: through an explicit normalization either in activation space [9, 10, 11, 12, 13, 3, 5, 14, 15, 6] or in weight space [16, 17, 18, 19, 20]; through the use of an analytic proxy to track activation statistics [21, 22, 23]; through a change of activation function [24, 5, 15].
In this paper, we push the endeavor to replace BN with a batch-independent normalization a step further. Our main contributions are as follows: (i) we introduce a novel framework to finely characterize the various neural network properties affected by the choice of normalization; (ii) using this framework, we show that while BN’s beneficial properties are not retained when solely using the prototypical batch-independent norms, they are retained when combining some of these norms with “Proxy Normalization”, a novel technique that we hereby introduce; (iii) we demonstrate on an extensive set of experiments that, by reproducing BN’s beneficial properties, our batch-independent normalization approach consistently matches or exceeds BN’s performance.
As a starting point of our analysis, we need to gain a better understanding of those beneficial properties of BN that we aim at reproducing.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).

heading: 2 Batch Normalization’s beneficial properties
text: We consider throughout this paper a convolutional neural network with d = 2 spatial axes. This neural network receives an input x ∈ RH×W×C0 which, unless otherwise stated, is assumed sampled from a finite dataset D. The neural network maps this input x to intermediate activations xl ∈ RH×W×Cl of height H , width W and number of channels Cl at each layer l. The value of xl at spatial position α ∈ {1, . . . ,H} × {1, . . . ,W} and channel c ∈ {1, . . . , Cl} is denoted as xlα,c, with the dependency on x kept implicit to avoid overloading notations.
The inclusion of BN at layer l leads in the full-batch setting to adding the following operations ∀α, c:
ylα,c = xlα,c − µc(xl)
σc(xl) , ỹlα,c = γ l cy l α,c + β l c, (1)
where µc(xl), σc(xl) are the mean and standard deviation of xl in channel c, and γlc, β l c are channelwise scale and shift parameters restoring the degrees of freedom lost in the standardization. In the mini-batch setting, the full-batch statistics µc(xl), σc(xl) are approximated by mini-batch statistics.
Table 1 summarizes the beneficial properties that result from including BN in the neural network. Below, we provide details on each of these properties, and we discuss whether each property is reproduced with batch-independent norms.
Scale invariance. When BN is present, the input-output mapping of the neural network is invariant to the scale of weights preceding any BN layer. With such scale invariance plus weight decay, the scale of weights during training reaches an equilibrium with an “effective” learning rate depending on both the learning rate and the weight decay strength [25, 26, 27, 8, 28, 29, 30]. Such mechanism of “auto rate-tuning” has been shown to provide optimization benefits [31, 26, 32].
This property is easy to reproduce. It is already obtained with most existing batch-independent norms.
Control of activation scale in residual networks. To be trainable, residual networks require the scale of activations to remain well-behaved at initialization [33, 34, 35, 20, 36, 37]. While this property naturally arises when BN is present on the residual path, when BN is not present it can also be enforced by a proper scaling decaying with the depth of the residual path. This “dilution” of the residual path with respect to the skip connection path reduces the effective depth of the network and enables to avoid coarse-grained failures modes [38, 39, 40, 41, 42, 35].
This property is easy to reproduce. It is already obtained with most existing batch-independent norms.
Regularizing noise. Due to the stochasticity of the approximation of µc(xl), σc(xl) by mini-batch statistics, training a neural network with BN in the mini-batch setting can be seen as equivalent to performing Bayesian inference [43, 44] or to adding a regularizing term to the training of the same network with full-batch statistics [45]. As a result, BN induces a specific form of regularization.
This regularization is not reproduced with batch-independent norms, but we leave it out of the scope of this paper. To help minimize the bias in our analysis and “subtract away” this effect, we will perform all our experiments without and with extra degrees of regularization. This procedure can be seen as a coarse disentanglement of normalization’s effects from regularization’s effects.
Avoidance of collapse. Unnormalized networks with non-saturating nonlinearities are subject to a phenomenon of “collapse” whereby the distribution with respect to x, α of the intermediate activation vectors (xlα,1, . . . ,x l α,Cl
)T becomes close to zero- or one-dimensional in deep layers [39, 46, 41, 47, 48, 49, 37]. This means that deep in an unnormalized network: (i) layers tend to have their channels imbalanced; (ii) nonlinearities tend to become channel-wise linear with respect to x, α and not add any effective capacity [39, 50, 51]. Consequently, unnormalized networks can neither effectively use their whole width (imbalanced channels) nor effectively use their whole depth (channel-wise linearity).
Conversely, when BN is used, the standardization at each layer prevents this collapse from happening. Even in deep layers, channels remain balanced and nonlinearities remain channel-wise nonlinear with respect to x, α. Consequently, networks with BN can effectively use their whole width and depth.
The collapse is, on the other hand, not always avoided with batch-independent norms [39, 18, 49]. Most notably, it is not avoided with Layer Normalization (LN) [9] or Group Normalization (GN) [3], as we show both theoretically and experimentally on commonly found networks in Section 4.
To the extent possible, we aim at designing a batch-independent norm that avoids this collapse.
Preservation of expressivity. We can always express the identity with Eq. (1) by choosing βlc = µc(x l) and γlc = σc(x l). Conversely, for any choice of βlc, γ l c, we can always “re-absorb” Eq. (1) into a preceding convolution with bias. This means that BN in the full-batch setting does not alter the expressivity compared to an unnormalized network, i.e. it amounts to a plain reparameterization of the hypothesis space.
The expressivity is, on the other hand, not always preserved with batch-independent norms. In activation space, the dependence of batch-independent statistics on the input x turns the standardization into a channel-wise nonlinear operation that cannot be “re-absorbed” into a preceding convolution with bias [18]. This phenomenon is most pronounced when statistics get computed over few components. This means e.g. that Instance Normalization (IN) [10] induces a greater change of expressivity than GN, which itself induces a greater change of expressivity than LN.
In weight space, the expressivity can also be altered, namely by the removal of degrees of freedom. This is the case with Weight Standardization (WS) [18, 20] and Centered Weight Normalization [17] that remove degrees of freedom (one per unit) that cannot be restored in a succeeding affine transformation. This reduction of expressivity could explain the ineffectiveness of these techniques in EfficientNets [52], as previous works observed [20] and as we confirm in Section 6.
To the extent possible, we aim at designing a batch-independent norm that preserves expressivity.

heading: 3 Theoretical framework of analysis
text: We specified the different properties that we wish to retain in our design of batch-independent normalization: (i) scale invariance, (ii) control of activation scale; (iii) avoidance of collapse; (iv) preservation of expressivity. We now introduce a framework to quantify the presence or absence of the specific properties (iii) and (iv) with various choices of normalization.
Propagation. For simplicity, we assume in our theoretical setup that any layer l up to depth L consists of the following three steps: (i) convolution step with weights ωl ∈ RKl×Kl×Cl−1×Cl ; (ii) normalization step; (iii) activation step sub-decomposed into an affine transformation with scale and shift parameters γl,βl ∈ RCl and an activation function φ which, unless otherwise stated, is assumed positive homogeneous and nonzero (e.g. φ = ReLU). If we denote xl,yl, zl ∈ RH×W×Cl the intermediate activations situated just after (i), (ii), (iii) with the convention z0 ≡ x, we may write the propagation through layer l as
xl = Conv(zl−1), ∀α, c : Conv(zl−1)α,c = (ωl ∗ zl−1)α,c, (2)
yl = Norm(xl), ∀α, c : Norm(xl)α,c = xlα,c − µIx,c(xl)
σIx,c(x l)
, 1 (3)
zl = Act(yl), ∀α, c : Act(yl)α,c = φ ( ỹlα,c ) = φ ( γlcy l α,c + β l c ) , (4)
where µIx,c(x l), σIx,c(x l) denote the mean and standard deviation of xl conditionally on Ix,c ≡ {c}, {x}, {x, c}, {x, c mod G} for the respective cases Norm = BN, LN, IN, GN with G groups.
1We omit the numerical stability constant and adopt the convention Norm(xl)α,c = 0, ∀α if σIx,c(x l) = 0.
Moments. Extending the previous notations, we use µ, σ, P indexed with a (possibly empty) subset of variables to denote the operators of conditional mean, standard deviation and power. If we apply these operators to the intermediate activations yl, that implicitly depend on the input x and that explicitly depend on the spatial position α and the channel c, we get e.g.
µc(y l) = Ex,α [ ylα,c ] , σc(y l) = √ Varx,α [ ylα,c ] , Pc(yl) = Ex,α [ (ylα,c) 2 ]
︸ ︷︷ ︸ µc(yl)2+σc(yl)2
,
µx,c(y l) = Eα [ ylα,c ] , σx,c(y l) = √ Varα [ ylα,c ] , Px,c(yl) = Eα [ (ylα,c) 2 ]
︸ ︷︷ ︸ µx,c(yl)2+σx,c(yl)2
,
where, by convention, x, α, c are considered uniformly sampled among inputs of D, spatial positions and channels, whenever they are considered as random.
Power decomposition. Using these notations, we may gain important insights by decomposing the power in channel c of yl, just after the normalization step, as
Pc(yl) = Ex [ µx,c(y l) ]2
︸ ︷︷ ︸ P(1)c (yl)
+ Varx [ µx,c(y l) ]
︸ ︷︷ ︸ P(2)c (yl)
+Ex [ σx,c(y l) ]2
︸ ︷︷ ︸ P(3)c (yl)
+ Varx [ σx,c(y l) ] .
︸ ︷︷ ︸ P(4)c (yl)
(5)
Since this four-terms power decomposition will be at the core of our analysis, we detail two useful views of it. The first view is that of a hierarchy of scales: P(1)c (yl) measures the power of µc(yl) at the dataset scale; P(2)c (yl) measures the power of µx,c(yl)− µc(yl) at the instance scale; the sum of P(3)c (yl) and P(4)c (yl) measures the power of ylα,c−µx,c(yl) at the pixel scale. A particular situation where the power would be concentrated at the dataset scale withP(1)c (yl) equal toPc(yl) would imply that yl has its distribution fully “collapsed” in channel c, i.e. that yl is constant in channel c.
The second view is that of a two-level binary tree: on one half of the tree, the sum of P(1)c (yl) and P(2)c (yl) measures the power coming from µx,c(yl), with the relative proportions of P(1)c (yl) and P(2)c (yl) functions of the inter-x similarity and inter-x variability of µx,c(yl); on the other half of the tree, the sum of P(3)c (yl) and P(4)c (yl) measures the power coming from σx,c(yl), with the relative proportions of P(3)c (yl) and P(4)c (yl) functions of the inter-x similarity and inter-x variability of σx,c(y
l). A particular situation where P(2)c (yl), P(4)c (yl) would be equal to zero would imply that µx,c(y l), σx,c(yl) have zero inter-x variability, i.e. that µx,c(yl), σx,c(yl) are constant for all x.
A version of Eq. (5) at the layer level instead of channel level will be easier to work with. Defining P(i)(yl) as the averages of P(i)c (yl) over c for i ∈ {1, 2, 3, 4}, we obtain
P(yl) = P(1)(yl) + P(2)(yl) + P(3)(yl) + P(4)(yl). It should be noted that P(yl) = 1 for any choice of Norm ∈ {BN,LN, IN,GN} as long as the denominator of Eq. (3) is nonzero for all x, c [C.1]. Consequently, the terms P(i)(yl) sum to one, meaning they can be conveniently seen as the proportion of each term i ∈ {1, 2, 3, 4} into P(yl).
Revisiting BN’s avoidance of collapse. When BN is used, yl is normalized not only layer-wise but also channel-wise with P(1)c (yl) = 0 and Pc(yl) = 1. As a first consequence, ỹl (that is only one affine transformation away from yl) is unlikely to have its channel-wise distributions collapsed. This means that the nonlinearity φ acting on ỹl is likely to be effectively nonlinear with respect to ỹl’s channel-wise distributions.2 As a result, each layer adds capacity and the network effectively uses its whole depth. This is opposite to the situation where ỹl has its channel-wise distributions collapsed with Pc(ỹl)− P(1)c (ỹl) Pc(ỹl) for all c, which results in φ being close to linear with respect to ỹl’s channel-wise distributions. This is illustrated in Figure 1 and formalized in Appendix C.2.
2Note that: (i) the effective nonlinearity of φ with respect to ỹl’s channel-wise distributions could be quantified in the context of random nets of Definition 1 with “reasonable” choices of β, γ; (ii) BN only guarantees an intra-distribution nonlinearity and not an intra-mode nonlinearity in contexts such as adversarial training [54, 55] or conditional GANs [56], unless modes are decoupled in BN’s computation [57, 58, 59, 60, 61].
Pc(ỹl) = 1, 1 2 , 1 4 , 1 8 for (a), (b), (c), (d), respectively.
When progressing from (a) to (d), the part of the distribution corresponding to active ReLU (shaded region) becomes either overly dominant (blue distribution) or negligible (red distribution). In either case, the channel-wise distribution ends up concentrated on only one side of piece-wise linearity.
As an additional consequence, yl is guaranteed to have its channels well balanced with equal power Pc(yl) for all c. As a result, the network effectively uses its whole width. This is opposite to the situation where a single channel c becomes overly dominant over the others with Pc(yl) Pc′(yl) for c 6= c′, which results in downstream layers only “seeing” this channel c and the network behaving as if it had a width equal to one at layer l.
Revisiting BN’s preservation of expressivity. When BN is used, P(1)c (yl) = 0 implies for all c that the terms P(2)c (yl),P(3)c (yl),P(4)c (yl) sum to one. Apart from that, BN does not impose any particular constraints on the relative proportions of each term into the sum. This means that the relative proportions of P(i)c (yl) and P(i)(yl) for i ∈ {2, 3, 4} are free to evolve as naturally dictated by the task and the optimizer during learning.
This absence of constraints seems sensible. Indeed, imposing constraints on these relative proportions would alter the expressivity, which would not have any obvious justification in general and could even be detrimental in some cases, as we discuss in Section 4.

heading: 4 Failure modes with batch-independent normalization
text: With our theoretical framework in hand, we now turn to showing that the prototypical batchindependent norms are subject to failures modes opposite to BN’s beneficial properties.
In the case of LN, the failure mode does not manifest in an absolute sense but rather as a “soft” inductive bias, i.e. as a preference or a favoring in the hypothesis space. This “soft” inductive bias is quantified by Theorem 1 in the context of networks with random model parameters.
Definition 1 (random net). We define a “random net” as a neural network having an input x sampled from the dataset D and implementing Eq. (2), (3), (4) in every layer up to depth L, with the components of ωl,γl,βl at every layer l ∈ {1, . . . , L} sampled i.i.d. from the fixed distributions νω ,νγ ,νβ (up to a fan-in’s square root scaling for ωl).
In such networks, we assume that: (i) none of the inputs in the dataset D are identically zero; (ii) νω , νβ, νγ have well-defined moments, with strictly positive associated root mean squares ω, γ, β > 0; (iii) νω , νβ are symmetric around zero.
Theorem 1 (layer-normalized networks collapse (informal)). [D.3] Fix a layer l ∈ {1, . . . , L} and νω, νβ, νγ , D in Definition 1. Further suppose Norm = LN and suppose that the convolution of Eq. (2) uses periodic boundary conditions.
Then for random nets of Definition 1, it holds when widths are large enough that
P(yl)− P(1)(yl) . ρl−1, P(yl) ' 1, (6)
where ρ ≡ γ2/(γ2 + β2) < 1, and . and ' denote inequality and equality up to arbitrarily small constants with probability arbitrarily close to 1 when min1≤k≤l Ck is large enough.
Discussion on LN’s failure mode. Theorem 1 implies that, with high probability, yl is subject to channel-wise collapse in deep layers (l 1) with P(yl)−P(1)(yl) P(yl). This means that ỹl (that is only one affine transformation away from yl) is likely to have its channel-wise distributions collapsed with Pc(ỹl)−P(1)c (ỹl) Pc(ỹl) for most c. The nonlinearity φ acting on ỹl is then likely to be close to linear with respect to ỹl’s channel-wise distributions [C.2]. Being close to channel-wise linear in deep layers, layer-normalized networks are unable to effectively use their whole depth.
Since the inequality . can be replaced by an equality ' in the case φ = identity of Theorem 1 [D.4], the aggravation at each layer l of the upper bound of Eq. (6) does not stem from the activation function itself but rather by the preceding affine transformation. The phenomenon of channel-wise collapse — also known under the terms of “domain collapse” [39] or “elimination singularity” [18] — is therefore not only induced by a “mean-shifting” activation function such as φ = ReLU [62, 20], but also by the injection of non-centeredness through the application of the channel-wise shift parameter βl at each layer l. The fact that the general case of positive homogeneous φ is upper bounded by the case φ = identity in Eq. (6) still means that the choice φ = ReLU can only be an aggravating factor.
Crucially, in the context of random nets of large widths, LN’s operation at each layer l does not compensate this “mean shift”. This comes from the fact that LN’s mean and variance statistics can be approximated by zero and a constant value independent of x, respectively. This means that LN’s operation can be approximated by a layer-wise constant scaling independent of x.3
The predominance of LN’s failure mode in the hypothesis space — implied by its predominance in random nets — is expected to have at least two negative effects on the actual learning and final performance: (i) being expected along the training trajectory and being associated with reduced effective capacity, the failure mode is expected to cause degraded performance on the training loss; (ii) even if avoided to some extent along the training trajectory, the failure mode is still expected in the vicinity of this training trajectory, implying an ill-conditioning of the loss landscape [62, 64] and a prohibition of large learning rates that could have led otherwise to generalization benefits [65].
After detailing LN’s failure mode, we now detail IN’s failure mode. Theorem 2 (instance-normalized networks lack variability in instance statistics). [E] Fix a layer l ∈ {1, . . . , L} and lift any assumptions on φ. Further suppose Norm = IN, with Eq. (3) having nonzero denominator at layer l for all inputs and channels.
Then it holds that
• yl is normalized in each channel c with
P(1)c (yl) = 0, Pc(yl) = 1;
• yl lacks variability in instance statistics in each channel c with
P(2)c (yl) = 0, P(3)c (yl) = 1, P(4)c (yl) = 0.
Discussion on IN’s failure mode. We see in Theorem 2 that yl’s power decomposition with IN is constrained to be such that P(1)(yl),P(2)(yl),P(4)(yl) = 0 and P(3)(yl) = 1. While the constraints on P(1)(ỹl), P(3)(ỹl) are removed by the affine transformation between yl and ỹl, the constraints on P(2)(ỹl), P(4)(ỹl), on the other hand, remain even after the affine transformation. These constraints on P(2)(ỹl), P(4)(ỹl) translate into 2Cl fixed constraints in activation space that apply to each ỹl ∈ RH×W×Cl associated with each choice of input x in the dataset D [53]. Such constraints on ỹl are symptomatic of an alteration of expressivity. They notably entail that some network mappings that can be expressed without normalization cannot be expressed with IN. One such example is the identity mapping [C.3]. Another such example is a network mapping that would provide in channel c through ỹlα,c, just before the nonlinearity, a detector of a given concept at position α in the input x. With IN, the lack of variability in instance statistics implies that the mean µx,c(ỹ
l) and standard deviation σx,c(ỹl) of the feature map in channel c are necessarily constant for all x, equal to βc and γc, respectively. This does not allow to express for some inputs x the presence of the concept at some position α: µx,c(ỹl) > 0, σx,c(ỹl) > 0; and for other inputs x the absence of the concept: µx,c(ỹl) = 0, σx,c(ỹl) = 0.
3In this view, we expect layer-normalized networks to be also subject to a phenomenon of increasingly imbalanced channels with depth [46, 63, 48].
This latter example is not just anecdotal. Indeed, it is accepted that networks trained on high-level conceptual tasks have their initial layers related to low-level features and their deep layers related to high-level concepts [66, 67]. This view explains the success of IN on the low-level task of style transfer with fixed “style” input, IN being then incorporated inside a generator network that only acts on the low-level features of the “content” input [10, 68, 69, 70]. On high-level conceptual tasks, on the other hand, this view hints at a harmful tension between IN’s constraints and the requirement of instance variability to express high-level concepts in deep layers. In short, with IN not only is the expressivity altered, but the alteration of expressivity results in the exclusion of useful network mappings.
Failure modes with GN. Group Normalization is a middle ground between the two extremes of LN (G = 1 group) and IN (G = Cl groups at layer l). Networks with GN are consequently affected by both failure modes of Theorem 1 and Theorem 2, but to a lesser extent than networks with LN for the failure mode of Theorem 1 and IN for the failure mode of Theorem 2. On the one hand, since GN becomes equivalent to a constant scaling when group sizes become large, networks with GN are likely to be subject to channel-wise collapse. On the other hand, since GN can be seen as removing a fraction — with an inverse dependence on the group size — of P(2)(yl), P(4)(yl) in between each layer, networks with GN are likely to lack variability in instance statistics.
The balance struck by GN between the two failure modes of LN and IN could still be beneficial, which would explain GN’s superior performance in practice. It makes sense intuitively that being weakly subject to two failure modes is preferable over being strongly subject to one failure mode.
Experimental validation. The “power plots” of Figure 2 show the power decomposition of yl as a function of the depth l in both a random net of Definition 1 and ResNet-50 (v2) [71] trained on ImageNet.4 Looking at the cases of BN and IN, LN, GN in these power plots, we confirm that: (i) unlike networks with BN and IN, networks with LN, and to a lesser extent GN, are subject to channel-wise collapse as depth increases (see Appendix A.2.1 for a precise verification of Theorem 1); (ii) networks with IN, and to a lesser extent GN, lack variability in instance statistics.
4To ensure that activation steps are directly preceded by normalization steps (cf Section 5), we always use v2 instantiations of ResNets and instantiations of ResNeXts having the same reordering of operations inside residual blocks as ResNets v2 (cf Appendix A.1).

heading: 5 Proxy Normalization
text: With the goal of remedying the failure modes of Section 4, we now introduce our novel technique “Proxy Normalization” (PN) that is at the core of our batch-independent normalization approach.
PN is incorporated into the neural network by replacing the activation step of Eq. (4) by the following “proxy-normalized activation step” (cf Figure 3 and the practical implementation of Appendix B):
z̃l = PN-Act(yl), ∀α, c : PN-Act(yl)α,c = φ ( γlcy l α,c + β l c ) − EY lc [ φ ( γlcY l c + β l c) ]
√ VarY lc [ φ ( γlcY l c + β l c) ] + , (7)
where ≥ 0 is a numerical stability constant and Y lc is a Gaussian “proxy” variable of mean β̃lc and variance (1 + γ̃lc)
2 depending on the additional parameters β̃l, γ̃l ∈ RCl of PN. Unless otherwise stated, we let β̃l, γ̃l be nonzero but still subject to weight decay and thus close to zero. We show in Appendix A.3.3 that it is also effective to let β̃l, γ̃l be strictly zero and Y lc ∼ N (0, 1). If we assume (as hinted by Section 4) that only the affine transformation and the activation function φ (i.e. the activation step) play a role in the aggravation at each layer l of channel-wise collapse and channel imbalance, then PN provides the following guarantee of channel-wise normalization. Theorem 3 (guarantee of channel-wise normalization in proxy-normalized networks (informal)). [F] Fix a layer l ∈ {1, . . . , L} and lift any assumptions on φ and x’s distribution. Further suppose that the neural network implements Eq. (2), (3), (7) at every layer up to depth L, with = 0 and Eq. (3), (7) having nonzero denominators for all layers, inputs and channels.
Then both yl and z̃l at layer l are channel-wise normalized if the following conditions hold:
(i) z̃l−1 at layer l − 1 is channel-wise normalized; (ii) The convolution and normalization steps at layer l do not cause any aggravation of channel-
wise collapse and channel imbalance;
(iii) yl at layer l is channel-wise Gaussian and PN’s additional parameters β̃l, γ̃l are zero.
Our batch-independent approach: LN+PN or GN+PN. At this point, we crucially note that PN: (i) is batch-independent; (ii) does not cause any alteration of expressivity. This leads us to adopt a batchindependent normalization approach that uses either LN or GN (with few groups) in the normalization step and that replaces the activation step by the proxy-normalized activation step (+PN). With such a choice of normalization step, we guarantee three of the benefits detailed in Table 1: “scale invariance”, “control of activation scale” and “preservation of expressivity”. With the proxy-normalized activation step, we finally guarantee the fourth benefit of “avoidance of collapse” without compromising any of the benefits provided by the normalization step.
Experimental validation. We confirm in Figure 2 that BN’s behavior is emulated in a fully batch-independent manner with our approach, LN+PN or GN+PN. Indeed, the power plots of networks with LN+PN resemble the power plots of networks with BN. As desired, PN remedies LN’s failure mode without incurring IN’s failure mode.
Approach strength 1: Normalizing beyond initialization. Our batch-independent approach maintains channel-wise normalization throughout training (cf Figure 2). In contrast, many alternative approaches, either explicitly or implicitly, focus on initialization [21, 24, 17, 18, 49, 20, 72]. Centered Weight Normalization [17], WS [18, 20, 72] or PreLayerNorm [49] notably rely on the implicit assumption that different channels have the same channel-wise means after the activation function φ. While valid in networks at initialization with βl = 0 and γl = 1 (cf Appendix A.2.2), this assumption becomes less valid as the affine transformation starts deviating from the identity. We see in Figure 2 that networks with LN+WS are indeed less effective in maintaining channel-wise normalization, both in networks with random βl, γl (top row) and in networks considered throughout training (bottom row). Such a coarser channel-wise normalization might in turn lead to a less effective use of model capacity and a degradation of the conditioning of the loss landscape [62, 64].
Approach strength 2: Wide applicability. Our batch-independent approach matches BN with consistency across choices of model architectures, model sizes and activation functions (cf Section 6). Its only restriction, namely that activation steps should be immediately preceded by normalization steps for yl and its associated proxy Y l to be at the same scale, has easy workarounds. Applicability restrictions might be more serious with alternative approaches: (i) alternative approaches involving a normalization in weight space [16, 17, 18, 19, 20] might be ill-suited to architectures with less “dense” kernels such as EfficientNets [20]; (ii) approaches involving the tracking of activation statistics [21, 22, 23] might be nontrivial to apply to residual networks [73, 74]; (iii) approaches involving a change of activation function [24, 5, 15] might precisely restrict the choice of activation function.
Approach strength 3: Ease of implementation. Our approach is straightforward to implement when starting from a batch-normalized network. It simply requires: (i) replacing all BN steps with LN or GN steps, and (ii) replacing all activation steps with proxy-normalized activation steps. The proxy-normalized activation steps themselves are easily implemented (cf Appendix B).

heading: 6 Results
text: We finally assess the practical performance of our batch-independent normalization approach. While we focus on ImageNet [75] in the main text of this paper, we report in Appendix A some additional results on CIFAR [76]. In Appendix A, we also provide all the details on our experimental setup.
Choices of regularization and batch size. As mentioned in Section 2, we perform all our experiments with different degrees of regularization to disentangle normalization’s effects from regularization’s effects. We detail all our choices of regularization in Appendices A.1 and A.3.4.
In terms of batch size, we set: (i) the “global batch size” in between weight updates to the same value independently of the choice of norm; (ii) the “normalization batch size” to a near-optimal value with BN. These choices enable us to be conservative when concluding on a potential advantage of our approach over BN at small batch size. Indeed, while the performance of batch-independent approaches would remain the same or slightly improve at small batch size [77, 65], the performance of BN would eventually degrade due to a regularization that eventually becomes excessive [2, 3, 4, 5, 6, 7].
Effect of adding PN. We start as a first experiment by analyzing the effect of adding PN on top of various norms in ResNet-50 (RN50). As visible in Table 2, PN is mostly beneficial when added on top of LN or GN with a small number of groups G. The consequence is that the optimal G shifts to lower values in GN+PN compared to GN. This confirms the view that PN’s benefit lies in addressing LN’s failure mode.
It is also visible in Table 2 that PN does not provide noticeable benefits to BN. This confirms again the view that PN’s benefit lies in addressing the problem — not present with BN — of channel-wise collapse. Importantly, since PN does not entail effects other than normalization that could artificially boost the performance, GN+PN can be compared in a fair way to BN when assessing the effectiveness of normalization.
This is unlike WS which has been shown to improve BN’s performance [18]. In our results of Table 2, the high performance of GN+WS without extra regularization and the fact that PN still provides benefits to GN+WS suggests that: (i) on top of its normalization benefits, WS induces a form of regularization; (ii) GN+WS is still not fully optimal in terms of normalization.
GN+PN vs. BN. Next, we turn to comparing the performance of our batch-independent approach, GN+PN, to that of BN across a broad range of models trained on ImageNet. As visible in Figure 4 and Tables 3, 4, GN+PN outperforms BN in ResNet-50 (RN50) and ResNet-101 (RN101) [71], matches BN in ResNeXt-50 (RNX50) and ResNeXt-101 (RNX101) [78], and matches BN in EfficientNet-B0 (EN-B0) and EfficientNet-B2 (EN-B2), both in the original variant with depthwise convolutions and
expansion ratio of 6 [52] and in an approximately parameter-preserving variant (cf Appendix A.1) with group convolutions of group size 16 and expansion ratio of 4 [79]. In short, our batch-independent normalization approach, GN+PN, matches BN not only in behavior but also in performance.
With regard to matching BN’s performance with alternative norms, various positive results have been reported in ResNets and ResNeXts [8, 7, 12, 13, 15, 20, 36] but only a limited number in EfficientNets [15]. In EfficientNets, we are notably not aware of any other work showing that BN’s performance can be matched with a batch-independent approach. As a confirmation, we assess the performance of various existing batch-independent approaches: GN [3], GN+WS [18], Evo-S0 [15], FRN+TLU [80, 5]. Unlike GN+PN, none of these approaches is found in Table 4 to match BN with consistency.
Normalization and regularization. Our results suggest that while an efficient normalization is not sufficient in itself to achieve good performance on ImageNet, it is still a necessary condition, together with regularization. In our results, it is always with extra regularization that GN+PN yields the most benefits. Importantly, the fact that GN+PN consistently leads to large improvements in training accuracy (cf Appendix A.3.2) suggests that additional benefits would be obtained on larger datasets without the requirement of relying on regularization [81, 72].
Table 3: BN vs. GN, GN+PN. ResNets and ResNeXts are trained on ImageNet with BN and GN, GN+PN. Results are formatted as in Table 2.
RN50 RN101 RNX50 RNX101
BN 76.3 / 75.8 77.9 / 78.0 77.6 / 77.2 78.7 / 78.9 GN 75.4 / 75.3 77.0 / 77.4 76.2 / 76.6 77.4 / 78.1 GN+PN 76.3 / 76.7 77.6 / 78.6 76.7 / 77.8 77.7 / 79.0

heading: 7 Summary and broader impact
text: We have introduced a novel framework to finely characterize the various neural network properties affected by the choice of normalization. Using this framework, we have shown that while BN’s beneficial properties are not retained when solely using the prototypical batch-independent norms, they are retained when combining some of these norms with the technique hereby introduced of Proxy Normalization. We have demonstrated on an extensive set of experiments that our batch-independent normalization approach consistently matches BN in both behavior and performance.
The main implications of this work could stem from the unlocked possibility to retain BN’s normalization benefits while removing batch dependence. Firstly, our approach could be used to retain BN’s normalization benefits while alleviating the burden of large activation memory stemming from BN’s requirement of sufficiently large batch sizes. This is expected to be important in memory-intensive applications such as object detection or image segmentation, but also when using A.I. accelerators that leverage local memory to provide extra acceleration and energy savings in exchange for tighter memory constraints. Secondly, our approach could be used to retain BN’s normalization benefits while avoiding BN’s regularization when the latter is detrimental. As discussed in Section 6, this is expected to be important in the context — that will likely be prevalent in the future — of large datasets.
Acknowledgments and Disclosure of Funding
We are thankful to Simon Knowles, Luke Hudlass-Galley, Luke Prince, Alexandros Koliousis, Anastasia Dietrich and the wider research team at Graphcore for the useful discussions and feedbacks. We are also thankful to the anonymous reviewers for their insightful comments that helped improve the paper.

Review summaries:
This paper analyzes the properties of normalization methods. It shows how layer normalization induces a collapse towards channel-wise constant functions with assumptions, and how the constraint imposed by normalization affects the performance of instance normalization. It also proposes Proxy Normalization that emulates batch normalization’s behavior and performance, when combined with batch-independent normalization (e.g, layer normalization or group normalization) .

This work proposes some theoretical analyses on different normalization schemes in deep neural networks. It found that LN suffers from the collapsing effect over the number of layers, which makes the computation more and more linear, whereas IN suffers from a lack of expressivity. Towards fixing these issues, the paper proposed ProxyNorm that adds another normalization layer after the activation function. Empirical results suggest that the combination of ProxyNorm and LayerNorm or GroupNorm can lead to matching or sometimes superior performance compared to BatchNorm.

This paper studies normalization strategies for deep residual networks, both theoretically and empirically, and finds the strategies that don't rely on batch-statistics to have particular failure modes, the two most notable ones being channel-wise collapse, where instances collapse to a constant in each channel, and reduction in instance-variability, where the first and second order statistics per channel are identical for each instance, thereby reducing expressivity and the ability to recognize higher order concepts. The paper then proceeds to propose proxy-normalization, a normalization method suitable for small-batch learning (i.e., not relying on batch-statistics) that, when combined with other normalization strategies, such as Layer Norm or Group Norm, alleviates these failure modes. On ImageNet and CIFAR experiments with deep residual networks (RN50 and larger), the work finds that Proxy-Normalization (with sufficient additional regularization such as label-smoothing, dropout, and stochastic depth), can be used to match the performance of BN without relying on batch statistics. The proposed method standardizes the activations of the network according to the mean and variance of a channel-wise gaussian random variable, with learnable mean and variance parameters.