Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependence
Authors: Antoine Labatie
Dominic Masters
Zach Eaton-Rosen
Carlo Luschi
Venue: NeurIPS 2021 Poster (2021)
Keywords: Batch-Independent Normalization
Batch Normalization
Layer Normalization
Group Normalization
Instance Normalization
Deep Learning Theory
CNNs
ResNets
ResNeXts
EfficientNets

Abstract: We investigate the reasons for the performance degradation incurred with batch-independent normalization. We find that the prototypical techniques of layer normalization and instance normalization both induce the appearance of failure modes in the neural network's pre-activations: (i) layer normalization induces a collapse towards channel-wise constant functions; (ii) instance normalization induces a lack of variability in instance statistics, symptomatic of an alteration of the expressivity. To alleviate failure mode (i) without aggravating failure mode (ii), we introduce the technique "Proxy Normalization" that normalizes post-activations using a proxy distribution. When combined with layer normalization or group normalization, this batch-independent normalization emulates batch normalization's behavior and consistently matches or exceeds its performance.

Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependence
Antoine Labatie
Zach Eaton-Rosen Carlo Luschi Graphcore Research, UK antoine wabatiieecaarralens.... {dominicm, zacher, carlo))graphcoree .i
Dominic Masters
Abstract
We investigate the reasons for the performance degradation incurred with batch- independent normalization. We find that the prototypical techniques of layer normalization and instance normalization both induce the appearance of failure modes in the neural network's 's pre- are-activations: (i) layer normalization induces collapse towards channel-wise constant functions; (ii) instance normalization induces lack of variability in instance statistics, symptomatic of an alteration the expressivity. To alleviate failure mode (i) without aggravating failure mode (ii), we introduce the technique "Proxy Normalization" that normalizes post-activations using proxy distribution. When combined with layer normalization or group nor- malization, this batch-independent normalization emulates batch normalization's ss behavior and consistently matches exceeds its performance.
Introduction
Normalization plays critical role in deep learning allows successful scaling large and deep models. In vision tasks, the most well-established normalization technique is Batch Normalization (BN) [1]. every layer in the network, BN normalizes the intermediate activations have zero mean and unit variance in each channel. While indisputably successful when training with large batch size, BN incurs a performance degradation the regime of small batch size [2, 3, 4, 6, 7J. This performance degradation is commonly attributed to an excessive or simply unwanted regularization stemming from the noise the approximation of bull-batch statisties by mini-batch statistics.
Many techniques have been proposed to avoid this issue, while at the same time retaining BN's benefits. Some techniques mimic BN's principle while decoupling the computational batch from the normalization batch [2, 8, 7]. Other techniques are "batch-indeped bbatch-independent" in that they operate indepen- dently of the batch in various modalities: through explicit normalization either in activation space [9, 10, 11, 12, 13, 3, 5, 14, 15, 6] or in weight space [16, 17, 18, 19, 20]; through the use of an analytic proxy to track activation statisties [21, 22, 23]; through a change of activation function [24, 5, 15].
In this paper, we push the endeavor to replace BN with a batch-independent normalization step further. Our main contributions characterize the various neural network properties affected by the choice of normalization; (ii) using this framework, show that while BN's beneficial properties the prototypical batch-independent norms, they are retained when combining some of these norms with "Proxy Normalization", a novel technique that hereby introduce; (iii) demonstrate an extensive set of experiments that, by reproducing BN's beneficial properties, our batch-independent normalization approach consistently matches exceeds BN's performance.
follows: (i) introduce novel framework to finely
retained when solely using
As a starting point of our analysis, we need to gain better understanding those beneficial properties of BN that we aim at reproducing.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

2 Batch Normalization's beneficial properties
We consider throughout this paper convolutional neural network with d spatial axes. This neural network receives an input x E IF #x XCC which, unless otherwise stated, is assumed sampled from a finite dataset D. The neural network maps this input x to intermediate activations x IR WxxC x of height H, width W and number of channels Cj at each layer I. The value of x' at spatial position (l..... H} x {l...., }) and channel e E (I....,Cr Ci} is denoted as x with the dependency on x kept implicit to avoid overloading notations.
The inclusion of BN layer I leads in the full-batch setting to adding the following operations Va,
x a.c xxxxx)
Mx(x')
ya.-
y/cc = =Yyo' + 3%
(1)
where (=(x'), oz are the mean and standard deviation of x' in channel c, and y': 3 are channel- wise scale and shift parameters restoring the degrees of freedom lost in the standardization. In the mini-batch setting, the full-batch statistics [=(x'), ox(xx) are approximated by mini- statistics.
Table 1 summarizes the beneficial properties that result from including BN in the neural network. Below, we provide details on each of these properties, and we discuss whether each property is reproduced with batch-independent norms.
Scale invariance. When BN is present, the input-output mapping of the neural network is invariant to the scale of weights preceding any BN layer. With such scale invariance plus weight decay, the scale of weights during training reaches an equilibrium with an "effective" learning rate depending on both the learning rate and the weight decay strength [25, 26, 27, 8, 28, 29, 30J. Such mechanism of "auto rate-tuning" has been shown provide optimization benefits [31, 26, 32].
This property is easy to reproduce. It is already obtained with most existing batch- sancch-ideeedeent norms.
Control of activation scale in residual networks. To be trainable, residual networks require the scale of activations to remain well-behaved at initialization [33, 34, 35, 20, 36, 37J. While this property naturally arises when BN is present on the residual path, when BN is not present it can also be enforced by a proper scaling decaying with the depth of the residual path. This "dilution" of the residual path with respect to the skip connection path reduces the effective depth of the network and enables to avoid coarse- graried failures modes [38, 39, 40, 41, 42, 35].
This property is easy to reproduce. It is already obtained with most existing batch-independent norms.
Regularizing noise. Due the stochasticity of the approximation of px(x'), (x(x') by mini-batch statistics, training a neural network with BN in the mini-batch setting can be seen as equivalent to performing Bayesian inference [43, 44] or adding regularizing term to the training of the same network with full-batch statistics [45]. As a result, BN induces a specific form of regularization.
This regularization is not reproduced with butch-independder norms, but we leave it out of the scope of this paper. To help minimize the bias in our analysis and "subtract away" this effect, we will perform all our experiments without and with extra degrees of regularization. This procedure can be seen as a coarse disentanglement of normalization's 's effects from regularization' 's effects.
Avoidance of collapse. Unnormalized networks with non non-saturating nonlinearities are subject to a phenomenon of "collapse" whereby the distribution with respect to x, a of the intermediate activation
vectors (x.. (x..1...., xa.c,) )T becomes close to zero- or one-dimensional in deep layers [39, 46, 41, 47, 48, 49,37,, 37J. This means that deep in an unnormalized network: (i) layers tend to have their channels imbalanced; (ii) nonlinearities tend to become channel- -wse linear with respect to x, a and not add any effective capacity [39, 50, 51]. Consequently, unnormalized networks can neither effectively use their whole width (imbalanced channels) nor effectively use their whole depth (channel-wis lincarity).
Conversely, when BN is used, the standardization at each layer prevents this collapse from happening, Even in deep layers, channels remain balanced and nonlinearities remain channel- -wise nonlinear with respect to x, a. Consequently, networks with BN can effectively use their whole width and depth.
The collapse is, on the other hand, not always avoided with batch- inahhiindeeendent norms [39, 18, 49]. Most notably. it is not avoided with Layer Normalization (LN) [9] Group Normalization (GN) [3], as we show both theoretically and experimentally on commonly found networks in Section 4.
To the extent possible, we aim at designing a batch- inceeddnddeenndent norm that avoids this collapse.

---

Table 1: BN's beneficial properties. We show whether each property is (at least approximately) present (v) or absent (X) with various batch-independen norms: Layer Normalization (LN), Instance Normalization (IN), Layer Normalization + Proxy Normalization (LN+PN, of Section 5). In this categorization, BN is considered in the mini-batch setting but still close to the full-batch setting, such that it approximately preserves expressivity [53].
Control of invariance activation scale
Scale
Regularizing noise
Avoidance of collapse
Preservation of expressivity
BN LN IN LN+PN
X
V
Preservation of expressivity. We can always express the identity with Eq. (1) by choosing 3¹ = =x(xx") and > ox(xx). Conversely, for any choice of B² Y we can always "re-absorb" Eq. (1) into preceding convolution with bias. This means that BN in the full-batch setting does not alter the expressivity compared to an unnormalized network, i.e. amounts to plain reparameteriza- tion of the hypothesis space.
The expressivity is, the other hand, always preserved with batch-independent norms. In activa- tion space, the dependence of aatch-independent statistics on the input x turns the standardization into thannel-wise nonlinear operation that cannot be "re-absorbed" into preceding convolution with bias [18]. This phenomenon is most pronounced when statistics get computed over few components. This means e.g. that Instance Normalization (IN) [10] induces a greater change of expressivity than GN, which itself induces a greater change of expressivity than LN.
In weight space, the expressivity can also be altered, namely by the removal of degrees of freedom. This is the case with Weight Standardization (WS) [18, 20] and Centered Weight Normalization [17] that remove degrees of freedom (one per unit) that cannot be restored in succeeding affine trans- formation. This reduction of expressivity could explain the ineffectiveness of these techniques in EfficientNets [52], previous works observed [20] and
confirm in Section 6.
To the extent possible, we aim at designing a batch-independen norm that preserves expressivity.
Theoretical framework of analysis
We specified the different properties that wish to retain in our design of batch-inde aatch-independent normalization: (i) scale invariance, (ii) control of activation scale; (iii) avoidance of collapse; (iv) preservation of expressivity. We now introduce framework quantify the presence absence of the specific properties (iii) and (iv) with various choices of normalization.
Propagation. For simplicity, we assume in theoretical setup that any layer up to depth L consists of the following three steps: (i) convolution step with weights w/ E K; x xXxxK C0-1x-xx xxi; (ii) normalization step; (iii) activation step sub-decomposed into affine transformation with scale and shift parameters > 3 E RCi and an activation function ø which, unless otherwise stated, is assumed positive homogeneous and nonzero (e.g. ReLU). If we denote x'. .y', z' IR Hx HxWXX, the intermediate activations situated just after (i), (ii), (iti) with the convention z = X, we may write the propagation through layer / as
x Conv(z!
va,c: Conv(z!-t)ace = =ww' xxl-1) Daye:
Va,c: Norm(x (x,)xx)))))))))))))) x 1 (xe)
y' = Norm(x'),
(3)
T (x!)
z' = -ct(('y'),
va,c: Act(y') = 6(F = o By),
(4)
where 11, (x)), (xx.)())) denote the mean and standard deviation of x' conditionally on Tx,c = [c].
{x), {x, c). {x, mod G} for the respective cases Norm = BN, LN, IN, GN with G groups.
We omit the numerical stability constant and adopt the convention Norm (omm(x))..c )a.c Vo if 12,0 (x') 0.

---

Moments. Extending the previous notations, we use μ., の, P indexed with a (possibly empty) subset of variables denote the operators of conditional mean, standard deviation and power. If we apply these operators to the intermediate activations y", that implicitly depend on the input x and that explicitly depend on the spatial position a and the channel e, we get e.g.
Me(y') Ex.a(Vaa.c)
aely') V Varx,a (orxxa((yddd [yd.c)
Poly') Ex,a [ (,,))) )
He Ho(y!)2 (grl)2
Mx,cly') Ba [Va.c]
xx,cly")
=V Vara (ona(yaas [y
Py.) E,g [[y
"x,c(yy) (24 (yr))2
where, by convention, a, are considered uniformly sampled among inputs of DD spatial positions and channels, whenever they are considered as random.
Power decomposition. Using these notations, we may gain important insights by decomposing the power in channel e of y', just after the normalization step, as
Py(y') Ex [Mxco(rr)] Varx [xx.o(l))
Ex [oxx(x1))] + Varx [oxxo(r')]
(5)
(y))
(yd)
p (y!)
(y!)
pe (y!)
Since this four-terms power decomposition will be at the core of our analysis, we detail two useful views of it. The first view is that of a hierarchy of scales: (1)) (y") measures the power of (e(")) at the instance scale; the sum of y'a.-- 5ixc-Mxx_ll (y") at the pixel scale. A particular situation
the dataset scale; p!?) px²)(xx)) (x-) measures the power of (x,c(y") -ye)) (e('"))
p! (y") and p! (y*) measures the power of
where the power would be concentrated at the dataset scale with P! 1)) (y'') equal to (yy') would imply that y has its distribution fully "collapsed" in channel c, i.e. that y is constant in channel c.
The second view is that of a two-level binary tree: on one half of the tree, the sum of p( (y') and p! (y") measures the power coming from (y")) with the relative proportions of P 1) (y") and p!.) (y))
(y)) functions of the inter-x similarity and inter-x variability of Mx,c(y'), ); on the other half of the
tree, the sum of (3) (x") and (!4)) (y²) measures the power coming from xx,e(y"), ), with the relative proportions of pP(3) (y") and p!!") (y") functions of the inter-x similarity and inter-x variability of (y") A particular situation where ((()(yy)) p! (y") would be equal to zero would imply that Mxxe(y"), (x") have zero inter-x variability, i.e. that (xxc(y"), xx,c(y") are constant for all x.
A version of Eq. (5) at the layer level instead of channel level will be easier to work with. Defining plo) (x') as the averages of p!²()x)) (y") over e for E {1, 2,, 4). we obtain
P(y') (()) (y²)) (y') + p(3)
p(
should be noted that Ply') = for any choice of Norm E {BN, LN, IN, GN) as long as the denominator of Eq. (3) is nonzero for all x, c [C.1]. Consequently, the terms p(i sum to one, meaning they can be conveniently seen as the proportion of each term E 2,,2,3, 4} into P(y").
Revisiting BN's avoidance of collapse. When BN is used, y' is normalized not only layer-wise but also channel-wise with PE") (y') = 0 and Po(y') = 1. As a first consequence, ý' (that is only one affine transformation away from y²) is unlikely to have its channel-wise distributions collapsed. This means that the nonlinearity & acting on ÿ is likely to be effectively nonlinear with respect to 5''s whan distributions." As a result, each layer adds capacity and the network effectively uses its whole depth. This is opposite to the situation where y has its channel-wise distributions collapsed with Po(5") - _! (5') < Py')") for all c, which results in ø being close to linear with respect to ' channel-wise distributions. This is illustrated in Figure and formalized in Appendix C.2.
"Note that: (i) the effective nonlinearity of á with respect to yss channel- distributions could be quantified in the context of random nets of Definition 1 with "reasonable" choices of B, 7: (ii) BN only guarantees an intra distr-dittibutionn nonlinearity and not an intra-mode nonlinearity in contexts such as adversarial training [54, 55] or conditional GANs [56]. unless modes are decoupled in BN's computation [57, 58, 59, 60, 61].

---

(a)
(b)
(0)
(d)
Figure I: Channel-wise collapse induces channel-wise linearity. Each subplot shows o = ReLU (black activation function) as well as two channel-wist wise distributions (blue and red distributions)
positioned symmetrically around with - (s') 1, 2: for (a), (b), (c), (d), respectively, When progressing from (a) to (d), the part of the distribution corresponding to active ReLU (shaded region) becomes either overly dominant (blue distribution) or negligible (red distribution). In either case, the channel-wit distribution ends up concentrated on only one side of piece-wise linearity.
P.(
As an additional consequence, y' is guaranteed to have its channels well balanced with equal power Pe(r"') for all As a result, the network effectively uses its whole width. This is opposite to the situation where single channel becomes overly dominant over the others with Pe Pa(y") >> Pe (y") for e A d', which results in downstream layers only "seeing" this channel and the network behaving as if it had width equal to at layer l.
Revisiting BN's preservation of expressivity. When BN is used, (y") 0 implies for all that the terms p!2) (y"), p( (y"), P! (y') sum to one. Apart from that, BN does not impose any particular constraints on the relative proportions of each term into the sum. This means that the relative proportions of p!! (y") and p(ù) (y') for i f2, 4) are free evolve as naturally dictated by the task and the optimizer during learning.
This absence of constraints seems sensible. Indeed, imposing constraints these relative proportions would alter the expressivity, which would not have any obvious justification in general and could even be detrimental in some cases, discuss in Section
Failure modes with batch-independent normalization
With our theoretical framework in hand, we now turn to showing that the prototypical batch- independent norms subject to failures modes opposite BN's beneficial properties.
In the case of LN, the failure mode does not manifest in an absolute sense but rather as "soft" inductive bias, i.e. as preference or favoring in the hypothesis space. This "soft" inductive bias is quantified by Theorem in the context of networks with random model parameters.
Definition (random net). We define a "random net" as neural network having an input x sampled from the dataset D and implementing Eq. (2), (3), (4) in every layer up depth L. with the components of w',n"', ,.,//,/ at every layer i E {1..... ...,. L} sampled i..dd from the fixed distributions Va, Ug (up to a fan-in' square root scaling for w²)
In such networks, we assume that: (i) none of the inputs in the dataset D are identically zero; (üi Va- ug, Vy have well-defindd moments, with strictly positive associated root mean squares w, 7r, 3 >
(iii) Pws va are symmetric around zero.
Theorem (layer-normalized networks collapse (informal)) [D.3] Fix layer I E {l,..., L} and Vwa V3, Vyp in Definition! Further suppose Norm LN and suppose that the convolution of Eq. (2) uses periodic boundary conditions.
Then for random nets of Definition 1, it holds when widths are large enough that
P(y') (p(((('))) S p'1
P(y') 1,
(6)
where p= 7=22/62 32) 1, and S and denote inequality and equality up to arbitrarily small constants with probability arbitrarily close to I when minj Cy is large enough
5

---

Discussion on LN's failure mode. Theorem 1 implies that, with high probability, y' is subject to channel-wise collapse in deep layers (1 >>>>/ 1) with P(y') (y") < P(y"). This means that y (that is only one affine transformation away from y") is likely to have its channel-wise distributions collapsed with Po(5") - (5') < (5))) for most c. The nonlinearity @ acting on y' is then likely to be close to linear with respect to y''s channel-wise distributions [C.2]. Being close to channel-wise linear in deep layers, layer-normalized networks are unable to effectively use their whole depth.
Since the inequality X can be replaced by an equality > in the case Φ = identity of Theorem 1 [D.4], the aggravation each layer of the upper bound of Eq. (6) does not stem from the activation function itself but rather by the preceding affine transformation. The phenomenon of channel-wise collapse
also known under the terms of "domain collapse" [39] or "elimination singularity" [18] is therefore not only induced by a "mean- -iiraan-iifting"" activation function such as Φ = ReLU [62, 20J. but also by the injection of non-centeredness through the application of the channel-wise shift parameter 3 at each layer l. The fact that the general case of positive homogeneous o is upper bounded by the case Φ = identity in Eq. (6) still means that the choice & ReLU can only be an aggravating factor.
Crucially, in the context of random nets of large widths, LN's operation at each layer does not compensate this "mean shift". This comes from the fact that LN's mean and variance statistics can be approximated by zero and a constant value independent of x, respectively. This means that LN's operation can be approximated by a layer-wise constant scaling independent of x.3
The predominance of LN's failure mode in the hypothesis space - implied by its predominance in random nets is expected to have at least two negative effects the actual learning and final performance: (i) being expected along the training trajectory and being associated with reduced effective capacity, the failure mode is expected to cause degraded performance on the training loss; (ii) even if avoided to some extent along the training trajectory, the failure mode is still expected in the vicinity of this training trajectory. implying ill-conditioning of the loss landscape [62, 64] and a prohibition of large learning rates that could have led otherwise to generalization benefits [65].
After detailing LN's failure mode, we now detail IN's failure mode.
Theorem 2 canstance-normalized networks lack variability in instance statistics). [E] Fix a layer I E {I.... L} and lift any assumptions on 6. Further suppose Norm = IN, with Eq. (3) having nonzero denominator at layer I for all inputs and channels.
Then it holds that
yd is normalized in each channel e with
p! = 0,
P.(y') =
y' lacks variability in instance statistics in each channel e with
p!.) (y') 0,
p(.) (y²) =1
P(u) = 0.
Discussion on IN's failure mode. We see in Theorem that y's power decomposition with IN is constrained to be such that (y"), (y"), p))))))))))
0 and p(3)(y') = 1. While the constraints on (y)) (54) are removed by the affine transformation between y' and ý the constraints on ((2) (f"), p(4) (F"). on the other hand, remain even after the affine transformation. These constraints p()) (5)), p()(()) translate into 2C₁ fixed constraints in activation space that
apply to each y' E R #XXXXX X associated with each choice of input x in the dataset D [53].
Such constraints on y" are symptomatic of alteration of expressivity. They notably entail that some network mappings that can be expressed without normalization cannot be expressed with IN. One such example is the identity mapping [C.3]. Another such example is network mapping that would provide in channel e through Ya,e-just just before the nonlinearity, a detector of a given concept at position a in the input x. With IN, the Jack of variability instance statistics implies that the mean (F") and standard deviation (x))("") of the feature map in channel e are necessarily constant for all x, equal to Be and Ye. respectively. This does not allow to express for some inputs x the presence of the concept at some position a: (5") > 0, xx)) > O; and for other inputs x the absence of the concept: oxx) = O, xx (5 O.
'In this view, we expect layer norr-noomalized networks to be also subject to a phenomenon of increasingly imbalanced channels with depth [46, 63, 48).

---

(y.)
p(al(y)
(y
0.10
0.5
0.05
10
15
10
15
5
10
15
5
10
15
Random net:
BN
IN
LN
GN
LN+PN
LN
(y!)
(y))
0.2
0.10
0.5
0.05
0.00
10
30
30
10
(10
40
RosNet-50
BN
LN GN
LN+PN
LN+WS
Figure "Power plots". The power decomposition of y shown as a function of the depth for BN and different batch-independent norms: IN, LN, GN, LN+PN, LN+WS. Top row: Random net of Defi- nition with á = ReL U, widths C = 1024, kernel sizes K; = 3, convolutions using periodic bound- ary conditions and components of 3 n sampled i..dd from (0, 0.2?) and (1, 0.2?), respectively, Bottom row: ResNet-50 (v2) throughout 100 epochs of training on ImageNet (IN is not shown in this row due to numerical stability issues). Further experimental details reported in Appendix A.2.
This latter example not just anecdotal. Indeed, accepted that networks trained on high-level conceptual tasks have their initial layers related to low-level features and their deep layers related to high-level concepts [66, 67]. This view explains the success IN on the low-level task of style transfer with fixed "style" input, IN being then incorporated inside a generator network that only acts on the low-level features of the "content" input [10, 68, 69, 70]. On high-level conceptual tasks, on the other hand, this view hints at a harmful tension between IN's constraints and the requirement of instance variability to express high-level concepts in deep layers. In short, with IN not only is the expressivity altered, but the alteration of expressivity results in the exclusion of useful network mappings.
Failure modes with GN. Group Normalization is middle ground between the two extremes of LN (G group) and IN (G C₁ groups at layer 1). Networks with GN are consequently affected by both failure modes of Theorem and Theorem 2, but to lesser extent than networks with LN for the failure mode of Theorem I and IN for the failure mode of Theorem 2. On the one hand, since GN becomes equivalent to constant scaling when group sizes become large, networks with GN are likely to be subject to channel-wise collapse. On the other hand, since GN can be seen as removing a fraction - with an inverse dependence on the group size of p(2) (y'), p(4) (y") in between each layer, networks with GN are likely to lack variability in instance statistics.
The balance struck by GN between the two failure modes of LN and IN could still be beneficial, which would explain GN's superior performance in practice. It makes sense intuitively that being weakly subject to two failure modes is preferable over being strongly subject to one failure mode.
Experimental validation. The "power plots" of Figure 2 show the power decomposition of y' as a function of the depth in both a random net of Definition and ResNet-50 (v2) [71] trained on ImageNet." Looking at the cases of BN and IN, LN, GN in these power plots, we confirm that: (i) unlike networks with BN and IN, networks with LN, and to a lesser extent GN, are subject to channel-wise collapse as depth increases (see Appendix A. for precise verification of Theorem I); (ii) networks with IN, and to a lesser extent GN, lack variability in instance statistics.
"To ensure that activation steps are directly preceded by normalization steps (cf Section 5), we always use v2 instantiations of ResNets and instantiations of ResNeXts having the same reordering of operations inside residual blocks as ResNets v2 (ef Appendix A.1).
7

---

5
Proxy Normalization
With the goal of remedying the failure modes of Section 4, we now introduce novel technique "Proxy Normalization" " (PN) that is at the core of our batch-independent normalization approach.
PN is incorporated into the neural network by replacing the activation step of Eq. (4) by the following activation step" (cf Figure and the practical implementation of Appendix B):
Va PN-Act(y")an = "(7's'. + +B,)) - Ey; [ (7)Y. + (8)
z' = PN-Accly"),
(7)
Vary, (arv:[@(11))) ( + +b)] +
where 2 0 is numerical stability constant and Y! is a Gaussian "proxy" variable of mean 3¹ and variance (1 + 52)² depending on the additional parameters 3 C of PN. Unless otherwise stated, we let 33, 5 be nonzero but still subject weight decay and thus close to zero. We show in Appendix 3.3.3 that it also effective to let 33, 7 be strictly zero and Y! ~ (O, 1).
If we assume (as hinted by Section 4) that only the affine transformation and the activation function ø (i.e. the activation step) play role in the aggravation each layer I of channel-wie collapse and channel imbalance, then PN provides the following guarantee of channel-wise normalization.
Theorem 3 (guarantee of channel- -wise normalization in proxy- noxmaazedd networks (informal)). [F] Fix a layer I E (I....,L) L} and lift any assumptions on & and x's distribution. Further suppose that the neural network implements Eq. (2), (3), (7) at every layer up to depth L, with O and Eq. (3), (7) having nonzero denominators, for all layers, imputs and channels.
Then both y and at at layer 1 are channel-wise normalized if the following conditions hold:
(i) z at layer I - 1 is shase normalized;
(ii) The convolution and normalization steps at layer I do not cause any aggravation of channel- wise collapse and channel imbalance;
(ii)) y' at layer l channel-wise Gaussian and PN's additional parameters 3 at are zero.
Our batch-independent approach: LN+PN or GN+PN. At this point, we crucially note that PN: (i) is batch-independent; (ii) does not cause any alteration of expressivity. This leads us to adopt a batch- independent normalization approach that uses either LN or GN (with few groups) in the normalization step and that replaces the activation step by the proxy-norm activation step (+PN). With such choice of normalization step, we guarantee three of the benefits detailed in Table I: "scale invariance", "control of activation scale" and "preserva- tion of expressivity". With the proxy-normalized activation step, we finally guarantee the fourth benefit of "avoidance of collapse" without compromising any of the benefits provided by the normalization step.
PN-Act
Act
Kny ol + 3
PN
z. a/r/r + st)
x/. E 2:
%
Viu [2]]
Experimental validation. We confirm in Figure 2 that BN's behav- ior is emulated in a fully batch-independent manner with our approach, LN+PN or GN+PN. Indeed, the power plots of networks with LN+PN resemble the power plots of networks with BN. As desired, PN reme- dies LN's failure mode without incurring IN's failure mode.
Figure 3: Plugging PN. PN is plugged into the ac- tivation step Act to yield the normalllae ac- tivation step PN-Act.
Approach strength 1: Normalizing beyond initialization. Our batch-independent approach maintains channel-wise normalization throughout training (cf Figure 2). In contrast, many alternative approaches, either explicitly or implicitly. focus on initialization [21, 24, 17, 18, 49, 20, 72]. Centered Weight Normalization [17]. WS [18, 20, 72] or PreLayerNorm [49] notably rely on the implicit assumption that different channels have the same channel-wise means after the activation function ø. While valid in networks at initialization with g and 7' (cf Appendix A.2.2), this assumption becomes less valid as the affine transformation starts deviating from the identity. We see in Figure 2 that networks with LN+WS are indeed less effective maintaining channel-wise normalization, both in networks with random 3 r (top row) and in networks considered throughout training (bottom row). Such a coarser channel-wise normalization might in turn lead to a less effective use of model capacity and a degradation of the conditioning of the loss landscape [62, 64].
8

---

Approach strength 2: Wide applicability. Our batch- independent approach matches BN with consistency across choices of model architectures, model sizes and activation functions (ef Section 6). Its only restriction, namely that activation steps should be immediately preceded by normalization steps for y' and its associated proxy y to be at the same scale, has easy workarounds. Applicability restrictions might be more serious with alternative approaches: (i) alternative approaches involving normalization in weight space [16, 17, 18, 19, 20] might be ill-suited to architectures with less "dense" kernels such EfficientNets [20]: (ii) approaches involving the tracking of activation statistics [21, 22, 23] might be nontrivial to apply to residual networks [73, 74]; (iii) approaches involving a
change of activation function [24, 5, 15] might precisely restrict the choice of activation function.
Approach strength 3: Ease of implementation. Our approach is straightforward to implement when starting from batch-normalized network. It simply requires: (i) replacing all BN steps with LN or GN steps, and (ii) replacing all activation steps with proxy-normalize activation steps. The mroxy-normalized activation steps themselves are easily implemented (cf Appendix B).
6 Results
We finally assess the practical performance of our batch-independent normalization approach. While we focus on ImageNet [75] in the main text of this paper, we report in Appendix A some additional results on CIFAR [76]. In Appendix A, we also provide all the details on our experimental setup.
Choices of regularization and batch size. As mentioned in Section 2, we perform all our experi- ments with different degrees of regularization to disentangle normalization's 's effects from regulariza- tion's 's effects. We detail all our choices of regularization in Appendices A. and A.3.4.
In terms of batch size, we set: (i) the "global batch size" in between weight updates to the same value independently of the choice of norm; (ii) the "normalization batch size" to a near-optimal value with BN. These choices enable us to be conservative when concluding on potential advantage of our approach over BN at small batch size. Indeed, while the performance of match-independen approaches would remain the same slightly improve at small batch size [77, 65], the performance of BN would eventually degrade due to a regularization that eventually becomes excessive [2, 3, 4, 5, 6, 7].
Effect of adding PN. We start as first experiment by analyzing the effect of adding PN on top of various norms in ResNet-50 (RN50). As visible in Table 2, PN is mostly beneficial when added on top of LN or GN with a small number of groups G. The conse- quence is that the optimal G shifts to lower values in GN+PN compared to GN. This confirms the view that PN's benefit lies in addressing LN's failure mode.
Table 2: Effect of adding PN. ResNet- is trained on ImageNet with BN and LN, GN, GN+WS with G groups, either without or with PN added on top (plain vs. PN). Results are formatted as X Y with X. Y the validation accuracies (%) without and with extra regularization, respectively.
It is also visible in Table 2 that PN does not provide noticeable benefits to BN. This confirms again the view that PN's benefit lies in addressing the problem not present with BN- - of channel-wise collapse. Importantly, since PN does not entail effects other than normalization that could artificially boost the per- formance, PN+ can be compared in a fair way to BN when assessing the effectiveness of normalization.
RN50
G
plain
+PN
BN
76.3 / 75.8
76.2 / 76.0
LN
1
74.5 74.6
75.9 / 76.5
GN
8
75.4
76.3 / 76.7
GN
32
75.4/
75.8 / 76.1
GN+WS
8
76.6
76.8 77.1
This is unlike WS which has been shown to improve BN's performance [18]. In our results of Table 2, the high performance of GN+WS without extra regularization and the fact that PN still provides benefits to GN+WS suggests that: (i) top of its normalization benefits, WS induces form of regularization; (ii) GN+WS is still not fully optimal in terms of normalization.
GN+PN vs. BN. Next, we turn to comparing the performance of our batch- independent approach, GN+PN, to that of BN across broad range of models trained on ImageNet. As visible in Figure and Tables 3, 4, GN+PN outperforms BN in ResNet-50 (RN50) and ResNet- 101 (RN101) [71]. matches BN in Xt-5 (RNX50) and ResNeXt-101 (RNX101 [78], and matches BN in EfficiintNet--00 (EN-BO) and EfficientNet-Be BB (EN-B2), both in the original variant with depthwise convolutions and

---

expansion ratio of 6 [52] and in an approximately parameter- preserving variant (cf Appendix A.1) with group convolutions of group size 16 and expansion ratio of [79]. short, our batch- independent normalization approach, GN+ matches BN not only in behavior but also in performance.
With regard matching BN's performance with alternative norms, various positive results have been reported in ResNets and ResNeXts [8, 7, 12, 13, 15, 20, 36] but only limited number in EfficientNets [15]. In Efficien(Nets, we are notably not aware of any other work showing that BN's performance can be matched with a batch-independent approach. As a confirmation, we assess the performance of various existing batch aatch-independers approaches: GN [3]. GN+WS [18], Evo-SO [15], FRN+TLU [80, 5]. Unlike GN+PN, none of these approaches is found in Table 4 to match BN with consistency.
Normalization and regularization. Our results suggest that while an efficient normalization is not sufficient in itself to achieve good performance on ImageNet, it is still a necessary condition, together with regularization. In results, it is always with extra regularization that GN+PN yields the most benefits. Importantly, the fact that GN+PN consistently leads to large improvements in training accuracy (ef Appendix A.3.2) suggests that additional benefits would be obtained on larger datasets without the requirement of relying on regularization [81, 72].
BN
Table 3:
BN vs. GN, GN+PN. ResNets and ResNeXts are trained on ImageNet with BN and GN, GN+PN. Results are formatted as in Table 2.
BN w/ extra regul
CN
CN w/ extra regul
GN+PN
GN+PN w/ extra regul
RNS0
RN101
RNX50
RNX101
76
BN
7
78.0
77.6/77.2
78.7 78,9
76
75.4/75.3
77.0
77.4/ 76.2/
GN+PN
76.3
78.6
7
77.7//99.0
766
RNN60
RNN3O
FX--TO
70
Table BN vs. batch-independent approaches. Efficient-
Nets are trained on ImageNet with BN and various batch-
independent approaches. Results are formatted as in Table 2.
RN101
HNNIO1
E2--12
depthwise convs [52]
group convs [79]
Figure 4: BN vs. GN, GN+PN. Validation accuracies (%) of ResNets, ResNeXts and Efficient- Nets trained on ImageNet with BN and GN, GN+PN, without and with extra regularization. Efficient- Nets are considered in the variant with group convolutions [79].
EN-BO
EN-B2
EN-BO
EN-B2
BN
76.9 77.2
79.4/ / 80.0
76.8 / 76.7
79.5 / 79.7
GN
76.2 76.2
78.9 79.4
76.2/76..
79.0 79.6
GN+PN
76.8/77.0
79.3 80.0
76.7/76.3
79.3 80.1
Bvo-S0
75.8/75.8
78.9/79.6 7 78.7
GN+WS
74.2/74.1
77.8/778
76.2
79.2
FRN+TLU
77.7/
78.4/78.8
78.2/ 7
Summary and broader impact
We have introduced novel framework to finely characterize the various neural network properties affected by the choice of normalization. Using this framework, we have shown that while BN's beneficial properties are not retained when solely using the prototypical batch-independer norms, they are retained when combining some of these norms with the technique hereby introduced of Proxy Normalization. We have demonstrated on extensive set of experiments that our batch-independent normalization approach consistently matches BN in both behavior and performance.
The main implications of this work could stem from the unlocked possibility to retain BN's normal- ization benefits while removing batch dependence. Firstly, our approach could be used to retain BN's normalization benefits while alleviating the burden of large activation memory stemming from BN's requirement of sufficiently large batch sizes. This is expected to be important in memory-intensin applications such as object detection or image segmentation, but also when using A.I. accelerators that leverage local memory to provide extra acceleration and energy savings in exchange for tighter mem- ory constraints. Secondly, our approach could be used to retain BN's s normalization benefits while avoiding BN's regularization when the latter is detrimental. As discussed in Section 6, this expected to be important in the context - that will likely be prevalent in the future - of large datasets.
10

---

Acknowledgments and Disclosure of Funding
We are thankful to Simon Knowles, Luke Hudlass-Galley, Luke Prince, Alexandros Koliousis, Anastasia Dietrich and the wider research team at Graphcore for the useful discussions and feedbacks. We are also thankful to the anonymous reviewers for their insightful comments that helped improve the paper.
References
[1] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In 32nd International Conference on Machine Learning, ICML 2015, pages
2015.
[2] Sergey Ioffe. Batch renormalization: Towards reducing minibatch dependence in batch- normalized models. In Advances Neural Information Processing Systems 30: Annual Conference Neural Information Processing Systems 2017, pages 1945-1953, 2017.
[3] Yuxin Wu and Kaiming He. Group normalization. Computer Vision ECCV 2018 15th European Conference, Proceedings, Part XIII, pages 3-19, 2018.
[4] Chris Ying. Sameer Kumar, Dehao Chen, Tao Wang, and Youlong Cheng. Image classification at supercomputer scale. CoRR, abs/1811 1.0699,, 2018
[5] Saurabh Singh and Shankar Krishnan. Filter response normalization layer: Eliminating batch dependence in the training of deep neural networks. In 2020 /EFE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2020, pages 1234-11243, 2020.
[6] Cecilia Summers and Michael J. Dinneen. Four things everyone should know to improve batch normalization. In 8th International Conference on Learning Representations, ICLR 2020, 2020.
[7] Junjie Yan, Ruosi Wan, Xiangyu Zhang, Wei Zhang, Yichen Wei, and Jian Sun. Towards stabilizing batch statistics in backward propagation of batch normalization. In 8th International Conference Learning Representations, ICLR 2020, 2020.
[8] Vitaliy Chiley, Ilya Sharapov, Atli Kosson, Urs Köster, Ryan Reece, Sofia Samaniego de la Fuente, Vishal Subbiah, and Michael James. Online normalization for training neural networks. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, pages 8431-8441 2019.
2019.
[9] Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. CoRR, Abs/1607.064550 2016.
[10] Dmitry Ulyanov, Andrea Vedaldi, and Victor S. Lempitsky, Instance normalization: The missing ingredient for fast stylization. CoRR, abs/1607.08022 2016.
[11] Mengye Ren, Renjie Liao, Raquel Urtasun, Fabian H. Sinz, and Richard S. Zemel. Normalizing the normalizers: Comparing and extending network normalization schemes. In 5th International Conference on Learning Representations, ICLR 2017, Conference Track Proceedings, 2017.
[12] Ping Luo, Jiamin Ren, Zhanglin Peng, Ruimao Zhang, and Jingyu Li. Differentiable learning- to- to-normalize via switchable normalization. In 7th International Conference on Learning Representations, ICLR 2019, 2019.
[13] Ping Luo, Zhanglin Peng, Wengi Shao, Ruimao Zhang, Jiamin Ren, and Lingyun Wu. Differ- entiable dynamic normalization for learning deep representation. In Proceedings of the 36th International Conference on Machine Learning, ICML. 2019, volume 97 of Proceedings of Machine Learning Research, pages 9203-4211 2019.
[14] Biao Zhang and Rico Sennrich. Root mean square layer normalization. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, pages 12360-12371, 2019.
11

---

[15] Hanxiao Liu, Andy Brock, Karen Simonyan, and Quoe Le. Evolving normalization activvation layers. In Advances in Neural Information Processing Systems 33: Anmual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, 2020.
[16] Tim Salimans and Diederik P. Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, page 901, 2016.
[17] Lei Huang, Xianglong Liu, Yang Liu, Bo Lang, and Dacheng Tao. Centered weight normal- ization in accelerating training of deep neural networks. In IEEE International Conference on Computer Vision, ICCV 2017, pages 2822-28300 2017.
[18] Siyuan Qiao, Huiyu Wang. Chenxi Liu, Wei Shen, and Alan L. Yuille. Weight standardization. CoRR, abs/1903. 10520,, 2019.
[19] Brendan Ruff, Taylor Beek, and Joscha Bach. Mean shift rejection: Training deep neural networks without minibatch statistics or normalization. CoRR, abs/1911.13173, 2019.
[20] Andrew Brock, Soham De, and Samuel L. Smith. Characterizing signal propagation to close the performance gap in unnormalized ResNets. In 9th International Conference on Learning Representations, ICLR 2021, 2021.
[21] Devansh Arpit, Yingbo Zhou, Bhargava Urala Kota, and Venu Govindaraju. Normalization propagation: A parametric technique for removing internal covariate shift in deep networks. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, volume 48 of JMLR Workshop and Conference Proceedings. pages 1168-1176, 2016.
[22] César Laurent, Nicolas Ballas, and Pascal Vincent. Recurrent normalization propagation. In 5th International Conference on Learning Representations, ICLR 2017, Workshop Track Proceedings, 2017.
[23] Alexander Shekhovtsov and Boris Flach. Normalization of neural networks using analytic variance propagation. CoRR, N0/11833.1055 2018.
[24] Günter Klambauer, Thomas Unterthiner. Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. In Advances in Neural Information Processing Systems 30: Anmual Conference on Neural Information Processing Systems 2017, pages 971-980, 2017.
[25] Twan van Laarhoven. abs/1706.05350, 2017.
L2 regularization versus batch and weight normalization. CoRR,
[26] Sanjeev Arora, Zhiyuan Li, and Kaifeng Lyu. Theoretical analysis of auto rate-tuning by batch normalization. In 7th International Conference on Learning Representations, ICLR 20/9, 2019.
[27] Elad Hoffer, Ron Banner, Itay Golan, and Daniel Soudry. Norm matters: efficient and accurate normalization schemes in deep networks. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, pages 2164-2174, 2018.
[28] Zhiyuan Li and Sanjeev Arora. An exponential learning rate schedule for deep learning. In 8th International Conference on Learning Representations, ICLR 2020, 2020.
[29] Guodong Zhang, Chaoqi Wang, Bowen Xu, and Roger B. Grosse. Three mechanisms of weight decay regularization. In 7th International Conference on Learning Represenaations, ICLR 2019, 2019.
[30] Ruosi Wan, Zhanxing Zhu, Xiangyu Zhang, and Jian Sun. Spherical motion dynamics of deep neural networks with batch normalization and weight decay. CoRR, 0bs/2006.08419, 2020.
[31] Minhyung Cho and Jaehyung Lee. Riemannian approach to batch normalization. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pages 5225-5235, 2017.
12

---

[32] Jonas Moritz Kohler, Hadi Daneshmand, Aurélien Lucchi, Thomas Hofmann, Ming Zhou, and Klaus Neymeyr. Exponential convergence rates for batch normalization: The power of decoupling in non-convex optimization. In The 22nd International Conference on Artificial Intelligence and Statistics, AISTATS 2019, volume 89 of Proceedings of Machine Learning Research, pages 806-815, 2019.
[33] Boris Hanin and David Rolnick. How to start training: The effect of initialization and architec- ture. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, pages 569-579, 2018.
[34] Hongyi Zhang, Yann N. Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization. In 7th International Conference on Learning Representations, ICLR 2019, 2019.
[35] Soham De and Samuel L. Smith. Batch normalization biases residual blocks towards the identity function in deep networks. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, 2020.
[36] Jie Shao, Kai Hu, Changhu Wang, Xiangyang Xue, and Bhiksha Raj. Is normalization indis- pensable for training deep neural network? In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, 2020.
[37] Ekdeep Singh Lubana, Robert P. Dick, and Hidenori Tanaka. Beyond batchnorm: Towards a general understanding of normalization in deep learning. CoRR, 0as/21066059566 2021.
[38] Andreas Veit, Michael J. Wilber, and Serge J. Belongie. Residual networks behave like ensembles of relatively shallow networks. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, pages 550- 558, 2016.
[39] George Philipp, Dawn Song, and Jaime G. Carbonell. Gradients explode deep networks are shallow ResNet explained. In 6th International Conference on Learning Representations, ICLR 2018, Workshop Track Proceedings, 2018.
[40] Greg Yang, Jeffrey Pennington, Vinay Rao, Jascha Sohl-Dickstein, and Samuel S. Schoenholz. A mean field theory of batch normalization. In 7th International Conference on Learning Representations, ICLR 2019, 2019.
[41] Antoine Labatie. Characterizing well-behaved vs. pathological deep neural networks. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, volume 97 of Proceedings of Machine Learning Research, pages 361 1-3621,,, 2019.
[42] Angus Galloway, Anna Golubeva, Thomas Tanay, Medhat Moussa, and Graham W. Taylor. Batch normalization a cause of adversarial vulnerability. CoRR, abs/1905.022 000111. 2019.
[43] Mattias Teye, Hossein Azizpour, and Kevin Smith. Bayesian uncertainty estimation for batch normalized deep networks. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, volume 80 of Proceedings of Machine Learning Research, pages 4914- 4923, 2018
[44] Alexander Shekhovtsov and Boris Flach. Stochastic normalizations as Bayesian learning. In Computer Vision ACCV 2018 14th Asian Conference on Computer Vision, Revised Selected Papers, Part II, volume 11 11362 of Lecture Notes in Computer Science, pages 463-479, 2018.
[45] Ping Luo, Xinjiang Wang, Wenqi Shao, and Zhanglin Peng. Towards understanding regular- ization in batch normalization. In 7th International Conference on Learning Representations, ICLR 2019, 2019.
[46] Johan Bjorck, Carla P. Gomes, Bart Selman, and Kilian Q Weinberger: Understanding batch normalization. Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, pages 7705-7716, 2018
13

---

[47] Arthur Jacot, Franck Gabriel, and Clément Hongler. Freeze and chaos for DNNs: an NTK view of batch normalization, checkerboard and boundary effects. CoRR, abs//997.0057/ 2019.
[48] Hadi Daneshmand, Jonas Moritz Kohler, Francis R. Bach, Thomas Hofmann, and Aurélien Lucchi. Batch normalization provably avoids ranks collapse for randomly initialised deep networks. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, 2020.
[49] Vinay Rao and Jascha Sohl-Dickstein. batch norm unique? An empirical investigation and prescription to emulate the best properties of common normalizers without batch dependence. CoRR, abs/2010. 10687, 2020.
[50] Boris Hanin and David Rolnick. Complexity of linear regions in deep networks. In Proceed- ings of the 36th International Conference on Machine Learning, ICML 2019, volume 97 of Proceedings of Machine Learning Research, pages 2596-2604 2019.
[51] Boris Hanin and David Rolnick. Deep relu networks have surprisingly few activation patterns. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, pages 359-368, 2019.
[52] Mingxing Tan and Quoe V. Le. EfficientNet: Rethinking model scaling for convolutional neural networks. In Proceedings of the 36th International Conference Machine Learning, ICML 2019, volume 97 of Proceedings of Machine Learning Research, pages 6105-6114, 2019.
[53] Lei Huang, Yi Zhou, Li Liu, Fan Zhu, and Ling Shao, Group whitening: Balancing learning efficiency and representational capacity. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, pages 2512-9521, 2021.
[54] Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang. Alan L. Yuille, and Quoc V. Le. Adversarial examples improve image recognition. In 2020 JEEE/CVF Conference on Computer Vision and Putlern Recognition, CVPR 2020, pages 816-825, 2020.
[55] Cihang Xie and Alan L. Yuille. Intriguing properties of adversarial training at scale. In 8th International Conference on Learning Representations, ICLR 2020, 2020.
[56] Takeru Miyato and Masanori Koyama. cGANs with projection discriminator. In 6th Interna- tional Conference on Learning Representations, ICLR 2018, Conference Track Proceedings, 2018.
[57] Harm de Vries, Florian Strub, Jérémic Mary, Hugo Larochelle, Olivier Pietquin, and Aaron C. Courville. Modulating early visual processing by language. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pages 6594-6603 2017.
[58] Lucas Deecke, lain Murray, and Hakan Bilen. Mode normalization. In 7th International Conference Learning Representations, ICLR 2019, 2019.
[59] Ximei Wang, Ying Jin, Mingsheng Long, Jianmin Wang, and Michael I. Jordan. Transferable normalization: Towards improving transferability of deep neural networks. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, pages 1951 2019.
[60] Woong- Gi Chang, Tackgeun You, Seonguk Sco, Suha Kwak, and Bohyung Han. Domain- specific batch normalization for unsupervised domain adaptation. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, pages 7354-7362, 2019.
[61] Yuxin Wu and Justin Johnson. Rethinking "batch" in batchnorm. CoRR, abs/2105.07576 2021.
[62] David Page. How to train your ResNet 7: Batch norm, 2019.
[63] Jeffrey Pennington and Pratik Worah. Nonlinear random matrix theory for deep learning. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pages 2637-2646, 2017.
14

---

[64] Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net opti- mization via Hessian eigenvalue density. In Proceedings of the 36th International Conference on Machine Learning, ICMI. 2019, volume 97 of Proceedings of Machine Learning Research, pages 2232-2241, 2019.
[65] Dominic Masters and Carlo Luschi. Revisiting small batch training for deep neural networks CoRR, abs/1804.00 804.07612, 2018.
[66] Dmitry Ulyanov, Vadim Lebedev, Andrea Vedaldi, and Victor S. Lempitsky. Texture networks: Feed-forward synthesis of textures and stylized images. In Proceedings of the 33nd International Conference Machine Learning, ICML 2016, volume 48 of JMLR Workshop and Conference Proceedings, pages 1349-1357, 2016.
[67] Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. Image style transfer using convolu- tional neural networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, pages 2414-2422 2016.
[68] Dmitry Ulyanov, Andrea Vedaldi, and Victor S. Lempitsky. Improved texture networks: Maxi- mizing quality and diversity in feed-forward stylization and texture synthesis. In 2017 IEEE Conference on Computer Vision and Pattem Recognition, CVPR 2017, pages 4105 4113, 2017.
[69] Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. A learned representation for artistic style. In 5th International Conference on Learning Representations, ICLR 2017, Conference Track Proceedings, 2017.
[70] Xun Huang and Serge J. Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In IEEE International Conference on Computer Vision, ICCV 2017, pages 1510-1519, 2017.
[71] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Computer Vision ECCV 2016 14th European Conference, 2016.
[72] Andy Brock, Soham De, Samuel L. Smith, and Karen Simonyan. High-performance large-scale image recognition without normalization. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, volume 139 of Proceedings of Machine Learning Research, pages 1059-1071, 2021.
[73] Wenling Shang, Justin Chiu, and Kihyuk Sohn. Exploring normalization in deep residual networks with concatenated rectified linear units. In Proceedings of the Thirty- First AAAI Conference on Artificial Intelligence, pages 1509-1516, 2017.
[74] Igor Gitman and Boris Ginsburg. Comparison of batch normalization and weight normalization algorithms for the large-scale image classification. CoRR, abs/1 htss/109900145,, 2017.
[75] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hicrarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), pages 248-255, 2009.
[76] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
[77] Priya Goyal, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: training ImageNet in 1 hour. CoRR, abs/ 7bb//7766.0677, 2017.
[78] Saining Xie, Ross B. Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, 2017.
[79] Dominic Masters, Antoine Labatie, Zach Paton-Rosen, and Carlo Luschi. Making EfficientNet more efficient: Exploring batch-independen normalization, group convolutions and reduced resolution training. CoRR, abs/2106.03640. 2021.
[80] Sitao Xiang and Hao Li. On the effect of batch normalization and weight normalization in generative adversarial networks. CoRR, abs/1704,03971,, 2017.
15

---

[81] Alexander Kolesnikov. Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big Transfer (BiT): General visual representation learning. In Computer Vision ECCV 2020 16th European Conference, 2020.
[82] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna Rethinking the inception architecture for computer vision. In 2016 IEEE Conference on Computer Vision and Patlern Recognition, CVPR 2016, pages 2818-2826, 2016.
[83] Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdi- nov. Dropout: simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929-15 19119999998,,. 2014.
[84] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinherger. Deep networks with stochastic depth. Computer Vision ECCV 2016 14th European Conference, Proceedings, Part IV, volume 9908, pages 646-661,2011
[85] Hongyi Zhang. Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In 6th International Conference on Learning Representations, ICLR 2018, Conference Track Proceedings, 2018.
[86] Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Seong Joon Oh, Youngjoon Yoo, and Junsuk Choe. Cutmix: Regularization strategy to train strong classifiers with localizable features. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, pages 6022-6031, 2019.
[87] Ekin Cubuk, Barret Dandelion Mané, Vijay Vasudevan, and Quoc V. Le. AutoAugment: Learning augmentation strategies from data. In IEEE Conference Computer Vision and Puttern Recognition, CVPR 2019, pages 113-123, 2019.
[88] Jasmine Collins, Johannes Ballé, and Jonathon Shlens. Accelerating training of deep neural networks with standardization loss. CoRR, abs/1903.00025,, 2019.
[89] Yann Dauphin and Ekin Dogus Cubuk. Deconstructing the regularization of BatchNorm. In 9th International Conference on Learning Representations, ICLR 2021, 2021.
[90] Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch normalization help optimization? In Advances in Neural Information Processing Systems 31: Annual Conference on Neurul Information Processing Systems 2018, NeurIPS 2018, pages 2018.
[91] Moshe Leshno, Vladimir Ya. Lin, Allan Pinkus, and Shimon Schocken. Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural Networks, 6(6):611-867, 1993.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the pa- per's contributions and scope? [Yes] We believe the paper's 's title and abstract are consistent with the theoretical results of Sections 4, and the experimental results of Sections 4, 5, 6.
(b) Did you describe the limitations of your work? [Yes] The scope and assumptions of our theoretical results are provided in Sections 3, 4, 5. The minor limitation of our approach stemming from the requirement that each activation step should be directly preceded by normalization step is stated in Section 5.
(c) Did you discuss any potential negative socictal impacts of your work? [No] Our work directly impacts the advancement of the field of computer vision. Its negative impacts could come only indirectly from the negative impacts of this field in general. We thought that this observation was not needed in the main text, is already implied.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
16

---

2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] The full assumptions of our theoretical results are provided in Sections 3, 4,
(b) Did you include complete proofs of all theoretical results? [Yes] Complete proofs of all our theoretical results are provided in Appendices C, D, E.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] The experimental results can be reproduced based on Appendices A, B.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] All the experimental details are reported in Appendix A.
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes] lo intervals associated with all our experiments are reported in Appendix A.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] Our experiments with batch- independent norms were run on Graphcore's ' MK1 and MK2 IPUs. We refer the reader [79] for throughput numbers on Graphcore's 's MK2 IPUs.
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No] ImageNet's images owned by copyright holders. We are unsure about the existence, and potentially the type, of
licenses of CIFAR-10 and CIFAR- 100.
(c) Did you include any new assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
17


Reviews:
[{"review_id": "yVZmwC-Ftm", "reviewer": "Reviewer_AgAg", "summary": "This work proposes some theoretical analyses on different normalization schemes in deep neural networks. It found that LN suffers from the collapsing effect over the number of layers, which makes the computation more and more linear, whereas IN suffers from a lack of expressivity. Towards fixing these issues, the paper proposed ProxyNorm that adds another normalization layer after the activation function. Empirical results suggest that the combination of ProxyNorm and LayerNorm or GroupNorm can lead to matching or sometimes superior performance compared to BatchNorm.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "------------------------------------------\n### Strengths\n- Insightful analysis on different failure modes of LN and IN.\n- Proposed a working method that shows empirical improvement over standard LN and GN, with a matching performance to BN. This is a very exciting result.\n- Overall, the experiment numbers seem solid.\n- The experimental details and implementation details are included in the appendix.\n\n------------------------------------------\n### Weaknesses\n- **More empirical studies on the collapsing phenomenon in LN and GN.** Currently the plot in Figure 3 is not very clear about the collapsing effect. $\\mathcal{P}^{(1)}$ does grow, but it does not converge to 1, and the rest of the terms do not converge to 0, unlike suggested by the theorem. So I am wondering if the paper could improve the empirical studies to verify the collapsing effect. First, you don’t have to use a ResNet-50, but it could be a regular convolutional network with many more layers. Second, I would also like to see other metrics to measure the linearity of the layers empirically over the number of layers. E.g. the proportion of negative entries in the pre-activation.\n- **Analysis depends on the use of ReLU activations:** Line 111 states that the activation needs to be positive homogeneous and non-zero, and based on this it is clear that we only have scaled or flipped ReLU type activations. There have also been other developments of activation functions such as SeLU that have self-normalizing effects. I wonder if this can be baked into the analysis and lead to other insights? Instead of adding another normalization layer after the activation function.\n- **Reliance to be preceded by another normalization function:** First of all, I don’t quite understand the requirement stated in Line 257-258: “The only required constraint for PN to work optimally is that each activation step should be immediately preceded by a normalization step.” I expect to see more concrete justification for this requirement and the transition of going into LN+PN and GN+PN is too fast. Secondly, doing a double normalization makes the computation less elegant. The work would have more impact by just introducing one single normalization step.\n- **Guarantees on LN+PN/GN+PN:** I might have entirely missed the point but I would appreciate it if there is a theorem or remark showing that LN+PN / GN+PN is able to avoid the collapsing effect.\n\n------------------------------------------\n### Clarification\n- **Computing statistics analytically:** If we assume that we have a ReLU activation, can we analytically compute the mean and variance of the proxy distribution? It can simplify the modification to standard networks a lot, by simply introducing pre-computed affine transformations instead of collecting statistics from samples.\n- **The reason for a proxy distribution:** It also was not clear to me what is the purpose for adding a proxy distribution. Why not just normalize the post-activations using the statistics from themselves?\n\n------------------------------------------\n### Conclusion\nIn conclusion, I found this is a nice paper for advancing our understanding of neural network normalizations. However, I found the proposed solution somehow less elegant, and there is some disconnect between the theoretical and the experimental sections. Some practical design choices are not well explained (e.g. the need for applying together with an existing normalization layer), and a lack of theoretical guarantees on the proposed LN+PN and GN+PN. Overall, I think the paper is above the bar for acceptance.\n\nUpdate: The review addresses most of my concerns and I have raised the score from 6 to 7.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "ZQAiBpdIFD", "reviewer": "Reviewer_Wjpi", "summary": "This paper studies normalization strategies for deep residual networks, both theoretically and empirically, and finds the strategies that don't rely on batch-statistics to have particular failure modes, the two most notable ones being channel-wise collapse, where instances collapse to a constant in each channel, and reduction in instance-variability, where the first and second order statistics per channel are identical for each instance, thereby reducing expressivity and the ability to recognize higher order concepts. The paper then proceeds to propose proxy-normalization, a normalization method suitable for small-batch learning (i.e., not relying on batch-statistics) that, when combined with other normalization strategies, such as Layer Norm or Group Norm, alleviates these failure modes. On ImageNet and CIFAR experiments with deep residual networks (RN50 and larger), the work finds that Proxy-Normalization (with sufficient additional regularization such as label-smoothing, dropout, and stochastic depth), can be used to match the performance of BN without relying on batch statistics. The proposed method standardizes the activations of the network according to the mean and variance of a channel-wise gaussian random variable, with learnable mean and variance parameters.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Originality & Significance\n- Improving residual normalization strategies to have no dependence on batch statistics is an important problem with great practical applicability. A broad body of work has been dedicated to this problem.\n- The proposed Proxy-Normalization strategy looks novel to the best of my knowledge, but is somewhat related to existing normalization strategies based on proxy-statistics of the activations [1,2,3].\n- The power decomposition is quite intriguing, and provides a nice grounding for subsequent arguments about normalization strategies and failure modes. Since such power quantities can be measured, it also makes it easy to directly verify these arguments with empirical observations, as the authors do in Figure 3. Their logic is as follows: Power concentration in P1 (at the dataset scale) means all instances have collapsed in that channel (equal to a constant). Collapsing channels result in a pseudo-linearity failure, whereby the network does not effectively utilize its depth. Lack of power concentration in P2 and P4 imply that all instance activations have the same exact mean and variance (no inter-instance variability). Lack of inter-instance variability (low power in P2 and P4) reduces expressivity of the network, making it more difficult to recognize higher order concepts.\n\n### Clarity\nThe paper is well written and relatively clear, however some details are unclear.\n- For example, from my understanding C_k in theorem 1 is the number of channels at k. Can you please clarify this point?\n- The arguments relating inter-instance variability to expressivity is also not entirely clear and is a little hand-wavy.\n\n### Quality\nThere are sufficient technical contributions in the paper, both empirically and theoretically. However, there are limitations\n- The proposed Proxy Normalization method while conceptually simple, introduces some non-trivial complexity into training, in having to sample from a Gaussian random variable with learnable parameters, and then feed that into the activation before standardizing.\n- The proposed proxy-normalization is demonstrated to address the channel collapse issue (large P1), but is not demonstrated to address the inter-instance variability issue. The concentration of power in the dataset scale (channel collapse) is more much more stark with LN and GN, but the improvement in inter-instance variability (expressivity), to be determined by P2+P4 moving away from 0, is not as significant in Figure 3, but I suppose that according to the provided arguments LN and GN should not really have this issue.\n- The proposed method requires additional regularization factors to match BN performance, and from my parsing of the appendix, these factors seem to vary from one architecture to another. Furthermore, it is unclear which regularization strategies are necessary or why. Such an ablation and detailed understanding would significantly improve the paper.\n\n\n[1] Arpit et al., Normalization propagation: A parametric technique for removing internal covariate shift in deep networks, 2016.\n\n[2] Laurent et al., Recurrent normalization propagation, 2017.\n\n[3] Shekhovtsov et al., Normalization of neural networks using analytic variance propagation, 2018.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No clear negative societal impacts.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "Scdrj2YU8gL", "reviewer": "Reviewer_89jK", "summary": "This paper analyzes the properties of normalization methods. It shows how layer normalization induces a collapse towards channel-wise constant functions with assumptions, and how the constraint imposed by normalization affects the performance of instance normalization. It also proposes Proxy Normalization that emulates batch normalization’s behavior and performance, when combined with batch-independent normalization (e.g, layer normalization or group normalization) .", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Pros:**\n\n+This paper summarizes many properties of normalization methods, even though these properties of normalization have been separately shown in previous works.   \n+The analysis shown in section 3 is novel, even though I have concerns on the claims.  \n+The experiments of this paper are comprehensive.  \n\n\n**Cons:**\n\n-It is not rigorous to state that BN preserve the expressivity. Particularly, there are work [1*] showing that the batch size affects the layer representation/expressivity of BN, and further affects the model’s representation/expressivity. For example, using a batch size of 2 for a MLP, it is clear that activation is constraint to be 1 and -1 during training. Therefore, BN could change the expressivity, noting that the optimization of BN is over the mini-batch statistics, not the population statistics during training. \n\n-I do not recognize the practicality of Proxy Normalization:  \n (1) This paper should conduct experiments on small batch size scenario (e.g., batch size of 2 on ImageNet or object detection tasks on CoCo) to support its claims in Line-306 and 309. Based on the current experimental results, I do not find why one person should use PN+GN/LN, rather than BN, considering that GN/LN introduces extra computational cost during inference.   \n(2) There are two extra hyper-parameters $\\tilde{\\beta}_c^l$ and  $\\tilde{\\gamma}_c^l$, How to determine them? Do the results sensitive to these hyper-parameters? This paper should conduct experiments to discuss the sensitivity of these two hyper-parameters over different architecture/dataset. \n\n\n-Some description and notation is not rigorous:   \n(1) Line 140, why  $\\tilde{y}^l$ is unlikely to have its channel-wise distributions collapsed?, if the $\\beta$ is biased to a very large negative value, all the normalized output will less than 0, leading to all 0 output.   \n(2) Line 117~118, what is $E_\\mathbf{x}$, a, does this mean the expectation over the mini-batch dimension? If yes, why denote $\\mathbf{x} \\in R^{H \\times W \\times C_l}$? \n\n\n**Ref:**\n\n[1*] Group Whitening: Balancing Learning Efficiency and Representational Capacity, arxiv: 2009.13333\n\n\n\n**Update**: I have read the authors responses and other reviewer’s comments, The authors mostly addressed my concerns relating to the clarity. I still has the concerns on the practicality, considering that LN/GN is required (For their scale invariant to stabilize training) in the solution, while LN/GN introduces extra computational cost during inference. \nI do recognize the analysis shown in section 3, which is novel to me. I thus keep positive to this paper and keep my score as 6.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"ethical_concerns": "no", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a theoretical analysis on different normalization schemes. It demonstrates that normalizing schemes which don't rely on batch-statistics suffer from channel collapse or lack or expressivity.  It then proposes a novel normalization scheme to address those issues.

Although the practicality of the algorithm could be improved, proxy-norm is able to match BN while not using the batch statistics. Additionally, the analysis presented in the paper is novel and reviewers agreed that it would be of value to the community.

Decision: Accept (Poster)