Characterizing signal propagation to close the performance gap in unnormalized ResNets
Authors: Andrew Brock
Soham De
Samuel L Smith
Venue: ICLR 2021 Poster (2021)
Keywords: normalizers
signal propagation
deep learning
neural networks
ResNets
EfficientNets
ImageNet
CNNs
ConvNets

Abstract: Batch Normalization is a key component in almost all state-of-the-art image classifiers, but it also introduces practical challenges: it breaks the independence between training examples within a batch, can incur compute and memory overhead, and often results in unexpected bugs. Building on recent theoretical analyses of deep ResNets at initialization, we propose a simple set of analysis tools to characterize signal propagation on the forward pass, and leverage these tools to design highly performant ResNets without activation normalization layers.  Crucial to our success is an adapted version of the recently proposed Weight Standardization.  Our analysis tools show how this technique preserves the signal in ReLU networks by ensuring that the per-channel activation means do not grow with depth. Across a range of FLOP budgets, our networks attain performance competitive with state-of-the-art EfficientNets on ImageNet.

Published as a conference paper at ICLR 2021
CHARACTERIZING SIGNAL PROPAGATION TO CLOSE THE PERFORMANCE GAP IN UNNORMALIZED RESNETS
Andrew Brock, Soham De & Samuel L. Smith
Deepmind
(a jarocc,, sohamde, slsmi t. begoogl.o...oooo
ABSTRACT
Batch Normalization is key component in almost all state-of-the-art image clas- sifiers, but it also introduces practical challenges: it breaks the independence be- tween training examples within batch, can incur compute and memory overhead, and often results in unexpected bugs. Building on recent theoretical analyses of deep ResNets at initialization, we propose simple set of analysis tools to charac- terize signal propagation on the forward pass, and leverage these tools to design highly performant ResNets without activation normalization layers. Crucial to our success is an adapted version of the recently proposed Weight Standardization. Our analysis tools show how this technique preserves the signal in networks with ReLU or Swish activation functions by ensuring that the per-channel activation means do not grow with depth. Across a range of FLOP budgets, our networks at- tain performance competitive with the state-of-the-art EfficientNets on ImageNet. Our code is available at http::://// :/ /tpp//ppdd.////////////////. ai/ //nfnets
INTRODUCTION
BatchNorm has become core computational primitive in deep learning (loffe & Szegedy 2015).
and it is used in almost all state-of-the-ar image classifiers (Tan Le 2019] Wei et al. 2020). A
number of different benefits of BatchNorm have been identified. It smoothens the loss landscape (Santurkar et al. 2018), which allows training with larger learning rates (Bjorck et al. 2018), and
the noise arising from the minibatch estimates of the batch statistics introduces implicit regular- ization (Luo et al. 2019). Crucially, recent theoretical work (Balduzzi et al.) 2017; De & Smith 2020) has demonstrated that BatchNorm ensures good signal propagation at initialization in deep residual networks with identity skip connections (He et al.) 2016bla). and this benefit has enabled practitioners train deep ResNets with hundreds or even thousands of layers (Zhang et 2019).
However, BatchNorm also has many disadvantages. Its behavior is strongly dependent on the batch size, performing poorly when the per device batch size is too small or too large (Hoffer al. 2017), and it introduces discrepancy between the behaviour of the model during training and at inference time. BatchNorm also adds memory overhead (Rota Buld al.. 2018). and is a common source of implementation errors (Pham et al., 2019). In addition, it is often difficult to replicate batch normalized models trained on different hardware. A number of alternative normalization layers have been proposed (Ba et al.) 20166) Wu & He 2018), but typically these alternatives generalize poorly or introduce their own drawbacks, such as added compute costs at inference.
Another line of work has sought to eliminate layers which normalize hidden activations entirely. A common trend is to initialize residual branches output zeros (Goyal et al. 2017 /Zhang et al. 2019)
De & Smith 2020 Bachlechner et al. 2020), which ensures that the signal is dominated by the skip path early in training. However while this strategy enables us to train deep ResNets with thousands of layers, it still degrades generalization when compared to well-tuned baselines (De & Smith 2020).
These simple initialization strategies are also not applicable to more complicated architectures like EfficientNets (Tan & Le 2019, the current state of the art ImageNet (Russakovsky et al. 2015).
This work seeks to establish general recipe for training deep ResNets without normalization layers, which achieve test accuracy competitive with the state of the art. Our contributions are as follows:

---

Published as a conference paper at ICLR 2021
We introduce Signal Propagation Plots (SPPs): simple set of visualizations which help us inspect signal propagation at initialization on the forward pass in deep residual net- works. Leveraging these SPPs, show how to design unnormalized ResNets which are constrained to have signal propagation properties similar to batch-normalized ResNets.
We identify a key failure mode in unnormalized ResNets with ReI.U or Swish activations and Gaussian weights. Because the mean output of these non-linearities is positive, the squared mean of the hidden activations on each channel grows rapidly as the network depth increases. To resolve this, we propose Scaled Weight Standardization, minor modification of the recently proposed Weight Standardization (Qiao et al. 2019 Huang et al. 2017b), which prevents the growth in the mean signal, leading to a substantial boost performance.
We apply our normalization- free network structure conjunction with Scaled Weight Stan- dardization to ResNets on ImageNet, where we for the first time attain performance which is comparable or better than batch-normalized ResNets on networks as deep as 288 layers.
Finally, we apply our normalization-fret approach to the RegNet architecture (Radosavovic et al.] 2020). By combining this architecture with the compound scaling strategy proposed by Tan & Le (2019), we develop a class of models without normalization layers which are competitive with the current ImageNet state of the art across range of FLOP budgets.
BACKGROUND
Deep ResNets initialization: The combination of BatchNorm (loffe & Szegedy 2015 and skip connections (Srivastava et al.) 2015 He et al.) (01 has allowed practitioners to train deep ResNets with hundreds or thousands of layers. To understand this effect, a number of papers have analyzed signal propagation in normalized ResNets at initialization (Balduzzi et al., 2017;) Yang al. 2019). In a recent work, De & Smith (2020) showed that in normalized ResNets with Gaussian initializa- tion, the activations on the 8 residual branch are suppressed by factor of O(VE), relative to the scale of the activations on the skip path. This biases the residual blocks in deep ResNets towards the identity function at initialization, ensuring well-behaved gradients. In unnormalized networks, one preserve this benefit by introducing learnable scalar at the end of each residual branch, initialized to zero (Zhang et al. 2019 De & Smith 2020 [achlechner et al.) 2020). This simple change is sufficient to train deep ResNets with thousands of layers without normalization. However, while this method is easy to implement and achieves excellent convergence on the training set, it still achieves lower test accuracies than normalized networks when compared well-tuned basclines.
These insights from studies of Batch-normalized ResNets are also supported by theoretical analyses of unnormalized networks (Taki) 2017 Yang & Schoenholz 2017 Hanin & Rolnick 2018 Q al.)
2020). These works suggest that, in ResNets with identity skip connections, if the signal does not explode on the forward pass, the gradients will neither explode nor vanish on the backward pass. Hanin & Rolnick (2018) conclude that multiplying the hidden activations on the residual branch by a factor of O(172) or less, where d denotes the network depth, sufficient to ensure trainability at initialization.
Alternate normalizers: To counteract the limitations of BatchNorm in different situations, a range of alternative normalization schemes have been proposed, each operating different components of the hidden activations. These include LayerNorm (Ba et al. 2016), InstanceNorm (Ulyanov) et al. 2016 GroupNorm (Wu & He 2018), and many more (Huang et al. 2020). While these alternatives remove the dependency on the batch size and typically work better than BatchNorm for very small batch sizes, they also introduce limitations of their own, such introducing additional computational costs during inference time. Furthermore for image classification, these alternatives still tend to achieve lower test accuracies than well-tuned BatchNorm baselines. As one exception, we note that the combination of GroupNorm with Weight Standardization (Qiao et al.) 2019) was recently identified as a promising alternative BatchNorm in ResNet-50 (Kolesnikov et 2019)
3 SIGNAL PROPAGATION PLOTS
Although papers have recently theoretically analyzed signal propagation in ResNets (see Section practitioners rarely empirically evaluate the scales of the hidden activations at different depths
2

---

Published as a conference paper at ICLR 2021
Avg Charne Vallance
1.00
Charne
1/ Depth (o)
Block Depth
Block Depth
Figure 1: Signal Propagation Plot for ResNetV2-600 initialization with BatchNorm, ReLU activations and He init, in response to an (0, 1) input at 512px resolution. Black dots indicate the end of a stage. Blue plots use the BN-ReLU-Conv ordering while red plots use ReLU-BN-Conv.
side specific deep network when designing new models proposing modifications to existing architectures. By contrast, we have found that plotting the statistics of the hidden activations at dif- ferent points inside network, when conditioned on a batch of either random Gaussian inputs real training examples, can be extremely beneficial. This practice both enables us to immediately detect hidden bugs in implementation before launching an expensive training also allows us to identify surprising phenomena which might be challenging to derive from scratch.
destined to fail, and
We therefore propose formalize this good practice by introducing Signal Propagation Plots (SPPs), a simple graphical method for visualizing signal propagation on the forward pass in deep ResNets. We assume identity residual blocks of the form Jz+1 fx(2)) + Hy, where ze denotes the input to the 4h block and fe denotes the function computed by the eth residual branch. We consider 4- dimensional input and output tensors with dimensions denoted by N NHC,, where N denotes the batch dimension, C' denotes the channels, and // and W denote the two spatial dimensions. To gen- erate SPPs, we initialize single set of weights according to the network initialization scheme, and then provide the network with batch of input examples sampled from a unit Gaussian distribution. Then, we plot the following hidden activation statistics at the output of each residual block:
Average Channel Squared Mean, computed the square of the mean across the NHW axes, and then averaged across the C axis. In a network with good signal propagation, we would expect the mean activations on each channel, averaged across a batch of examples, to be close to zero. Importantly, we note that it is necessary to measure the averaged squared value of the mean, since the means of different channels may have opposite signs.
Average Channel Variance, computed by taking the channel variance across the NHW axes, and then averaging across the C' axis. We generally find this to be the most informa- tive measure the signal magnitude, and clearly show signal explosion or attenuation.
Average Channel Variance the end of the residual branch, before merging with the skip path. This helps assess whether the layers on the residual branch are correctly initialized.
We explore several other possible choices of statistics one could measure in Appendix G but we have found these three to be the most informative. We also experiment with feeding the network real data samples instead of random noise, but find that this step does not meaningfully affect the key trends. We emphasize that SPPs do not capture every property of signal propagation, and they only consider the statistics the forward pass. Despite this simplicity, SPPs are surprisingly useful for analyzing deep ResNets in practice. We speculate that this may be because in ResNets, as discussed in Section 2 (Taki] 2017 Yang & Schoenholz 2017 Hanin & Rolnick 2018), the backward pass will typically neither explode nor vanish long as the signal the forward pass is well behaved.
As an example, in Figure [ we present the SPP for a 600-layer pre-activation ResNet (He et al.) 201 with BatchNorm, ReLU activations, and He initialization (He et al.) (015). We compare the standard BN-ReLU-Comw ordering to the less common ReLU-BN-Conv ordering. Immediately, several key patterns emerge. First, we note that the Average Channel Variance grows linearly with the depth in given stage, and resets at each transition block to a fixed value close to 1. The linear growth arises because, initialization, the variance of the activations satisfy Var( = Var(xz) + Var(fe(xz)) while BatchNorm ensures that the variance of the activations at the end
See Appendix P for overview of ResNet blocks and their order of operations.
3

---

Published as a conference paper at ICLR 2021
Aug Squared Chernel Meen
Been bug Coeeral Verweze
200
%
123 120 Depth
250
175
770
306 135 150 Block Depth
155
390
200 Depth
Figure 2: SPPs for three different variants of the ResNetV2-600 network (with ReLU activations). In red, we show a batch normalized network with ReLU-BN-Cony ordering. In green we show a cermalizer-free network with He-init and a 1. In cyan, show the same normalizer-fiee free net- work but with Scaled Weight Standardization. We note that the SPPs for a normalizer-fin network with Scaled Weight Standardization almost identical to those for the batch normalized network.
of each residual branch is independent of depth (De & Smith 2020). The variance is reset at each transition block because in these blocks the skip connection is replaced by a convolution operating on normalized input, undoing any signal growth on the skip path in the preceding blocks.
With the RN-ReLU- ordering, the Average Squared Channel Means display similar behavior, growing linearly with depth between transition blocks. This may seem surprising, since expect BatchNorm to center the activations. However with this ordering the final convolution on a residual branch receives rectified input with positive mean. As we show in the following section, this causes the outputs of the branch on any single channel to also have non-zero mean, and explains why Var(fe(ze) s 0.68 for all depths f. Although this "mean-shift" is explicitly counteracted by the normalization layers in subsequent residual branches, it will have serious consequences when attempting to remove normalization layers, as discussed below. In contrast, the Rel.U-BN-Cow ordering trains equally stably while avoiding this mean-shift issue, with Var(fe(xz)) = 1 for all f.
NORMALIZER-FREB RESNETS (NF-RESNETS)
With SPPs in hand to aid our analysis, we now seek to develop ResNet variants without normaliza- tion layers, which have good signal propagation, are stable during training, and reach test accuracies competitive with aitch-normalized ResNets. We begin with two observations from Section B First, for standard initializations, BatchNorm downscales the input to each residual block by a factor pro- portional to the standard deviation of the input signal (De & Smith 2020). Second, each residual block increases the variance of the signal by an approximately constant factor. We propose to mimic this effect by using residual blocks of the form I/+1 = ze + fze ( Be), where xy denotes the input to the fth residual block and fe(-) denotes the pth residual branch. We design the network such that:
f(.), the function computed by the residual branch, is parameterized to be variance pre- serving initialization, i.e., (=)) Var(z) for all l. This constraint enables to reason about the signal growth in the network, and estimate the variances analytically.
3e is fixed scalar, chosen as / Var(zz), the expected empirical standard deviation of the activations 2y at initialization. This ensures the input to fe(-) has unit variance.
* a is scalar hyperparameter which controls the rate of variance growth between blocks.
We compute the expected empirical variance at residual block é analytically according to Var(zz) = Var(+z-1) + a², with an initial expected variance of Var(x.) 1, and we set Be = / /arr(zz). A similar approach was proposed by |rrit et al. (2016) for non-residual networks. As noted in Section 3 the signal variance in normalized ResNets is reset at each transition layer due to the shortcut convolution receiving a normalized input. We mimic this reset by having the shortcut convolution in transition layers operate on (xz/3t) rather than Iz, ensuring unit signal variance at the start of each stage (Var(x++1) = 1 + a² following each transition layer). For simplicity, we call residual networks employing this simple sealing strategy Normalizer- Free ResNets (NstRss
4

---

Published as a conference paper at ICLR 2021
4.1 RELU ACTIVATIONS INDUCE MEAN SHIFTS
We plot the SPPs for Normalizer-Free ResNets (NF-ResNets) with a = in Figure 2 In green, we consider a NE-ResNet, which initializes the convolutions with Gaussian weights using He initializa- tion (He et | Although one might expect this simple recipe to be sufficient to achieve good signal propagation, we observe two unexpected features in practice. First, the average value of the squared channel mean grows rapidly with depth, achieving large values which exceed the average channel variance. This indicates a large "mean shift", whereby the hidden activations for different training inputs (in this case different vectors sampled from the unit normal) are strongly correlated (Jacol et al. 2019; Ruff et al., 2019). Second, as observed for EN-ReLU-Conv networks in Section 3 the scale of the empirical variances on the residual branch are consistently smaller than one.
To identify the origin of these effects, in Figure (in Appendix we provide a similar SPP for a linearized version of ResNetV2-600 without ReLU activation functions. When the ReLU activations are removed, the averaged squared channel means remain close to zero for all block depths, and the empirical variance on the residual branch fluctuates around one. This motivates the following question: why might ReLU activations cause the scale of the mean activations on a channel to grow?
To develop an intuition for this phenomenon, consider the transformation arbitrary and fixed, and g() is an activation function that acts component-wiss wise on iid inputs x such that g(xx) is also iid. Thus, g()) can be any popular activation function like ReLU,, tanh, SiLU, etc. Let (x()())) = Mg and (gz() = og for all dimensions i. It is straightforward to show that the expected value and the variance of any single unit i of the output 2; = ΣS Wi,g((r)) is given by:
Wg(z), where W is
B(z;) = N and Var(z;) = No + rw..))
(I)
where ,..,, and ow. are the mean and standard deviation of the ith row of W:
Aw. = 3/4 Σ We,5, and o.. = N/4 Σ
N W - u....
(2)
Now consider g()) to be the ReLU activation function, i.e., g(x) max(æ, 0). Then g(x) 2 0. which implies that the input to the linear layer has positive mean (ignoring the edge case when all inputs are less than or equal to zero). In particular, notice that if x; ~ (O 1) for all i, then
lg = 1/V2n. Since we know that Pg > O, if jw,, also non-zero, then the output of the transformation, z,, will also exhibit a non-zero mean. Crucially, even if we sample W from distribution centred around zero, any specific weight matrix drawn from this distribution will almost surely have non-zero empirical mean, and consequently the outputs of the residual branches any specific channel will have non-zero mean values. This simple NF-ResNet model with He-initialized weights is therefore often unstable, and it is increasingly difficult to train as the depth increases.
4.2 SCALED WEIGHT STANDARDIZATION
To prevent the emergence of a mean shift, and to ensure that the residual branch fe(.) is variance preserving, we propose Scaled Weight Standardization, a minor modification of the recently pro- posed Weight Standardization (Qiao et al. 2019) which is also closely related to Centered Weight Normalization (Huang al., 20176). We re- re-parameterize the convolutional layers, by imposing.
W., = W NWW. O. VN
where the mean i and variance a are computed across the fan-in extent of the convolutional filters. We initialize the underlying parameters W from Gaussian weights, while 7 is a fixed constant. As in Qiao et al. (2019). we impose this constraint throughout training as a differentiable operation in the forward pass of the network. Recalling equation W (z),, has expected value E(2,) = 0 for all i,
I we can immediately see that the output
of the transformation using Scaled WS, thus eliminating the mean shift. Furthermore, the variance Var(z,) = 720² meaning that for a correctly chosen 7, which depends on the non-linearity g. the layer will be variance preserving. Scaled Weight Standardization is cheap during training and free at inference, scales well (with the number of parameters rather than activations). introduces dependence between batch elements and no discrepancy in training and test behavior, and its implementation does not differ in distributed training. These desirable properties make it a compelling alternatve for replacing Narch
5

---

Published as a conference paper at ICLR 2021
The SPP of a normalizer-free ResNet-600 employing Scaled WS is shown in Figure in cyan. As we can see, Scaled Weight Standardization eliminates the growth of the average channel squared mean at initialization. Indeed, the SPPs are almost identical to the SPPs for a batch- normalzzddd network employing the Rel.U-BN-Conv ordering, shown in red. Note that we select the constant ? to ensure that the channel variance the residual branch is close one (discussed further below). The variance on the residual branch decays slightly near the end of the network due to zero padding.
4.3 DETERMINING SUNLINEARITY-SPECIIE CONSTANTS Y
The final ingredient we need is determine the value of the gain 7, in order ensure that the variances of the hidden activations on the residual branch are close to at initialization. Note that the value of y will depend on the specific nonlinearity used in the network. We derive the value of y by assuming that the input x to the nonlinearity is sampled iid from (00, 1). For ReLU networks, this implies that the outputs g(x) = max 0) will be sampled from the rectified Gaussian distribution with variance oj
(1/2) (1/m)) (Arpit et al. 2016). Since Vax)W = 720g. we set y == 1/0g 2 to ensure that Var(W g(x)) = 1. While the assumption x NN 1) is not typically true unless the network width is large, we find this approximation to work well in practice.
For simple nonlinearities like ReLU tanh, the analytical variance of the non-linearity g(z) when is drawn from the unit normal may be known or easy to derive. For other nonlinearities, such as SiLU (Bliwing et al., 2018 Hendrycks & Gimpel 2016),, recently popularized as Swish (Ramachandran et al. (0188). analytically determining the variance can involve solving difficult integrals, or may even not have an analytical form. In practice, we find that it sufficient to numerically approximate this value by the simple procedure of drawing many N dimensional vectors æ from the Gaussian distribution, computing the empirical variance Var(g(z)) for each vector, and taking the square root of the average of this empirical variance. We provide an example in Appendix D showing how this can be accomplished for any nonlinearity with just a few lines of code and provide reference values.
4.4 OTHER BUILDING BLOCKS AND RELAXED CONSTRAINTS
Our method generally requires that any additional operations used in a network maintain good signal propagation, which means many common building blocks must be modified. As with selecting ? values, the necessary modification can be determined analytically or empirically. For example, the popular Squeeze-ande and-Excitation operation (S+E. Hu et al. (2018)) ), y = sigmoid(MLPP) M L. (ppoo((h)) h. involves multiplication by activation in [o, IJ. and will tend to attemuate the signal and make the model unstable. This attenuation can clearly be seen in the SPP of a normalizer- free ResNet using these blocks (see Figure 9 in Appendix F). If we examine this operation in isolation using our simple numerical procedure explained above, we find that the expected variance is 0.5 (for unit normal inputs), indicating that we simply need to multiply the output by 2 recover good signal propagation. We empirically verified that this simple change is sufficient to restore training stability.
In practice, we find that either a similarly simple modification to any given operation is sufficient to maintain good signal propagation, or that the network is sufficiently robust to the degradation induced by the operation to train well without modification. We also explore the degree to which we can relax our constraints and still maintain stable training. As an example of this, to recover some of the expressivity of a normal convolution, we introduce learnable affine gains and biases to the Scaled WS layer (the gain is applied to the weight, while the bias is added to the activation, as is typical). While we could constrain these values to enforce good signal propagation by, for example, downscaling the output by a scalar proportional to the values of the gains, we find that this is not necessary for stable training, and that stability is not impacted when these parameters vary freely. Relatedly, we find that using learnable scalar multiplier at the end of the residual branch initialized to 0 Gooal et al. 2017 De & Smith (020) helps when training networks over 150 layers, even if ignore this modification when computing 3e. In our final models, we employ several such relaxations without loss of training stability. We provide detailed explanations for each operation and any modifications make in Appendix (also detailed in our model code in Appendix D).

---

Published as a conference paper at ICLR 2021
4.5 SUMMARY
In summary, the core recipe for a Normalizer-Free ResNet (NF-ResNet) is:
1. Compute and forward propagate the expected signal variance 82, which grows by a² after each residual block (3) 1). Downscale the input to each residual branch by Bg.
2. Additionally, downscale the input to the convolution on the skip path in transition blocks by Be, and reset Be+1 = + a² following a transition block.
3. Employ Scaled Weight Standardization in all convolutional layers, computing % the gain specific to the activation function (zx), as the reciprocal of the expected standard deviation, assuming x ~ (0,, 1).
VVar(g(w))
Code is provided in Appendix D for a reference Normalizer-Free Network.
EXPERIMENTS
5.1
AN EMPIRICAL EVALUATION ON RESNETS
FixUp
SkipInit
Ne-ResNets (ours)
Re-Restets
Unreg.
Reg.
Unreg.
Reg.
Unreg.
Reg.
Unreg.
Reg,
RN50
74.0 5
75.9 + 3
73.7 2
75.8 +
75.8 +.1
76.8 + .1
76.8 =11
76.4 + .1
RN101
75.4 .6
77.6 3
75.1 .1
77.3 +
77.1
78.4 +
78.0 .1
78.1 1.11.1 .1
RN152
75.8 +
78.4 + 3
75.7 2
78.0
77.6 + .1
79.1 +
78.6 + 2
78.8 +
RN200
76.2
78.7 + 3
75.9 2
78.2
77.9 + 2
79.6 +
79.0 =
79.2
RN288
76.2 .4
78.4 44
76.3 2
78.7
78.1 1111111
79.5
78.8
79.5 + .1
Table 1: ImageNet Top-1 1 Accuracy (%) for ResNets with FixUp (Zhang et al. 2019) or SkipInit (De Smith, 2020). Normalizer-Free Free ResNets (ours), and Patch-Normalizer ResNets. We train all variants both with and without additional regularization (stochastic depth and dropout). Results are given as the median accuracy the standard deviation across random seeds. indicates a setting where two runs collapsed and results are reported only using the 3 seeds which train successfully.
We begin by investigating the performance of Normalizer-Free pre-activation ResNets on the ILSVRC dataset (Russakovsky et al.] 2015), for which we compare our networks to FixUp initial- ization (Zhang et al. 2019). Skiplnit (De & Smith 2020), and batch batch-normalized ResNets. We use training setup based on Goyal al. (2017), and train our models using SGD (Robbins & Monro
1951) with Nesterov's ' Momentum (Nesterov 1983 Sutskever et al. 2013) for 90 epochs with a batch size of 1024 and a learning rate which warms up from zero to 0.4 over the first 5 epochs, then decays to zero using cosine annealing (Loshchilov & Hutter 2017). We employ standard baseline preprocessing (sampling and resizing distorted bounding boxes, along with random flips), weight decay of 5e-5, and label smoothing of 0.1 (Szegedy et al., 2016). For Normalizer-Frin -Free ResNets (NF-ResNetss) we chose a 0.2 based on a small sweep, and employ Skiplnit as discussed above. For both FixUp and SkipInit we had to reduce the learning rate to 0.2 to enable stable training.
We find that without additional regularization, our NE-ResNets achieve higher training accuracies but lower test accuracies than their batch- normalizedd counterparts. This is likely caused by the known regularization effect of BatchNorm (Hoffer et al. 2017 Luo et al., 2019: De & Smith 2020). We therefore introduce stochastic depth (Huang et al.) 2016) with rate of 0.1, and Dropout (Srivas-) tava et al., 2014) before the final linear layer with a drop probability of 0.25. We note that adding this same regularization does not substantially improve the performance of the normalized ResNets in our setup, suggesting that BatchNorm is indeed already providing some regularization benefit.
In Table I we compare performance of our networks (NF-ResNets) against the baseline (BN- ResNets), across a wide range of network depths. After introducing additional regularization, NF. ResNets achieve performance better than Fix FixUp/Skiplnit and competitive with BN across all net- work depths, with our regularized NF-ResNet- 288 achieving top-1 accuracy of 79.5%. However, some of the 288 layer remeleiee--rre models undergo training collapse at the chosen learning rate,

---

Published as a conference paper at ICLR 2021
NesResNets (ours)
BN-ResNets
BS=1024
BS=8
BS=4
BS=1024
BS=8
BS=4
ResNet-50 69.9 + 0.1
69.6 0.1
69.9
0.1
70.9 1 0.1 65.7 0.2
55.7 0.3
Table 2: ImageNet Top-1 Accuracy (%) for Normalizer-Frre ResNets and Batch-Normalized ResNet-50s on ImageNet, using very small batch sizes trained for 15 epochs. Results are given as the median accuracy + the standard deviation across 5 random seeds. Performance degrades severely for Batch-Normalized networks, while Normalizer-Free ResNets retain good performance.
but only when unregularized. While we can remove this instability by reducing the learning rate 0.2, this comes at the cost of test accuracy. We investigate this failure mode Appendix A
One important limitation of BatchNorm is that its performance degrades when the per-device batch size is small (Hoffer et al. 2017]] Wu & He 2018). To demonstrate that our normalizer-free models overcome this limitation, we train ResNet-50s on ImageNet using very small batch sizes of 8 and 4, and report the results in Table These models are trained for 15 epochs (4.8M and 2.4M training steps, respectively) with a learning rate of 0.025 for batch size 8 and 0.01 for batch size For comparison, we also include the accuracy obtained when training for 15 epochs at batch size 1024 and learning rate 0.4. The NE-ResNet achieves significantly better performance when the batch size is small, and is not affected by the shift from batch size 8 to 4, demonstrating the usefulness of our approach in the microbatch setting. Note that we do not apply stochastic depth or dropout in these experiments, which may explain superior performance of the BN-ResNet at batch size 1024. We also study the transferability of our learned representations to the downstream tasks of semantic segmentation and depth estimation, and present the results of these experiments in Appendix H
5.2 DESIGNING PERFORMANT NORMALIZER-FRER NETWORKS
We turn our attention to developing unnormalized networks which competitive with the state-of-the-art EfficientNet model family across a range of FLOP budgets (Tan & Le| (201)), We focus primarily on the small budget regime (EfficientNets BO-B4), but also report results for B5 and hope to extend our investigation to larger variants in future work.
First, apply Scaled WS and our Normalizer-Free structure directly to the EfficientNet backbone While we suc- ceed in training these networks stably without normalization, we find that even after extensive tuning our Normalizer- Free EfficientNets still substantially un- derperform their batch-normalized base- lines. For example, our normalization free BO variant achieves 73.5% top-1, 3.2% absolute degradation relative to the baseline. We hypothesize that this degra- dation arises because Weight Standard- ization imposes a very strong constraint on depth-wise convolutions (which have an input channel count of 1), and this
84
84
D3
ImageNet Top-1 Accuracy
ReqNaCY-B.OGG
RegNetr-4.OGF
Inception
ResNet50
ResNo:152
RooNetY L.6GF
DeroeNeet-261
constraint may remove substantial frac-
RegNelY 800MT DenseNuut110
tion of the model expressivity. To sup- port this claim, we note that removing Scaled WS from the depth-wise con- volutions improves the test accuracy of Normalizer-Free EfficientNets, although this also reduces the training stability.
76
RegNety GOUMF
Shuftistet x
FfficientNees (AA/RA)
RegNety-40OMF
HH-HHgNNcts (CutMix MixUpp)
(billions)
Figure ImageNet Top-1 test accuracy versus FLOPS.
We were unable to train EfficientNets using SkipInit DDe & Smith 2000 Bachlechner et al. 2020). We
speculate this may he hecause the EfficientNet backhone contains both residual and non-residual components.
8

---

Published as a conference paper at ICLR 2021
Therefore, to overcome the potentially poor interactions between Weight Stan- dardization and depth-wise convolutions, we decided to instead study Normalizer-Fire variants of the RegNet model family (Radosavovic ee al., (020). RegNets are slightly modified variants of ResNeXts (Xie et al. 2017). developed via manual architecture search. Crucially, RegNets employ grouped convolutions, which we anticipate are more compatible with Scaled WS than depth-wise convolutions, since the fraction of the degrees of freedom in the model weights remaining after the weight standardization operation is higher.
We develop new base model by taking the 0.4B FLOP RegNet variant, and making several mi- nor architectural changes which cumulatively substantially improve the model performance. We describe our final model in full in Appendix G however we emphasize that most of the architec- ture changes we introduce simply reflect well-known best practices from the literature (Tan & Lej 2019 He al. 2019). To assess the performance of our Normalizer-Free RegNets across a range of FLOPS budgets, we apply the EfficientNet compound scaling approach (which increases the width, depth and input resolution in tandem according to a set of three power laws learned using architecture search) to obtain model variants at a range of approximate FLOPS targets. Denoting these models NF-RegNets, we train variants BO-B5 (analogous to EfficientNet variants) using both baseline pre- processing and combined CutMix (Yun et al. 2019) and MixUp (Zhang et al. (2088) augmentation. Note that we follow the same compound scaling hyper-parameterss used by EfficientNets, and do not retune these syper--arameters on our own architecture. We compare the test accuracies of Efficient- Nets and NF-RegNets on ImageNet in Figure 3] and we provide the corresponding numerical values We present a comparison of training speeds in Table of of Appendix A
in Table 3 of Appendix A
For each FLOPS and augmentation setting, Re-R attain comparable but slightly lower test accuracies than EfficientNets, while being substantially faster to train. In the augmented setting, we report EfficientNet results with AutoAugment (AA) or RandAugment (RA). (Cubuk et al. 2019) 2020), which we find performs better than training EfficientNets with MixMi However, both AA and RA degrade the performance and stability of NegRsg,sss, and hence we report results of NF-RegNets with CuMMixMMxxup instead. We hypothesize that this occurs because AA and RA were developed by applying architecture search on batch- normalized models, and that they may therefore change the statistics of the dataset in a way that negatively impacts signal propagation when normalization layers removed. To support this claim, we note that inserting single BatchNorm layer after the first convolution in an Ne-RegNet removes these instabilities and enables us to train stably with either AA RA, although this approach does not achieve higher test set accuracies.
These observations highlight that, although our models do benefit from most of the architectural improvements and best practices which researchers have developed from the hundreds of thousands of device hours used while tuming batch-normalized models, there are certain aspects of existing state-of-the-art models, like AA and RA, which may implicitly rely on the presence of activation normalization layers in the network. Furthermore there may be other components, like depth wise convolutions, which are incompatible with promising new primitives like Weight Standardization. It is therefore inevitable that some fine-tuning and model development is necessary to achieve com- petitive accuracies when removing a component like batch normalization which crucial to the performance of existing state-of-the-art networks. Our experiments confirm for the first time that it is possible to develop deep ResNets which do not require batch normalization or other activation normalization layers, and which not only train stably and achieve low training losses, but also attain test accuracy competitive with the current state of the art on a challenging benchmark like ImageNet.
CONCLUSION
We introduce Normalizer-Free Networks, simple approach for designing residual networks which do not require activation normalization layers. Across a range of FLOP budgets, our models achieve performance competitive with the state-of-the-art EfficientNeet on ImageNet. Meanwhile, our em- pirical analysis of signal propagation suggests that batch normalization resolves two key failure modes at initialization in deep ResNets. First, it suppresses the scale of the hidden activations on the residual branch, preventing signal explosion. Second, it prevents the mean squared scale of the activations on each channel from exceeding the variance of the activations between examples. Our Normalizer-Free Networks were carefully designed to resolve both of these failure modes.

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
We would like to thank Karen Simonyan for helpful discussions and guidance, as well as Guillaume Desjardins, Michael Figurnov, Nikolay Savinov, Omar Rivasplata, Relja Arandjelovic, and Rishub Jain.
REFERENCES
Devansh Arpit, Yingbo Zhou, Bhargava Kota, and Venu Govindaraju. Normalization propagation: A parametric technique for removing internal covariate shift in deep networks. In International Conference on Machine Learning, pp. 1168-1176, 2016.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06440, 2016.
Thomas Bachlechner, Bodhisattwa Prasad Majumder, Huanru Henry Mao, Garrison W Cottrell, and Julian McAuley. Rezero is all you need: Fast convergence at large depth. arXiv preprint vrXiv:2003.04887 2020.
David Balduzzi, Marcus Frean, Lennox Leary, JP Lewis, Kurt Wan-Duo Ma, and Brian McWilliams. The shattered gradients problem: If resnets are the answer, then what is the question? In Interna- tional Conference on Machine Learning, pp. 342 142-350,22 2017.
Nils Bjorek, Carla P Gomes, Bart Selman, and Kilian Q Weinberger. Understanding batch normal- ization. In Advances in Neural Information Processing Systems, pp. 7694-7705, 2018.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, and Skye Wanderman-Milhe JAX: composable transformations of Python+NumPP programs, 2018. httttt/t://gthtt com/ google/jax
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 113-123, 2019.
Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 702-703, 2020.
Soham De and Sam Smith. Batch normalization biases residual blocks towards the identity function in deep networks. Advances in Neural Information Processing Systems, 33, 2020.
Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid- linear units for neural network function approximation in reinforcement learning. Neural Networks, 107:3-11,201
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint arXiv: 706702677 2017.
Slan-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in Neural Information Processing Systems, 33, 2020.
Ishaan Gulrajani, Faruk Ahmed, Martín Arjovsky, Vincent Dumoulin, and Aaron C. Courville. Im- proved training of Wasserstein GANs. In Advances in neural information processing systems, 2017.
Boris Hanin and David Rolnick. How to start training: The effect of initialization and architecture. In Advances in Neural Information Processing Systems, pp. 571-581, 2018
10

---

Published as a conference paper at ICLR 2021
Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg. Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Kérard-Marchhar Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array program- ming with numpy. Nature, 585(7825):357-366 Sep 2020. ISSN 1476-4687.
Kevin Sheppard, Tyler Reddy,
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the 2015 IEEE Interna- tional Conference on Computer Vision (ICCV), pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645, Springer, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016b.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-9978, 2020.
Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of tricks for image classification with convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 558-567, 2019.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint 1xXiv:6066000 08415, 2016.
Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generaliza- tion gap in large batch training of neural networks. In Advances in Neural Information Processing Systems, pp. 1731-1741, 2017.
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint urXiv:/704.04580 2017.
Jie Hu, Shen, and Gang Sun. Squeeze-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In European conference on computer vision, pp. 646-661. Springer, 2016.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017a.
Lei Huang, Xianglong Liu, Yang Liu, Bo Lang, and Dacheng Tao. Centered weight normaliza- tion in accelerating training of deep neural networks. In Proceedings of the IEEE International Conference Computer Vision, pp. 2803-2811, 2017b.
Lei Huang, Jie Qin, Yi Zhou, Fan Zhu, Li Liu, and Ling Shao. Normalization techniques in training DNNs: Methodology, analysis and application. arXiv preprini arXiv:2009. /2836, 2020.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In JCML, 2015.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Freeze and chaos for dnns: an ntk view of batch normalization, checkerboard and boundary effects. arXiv preprint arXiv: 1rX.v:1907.05715 2019.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. In Advances in Neural Information Processing Systems, pp. 10215-10224, 2018.
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. celf-normalizing neural networks. In Advances in neural information processing systems, pp. 971-980, 2017.

---

Published as a conference paper at ICLR 2021
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Large scale learning of general visual representations for transfer. arXiv preprint arXhv:1912.113: 2019.
Iro Laina, Christian Rupprecht, Vasileios Belagiannis, Federico Tombari, and Nassir Navab, Deeper depth prediction with fully convolutional residual networks. In 2016 Fourth international confer- ence on 3D vision (3DV), pp. 239-248. IEEE, 2016.
Jonathan Long, Evan Shelhamer, and Trevor Darrel.. Fully convolutional networks for semantic segmentation. In Proceedings of IEEE conference on computer vision and pultern recognition, pp. 3431 1431-3440, 2015.
Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. In Sth International Conference on Learning Representations. ICLR, 2017.
Iya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Represeniations, ICLR, 2019.
Ping Luo, Xinjiang Wang, Wenqi Shao, and Zhanglin Peng. Towards understanding regularization in batch normalization. In 7th International Conference on Learning Representations, ICLR, 2019.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In Proceedings of the European conference on computer vision (ECCV), pp. 116-131, 2018.
Dmytro Mishkin and Jiri Matas. All you need is good init. In 4th International Conference on Learning Representations, ICLR, 2016.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In ICLR, 2018.
Y. NeNeeroo. A method for unconstrained convex minimization problem with the rate of convergence O(1/k²). Doklady AN USSR, pp. (269), 543-547, 1983.
Art B Owen. A robust hybrid of lasso and ridge regression. 2007.
Hung Viet Pham, Thibaud Lutellier, Weizhen Qi. and Lin Tan. Cradle: cross-backend validation to detect and localize bugs in deep learning libraries. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), pp. 1027-1038. IEEE, 2019.
Boris Polyak. Some methods of speeding up the convergence of iteration methods. USSR Compu-
tational Mathematics and Mathematical Physics, pp. 4(5):1117, 1964.
Haozhi Qi. Chong You, Xiaolong Wang, Yi Ma, and Jitendra Malik. Deep isometric learning for visual recognition. In International Conference Machine Learning, pp. 7824-7835. PMLR, 2020.
Siyuan Qiao, Huiyu Wang, Chenxi Liu, Wei Shen, and Alan Yuille. Weight standardization. arXiv preprint arXiv:1903. 2019.
Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 10428-10436. 2020.
Prajit Ramachandran, Barret Zoph, and Quoe V. Le. Searching for activation functions. In 6th In- ternational Conference on Learning Representations, ICIR, Workshop Track Proceedings, 2018. 8.
Herbert Robbins and Sutton Monro. A stochastic approximation method. The Annals of Mathemat- ical Statistics, pp. 22(3):400-407, 1951.
Samuel Rota Bulò, Lorenzo Porzi, and Peter Kontschieder. In-place activated batchnorm for memory- memory-optimizee training of dnns. In Proceedings of the IEEE Conference on Computer Vi- sion and Pattern Recognition, pp. 5639-5647, 2018.
12

---

Published as a conference paper at ICLR 2021
Brendan Ruff, Taylor Beck, and Joscha Bach. Mean shift rejection: Training deep neural networks without minibatch statistics or normalization. arXiv preprint 1rXiv:1911.13173, 2019.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, and Michael Bernstein. ImageNet large scale visual recognition challenge. I/CV, 115:211-252, 2015.
Tim Salimans and Durk Kingma. Weight normalization: simple reparameterization accelerate training of deep neural networks. In Advances in neural information processing systems, pp. 901- 909, 2016.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chigh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018.
Shibani Santurkar, Dimitris Tsipras, Andrew llyas, and Aleksander Madry. How does batch nor- malization help optimization? In Advances in Neural Information Processing Systems. pp. 2483- 2493, 2018.
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and sup- port inference from rgbd images. In European conference on computer vision, pp. 746-760. Springer, 2012.
Samuel Smith, Erich Elsen, and Soham De. On the generalization benefit of noise in stochastic gradient descent. In International Conference on Machine Learning, pp. 9058-9067 PMLR, 2020.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 1599)1192911958 2014.
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. arXiv preprint 100x::1505500387.. 2015.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initial- ization and momentum in deep learning. In International conference on machine learning. pp- 1139-1147, 2013.
C Szegedy, V Vanhoucke, loffe, J Shlens, and Z Wojna. Rethinking the inception architecture for computer vision. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2818-22888 18282222622211 2016.
Christian Szegedy, Sergey loffe, Vincent Vanhoucke, and Alexander A. Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning. In Proceedings of the Thirty- First AAAI Conference on Artificial Intelligence, pp. 4278-4284, 2017.
Masato Taki. Deep residual networks and weight initialization. arXiv preprint arXiv:1709.022. 2017.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International Conference on Machine Learning, pp. 6105-6114, 2019.
Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Hervé Jégou. Fixing the train-test resolution discrepancy. In Advances in Neural Information Processing Systems, pp. 8252-8262, 2019.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing in-
gredient for fast stylization. arXiv preprint arXiv:1607.08022. 2016.
Longhui Wei, An Xiao, Lingxi Xie, Xiaopeng Zhang. Xin Chen, and Qi Tian. Cireumventing outliers of autoaugment with knowledge distillation. ECCV, pp. 608 608-625. 2020.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 3-19, 2018.
13

---

Published as a conference paper at ICLR 2021
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492-1500, 2017.
Yunyang Xiong, Hanxiao Liu, Suyog Gupta, Berkin Akin, Gabriel Bender, Pieter-Jan Kindermans, Mingxing Tan, Vikas Singh, and Bo Chen. Mobiledets: Searching for object detection architec- tures for mobile accelerators. arXiv preprint arXiv:2004. 2020.
Ge Yang and Samuel Schoenholz. Mean field residual networks: On the edge of chaos. In Advances in neural information processing systems, pp. 7103-7114, 2017.
Greg Yang. Jeffrey Pennington, Vinay Rao, Jascha Sohl-Dickstein, and Samuel S. Schoenholz. A mean field theory of batch normalization. In 7th International Conference on Learning Represen- tations, ICLR, 2019.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceed- ings of the IEEE International Conference on Computer Vision, pp.
2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Proceedings of the British Machine Vision Conference 2016, BMVC, 2016.
Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond em- pirical risk minimization. In 6th International Conference on Learning Representations, ICLR, 2018.
Hongyi Zhang, Yann N. Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization. In 7th International Conference on Learning Representations. ICLR, 2019.
14

---

Published as a conference paper at ICLR 2021
APPENDIX A EXPERIMENT DETAILS
Model
#FLOPs
#Params
Top-1 w/o Augs
Top-1 w/ Augs
Ne-RegNet-BO
0.44B
8.3M
76.8 + 0.2 76.7
77.0 0.1 77.1
EfficieenNNetB0
0.398
5.3M
RegNerY-400MF
0.40B
4.3M
74.1
NF-RegNet-BI
73B
9.8M
78.6 + 0.1
78.7 + 0.1
EfficientNet-B1
0.70B
7.8M
78.7
79.1
RegNecY-600MMF
0.60B
6.1M
75.5
RegNerY-800MMF
0.80B
6.3M
76.3
MobileNet (Howard et al.
2017)
0.60B
4.2M
70.6
MobileNet V2 (Sandler et 2018
0.59B
6.9M
74.7
ShuffleNet v2
(Ma et al. 2018
0.59B
74.9
NF-RegNet-B2
1.09B
13.4M
79.6 + 0.1 79.8
80.0 + 0.1 80.1
EfficientNet-B2
1.00B
9.2M
NF-RegNet-BB3
1.98B
17.6M
80.6 + 0.1 81.1
81.2 + 0.1 81.6
EfficientNet-B3
1.80B
12.0M
NF-RegNet-B4
4.43B
28.5M
81.7 0.1
82.5 0.1
EfficientNet-B4
4.20B
19.0M
82.5
82.9
RegNeeY--40GF
4.00B
20.6M
79.4
ResNet50
4.10B
26.0M
76.8
78.6
DenseNet-169 (Huang et al. 2017a
3.50B
14.0M
76.2
NF-RegNet-B3
10.48B
47.5M
82.0 + 0.2
83.0 + 0.2
EfficientNet-B5
9.90B
30.0M
83.1
83.7
RegNetY-12GF
12.10B
51.8M
80.3
ResNet152
11.00B
60.0M
78.6
Inception-ve v4 (Szegedy et al.
2017)
13.00B
48.0M
80.0
Table 3: ImageNet Top-1 Accuracy (%) comparison for NF-RegNets and recent state-of-the-ar models. "w/ Augs" refers to accuracy with advanced augmentations: for EfficientNets, this is with AutoAugment RandAugment. For NF-RegNets, this is with CutMix MixUp. Ne-RegNet results are reported as the median and standard deviation across 5 random seeds.
A. STABILITY, LEARNING RATES, AND BATCH SIZES
Previous work Goyal et al. 2017) has established a fairly robust linear relationship between the optimal learning rate (or highest stable learning rate) and batch size for Batch-Normalized ResNets. As noted in [Smith et al. 2020). we also find that this relationship breaks down past batch size 1024 for unnormalized ResNets, opposed 2048 4096 for normalized ResNets. Both the optimal learning rate and the highest stable learning rate decrease for higher batch sizes. This also appears correlate with depth: when not regularized, our deepest models are not always stable with a learning rate of 0.4. While we can mitigate this collapse by reducing the learning rate for deeper nets, this introduces additional tuning expense and is clearly undesirable. It is not presently clear why regularization aids in stability; we leave investigation of this phenomenon to future work.
Taking a closer look collapsed networks, we find that even though their outputs have exploded (becoming large enough go NaN), their weight magnitudes are not especially large, even if we re- move our relaxed affine transforms and train networks whose layers are purely weight-standarddied. The singular values of the weights, however, end up poorly conditioned, meaning that the Lipschitz constant the network can become quite large, an effect which Scaled WS does not prevent. One might consider adopting strain this constant (Gulrajani et al. 2017 Miyato et al., 2018). but we have found that this added complexity and expense is not necessary develop performant unnormalized networks.
of the many techniques from the GAN literature to regularize
15

---

Published as a conference paper at ICLR 2021
This collapse highlights important limitation of our approach, and of SPPs: as SPPs only show signal prop for a given state of a network (i.e., at initialization), no guarantees are provided far from initialization. This fact drives to prefer parameterizations like Scaled WS rather than solely relying on initialization strategies, and highlights that while good signal propagation is generally necessary for stable optimization, is not always sufficient.
A.2 TRAINING SPEED
BSI6
BS32
BS64
NF
BN
NF
BN
NF
BN
ResNet-50
17.3
16.42
10.5
9.45
5.79
5.24
ResNet- 101
10.4
9.4
6.28
5.75
3,46
3.08
ResNet- 152
7.02
5.57
4.4
3.95
2.4
2.17
ResNet-200
5.22
3.53
3.26
2.61
OOM
OOM
ResNet-288
3.0
2.25
OOM
OOM
OOM
OOM
Table 4: Training speed (in training iterations per second) comparisons of Ne-ResNets and BN- ResNets on a single 16GB V100 for various batch sizes.
BS16
BS32
BS64
NF-RegNet
EfNNet
Re-RegNet
EffNet
Ne-RegNet
EffNet
B0
12.2
5.23
9.25
3.61
6.51
2.61
B1
9.06
3.0
6.5
2.14
4.69
1.55
B2
5.84
2.22
4.05
1.68
2.7
1.16
B3
4.49
1.57
3.1
1.05
2.13
OOM
B4
2.73
0.94
1.96
OOM
OOM OOM
OOM
B5
1.66
OOM
OOM
OOM
OOM
Table 5: Training speed (in training iterations per second) comparisons of NF-RegNets and Batch- Normalized EfficientNets on a single 16GB V100 for various batch sizes.
We evaluate the relative training speed of our normalizer-fie models against batch- normalized mod- els by comparing training speed (measured as the number of training steps per second). For com- paring NF-RegNets against EfficientNets, we measure using the EfficientNet image sizes for each variant to employ comparable settings, but practice we employ smaller image sizes so that our actual observed training speed for Ne-RegNets is faster.
16

---

Published as a conference paper at ICLR 2021
APPENDIX B MODIFIED BUILDING BLOCKS
In order to maintain good signal propagation in our Normalizer-Free models, we must ensure that any architectural modifications do not compromise our model's conditioning, as we cannot rely on activation normalizers to automatically correct for such changes. However, our models are not so fragile as to be unable handle slight relaxations in this realm. We leverage this robustness to improve model expressivity and to incorporate known best practices for model design.
B. AFFINE GAINS AND BIASES
First, we add affine gains and biases to our network, similar to those used by activation normaliz- These applied as vector gain, each element of which multiplies given output unit of reparameterized convolutional weight, and a vector bias, which added to the output of each con- volution. We also experimented with using these separate affine transform applied before the ReL.U, but moved the parameters next to the weight instead to enable constant-folding for inference. As is common practice with normalizer parameters, do not weight decay or otherwise regularize these weights.
Even though these parameters are allowed vary freely, do not find that they responsible for training instability, even in networks where we observe collapse. Indeed, we find that for settings which collapse (typically due to learning rates being too high), removing the affine transform has no impact on stability. As discussed in Appendix A we observe that model instability arises result of the collapse of the spectra of the weights, rather than any consequence of the affine gains and biases.
B.2 STOCHASTIC DEPTH
We incorporate Stochastic Depth (Huang et al. 2016). where the output of the residual branch of block is randomly set to zero during training- This often implemented such that if the block is kept, its value is divided by the keep probability. We remove this rescaling factor to help maintain signal propagation when the signal is kept, but otherwise do not find it necessary to modify this block.
While possible that we might have an example where many blocks are dropped and signals are attenuated, in practice we find that, as with affine gains, removing Stochastic Depth does not improve stability, and adding does not reduce stability. One might also consider slightly more principled variant of Stochastic Depth in this context, where the skip connection is upscaled by + the residual branch is dropped, resulting in the variance growing as expected, but we did not find this strategy necessary.
B.3 SQUEEZE AND EXCITE LAYERS
As mentioned Section 4.4 we incorporate Squeeze and Excitation layers (Hu et al. 2018), which we empirically find reduce signal magnitude by factor of 0.5, which simply corrected by multiplying by 2. This was determined using similar procedure to that used to find values for a given nonlinearity, as demonstrated in Appendix [ We validate this empirically by training NF- RegNet models with unmodified S+E blocks, which do not train stably, and NF-RegNet models with additional correcting factor 2, which do train stably.
B.4 AVERAGE POOLING
In line with best practices determined by He al. (2019), in our NF-RegNet models replace the strided Ixl convolutions with average pooling followed by Ixl convolutions, a common alternative also employed in Zagoruyko Komodakis (2016). We found that average pooling with kernel of size & x l tended to attenuate the signal by a factor of k, but that it was not necessary to apply any correction due to this. While this will result mis-estimation of I values at initialization, does not harm training (and average pooling in fact improved results over strided 1x1 convolutions every case we tried), so we simply include this operation as-is.
17

---

Published as a conference paper at ICLR 2021
APPENDIX C MODEL DETAILS
We develop the Ne-RegNet architecture starting with a RegNerY-400MF architecture (Radosavovic
al. (2020) low-latency RegNet variant which also uses Squeeze+Excite blocks (Hu et al. 2018)) and uses grouped convolutions with a group width of 8. Following EfficientNets, we first add an additional expansion convolution after the final residual block, expanding to 1280w channels. where w is model width multiplier hyperparameter. We find this to be very important for performance: if the classifier layer does not have access to a large enough feature basis, it will tend to underfit (as measured by higher training losses) and underperform. We also experimented with adding an additional linear expansion layer after the global average pooling, but found this not to provide the same benefit.
Next, we replace the strided 1x1 convolutions in transition layers with average pooling followed by Ixl convolutions (following [He et al. (2019)), which we also find to improve performance. We switch from ReLU activations to SiLU activations (Elfwing ct al. 2018 Hendrycks & Gimpel| [0116] Ramachandran al. 2018). We find that SiLU's benefits are only realized when used in conjunction with EMA (the model averaging we use, explained below), as in EfficientNets. The performance
of the underlying weights does not seem to be affected by the difference in nonlinearities, so the improvement appears to come from SiLU apparently being more amenable to averaging.
We then tune the choice of width w and bottleneck ratio g by sweeping them the 0.4B FLOP model. Contrary to Radosavovic et al. (2020) which found that inverted bottlenecks (Sandler et al.) 2018 were not performant, we find that inverted bottlenecks strongly outperformed their compres- sive bottleneck counterparts, and select w 0.75 and g 2.25. Following EfficientNets (Tan & Le 2019), the very first residual block in network uses g = 1, FLOP-reducing strategy that does not appear to harmfully impact performance.
We also modify the S+E layers to be wider by making their hidden channel width a function of the block's expanded width, rather than the block's 's input width (which is smaller in an inverted bottleneck). This results in our models having higher parameter counts than their equivalent FL.OP target EfficientNets, but has minimal effect on FLOPS, while improving performance. While both FLOPS and parameter count play a part in the latency of a deployed model, (the quantity which is often most relevant in practice) neither are fully predictive of latency (Xiong et al.) 2020). We choose to focus on the FLOPS target instead of parameter count, as one can typically obtain large improvements in accuracy the input image, which will dramatically increase the FLOPS.
given parameter count by, for example, increasing the resolution of
With our baseline model in hand, apply the EfficiintNNt compound scaling (increasing width, depth, and input image resolution) to obtain a family of models at approximately the same FLOP targets as each EfficientNet variant. We directly use the EfficientNet width and depth multipliers for models BO through B5, and tune the test image resolution to attain similar FLOP counts (although our models tend to have slightly higher FLOP budgets). Again contrary to Radosavovic al. (2020), which scales models almost entirely by increasing width and group width, we find that the Effcient- Net compound scaling works effectively as originally reported, particularly with respect to image size. Improvements might be made by applying further architecture search, such as tuning the w and g values for each variant separately, or by choosing the group width separately for each variant.
Following Touvron et al. (2019), we train on images of slightly lower resolution than we test on, primarily to reduce the resource costs of training. We do not employ the fine-tuning procedure of Touvron et al. (201)) The exact train and test image sizes we use are visible in our model code in Appendix D
We train using SGD with Nesterov Momentum, using batch size of 1024 for 360 epochs, which is chosen to be in line with EfficientNet's schedule of 360 epoch training at batch size 4096. We employ a 5 epoch warmup to a learning rate of 0.4 (Goyal et al. 2017). and cosine annealing to 0 over the remaining epochs (Loshchilov & Hutter 2017). As with EfficientNets, we also take an exponential moving average of the weights (Polyak 1964h, using a decay of 0.99999 which employs a warmup schedule such that at iteration i, the decay is decay = min(i, 1047) ). We choose a larger decay than the EfficientNets value of 0.9999, as EfficientNets also take an EMA of the running average statistics of the BatchNorm layers, resulting in longer horizon for the averaged model.
18

---

Published as a conference paper at ICLR 2021
As with EfficientNets, find that some of our models attain their best performance before the end of training, but unlike EfficientNets we do not employ early stopping, instead simply reporting per- formance the end of training. The source of this phenomenon is that some models (particularly larger models) reach the end of their decay schedule, the rate of change their weights slows, ulti- mately resulting in the averaged weights converging back towards the underlying (less performant) weights. Future work in this area might consider examining the interaction between averaging and learning rate schedules.
Following EfficientNets, we also use stochastic depth (modified to remove the rescaling by the keep rate, as to better preserve signal) with drop rate that scales from 0 to 0.1 with depth (reduced from the EfficientNets value of 0.2). We swept this value and found the model to not be especially sensitive as long as it was not chosen beyond 0.25. We apply Dropout (Srivastava et al.) 2014) to the final pooled activations, using the same Dropout rates as EfficientNets for each variant. We also use label smoothing (Szegedy et al. 2016) of 0.1, and weight decay of 5e-5.
19

---

Published as a conference paper at ICLR 2021
APPENDIX D MODEL CODE
We here provide reference code using Numpy (Harris et al. 2020 and JAX (Bradbury et al. 2018
Our full training code is publicly available at dpmd. ai ai/fnets s
D.1 NUMERICAL APPROXIMATIONS NONLINEARITY-SPECIIC GAINS
It is often faster to determine the cettineaartt--speciie constants y empirically, especially when the chosen activation functions are complex or difficult to integrate. One simple way to do this for the SiLU function is to sample many (say, 1024) random C-dimensional vectors (of say size 256) and compute the average variance, which will allow for computing an estimate of the constant. Empirically estimating constants to ensure good signal propagation in networks at initialization has previously been proposed in Mishkin & Matas (0016) and Kingma & Dhariwal 2018).
import jax
import jax.numpy numpy as jnp
key = jax. wwwnddm.PNNeaae (2) # Arbitrary key
large batch random noise vectors x = jax. random normal (key, (1024, 256))
# Produce
y jax. tax.nn.silu
# Take the average variance of many random batches
gamna inp. mean (jnp. var (y, axis=1)) *x -0.5
20

---

Published as a conference paper at ICLR 2021
APPENDIX E OVERVIEW OF EXISTING BLOCKS
This appendix contains an overview of several different types of residual blocks.
bil Conv
1xl Conv
ReIUU
BetchNorm
3x3 Strided
3x3 Conv
Conv
ReLU
BatchNorm
RatchNorm
Ixil Conv
1xl Conv
Ixl Stricied
Convv
ReLU
BetchNorm
BatchNorm
(a) Pre-Activation ResNet Block
(b) Pre-Activation ResNet Transition Block
Figure 4: Residual Blocks for pre-activation ResNets (He et al. 2016a). Note that some variants swap the order of the nonlinearity and the BatchNorm, resulting in signal propagation which is more similar to that of our normalizer-free networks.
21

---

Published as a conference paper at ICLR 2021
ReLU
Rwl
BatchNorm
I Conv
Ixl Conv
listohoorm
Rei U
BatchNorm
BatchNorm
Batchhirrm
1x1 Conv
Stridie
(a) Post-Activation ResNet Block
(b) Post- Aostv-cctvvtion ResNet Transition Block
Figure Residual Blocks post-activation (original) ResNets (He et al. 2016b).
22

---

Published as a conference paper at ICLR 2021
APPENDIX F ADDITIONAL SPPS
In this appendix, we include additional Signal Propagation Plots. For reference, given an NHWC lensor, we compute the measured properties using the equivalent of the following Numpy (Harris et al. 2020) snippets:
Average Channel Mean Squared:
np. mean (np mean (y, axis- 10, 1, 2])
2)
Average Channel Variance:
ap mean (np. var (y, axis=[0, 1, 21)
Residual Average Channel Variance:
np.mean mean (np. var (f (x), axis [0, 1, 21)
Vertance
Cheene
Residui
15
Figure 6: Signal Propagation Plot for ResNerV2-600 with ReLU and He initialization, without any normalization, on semilog scale. The scales of all three properties grow logarithmically due signal explosion.
Arg Charmel Var lanccc
Block Depth
Block Depth
Block
Deptn
(0)
Figure Signal Propagation Plot for 600-layer ResNetV2s with linear activations, comparing BatchNorm against with normalizer-free scaling. Note that the max-pooling operation in the ResNet stem has here been removed so that the inputs to the first blocks are centered.
Eanimtee Auy Cherree Ueitevoe
Bog Chaoee Veltere
4/g3
Clock Depts
Block Depth
Block Depti
Figure 8: Signal Propagation Plot for a Normalizer-Fn Free ResNetV2-600 with ReLU and Scaled WS. using ? 2, the gain for ReLU from (He al. 2015). As this gain (derived from / E[a(ayy)) is
lower than the correct gain (v Var(s(æ)) signals attenuate progressively in the first stage, then are further downscaled at each transition which uses 3 value that assumes higher incoming scale.
23

---

Published as a conference paper at ICLR 2021
deg Cheore Vicaere
Clock Dapes
Block Depth
Block Depes
Figure 9: Signal Propagation Plot for a Normalizer-Fite Free ResNetV2-600 with ReLU, Scaled WS with correctly chosen gains, and unmodified Burrezze-and-Exee Blocks. Similar to understimating values, unmodified S+E blocks will attenuate the signal.
Avg Channel Varamce
Dusidaal Aug
200
129 130 (8)
120 (b)
Block Depth
Figure 10: Signal Propagation Plot for a ResNet-600 with FixUp. Due to the zero-initialized weights, FixUp networks have constant variance in stage, and still demonstrate variance resets across stages.
Rendiul Avg Diemmt Vertance
Heg Charmel Jariarce Cc
Avg squars
15
100
100 Depth (0)
100
200
120
Depth
Depth
Figure 11: Signal Propagation Plot for ResNet-600 VI with post-activation ordering. As this variant applies BatchNorm at the end the residual block, the Residual Average Channel Variance is kept constant at throughout the model. This ordering also applies BatchNorm shortcut 1x1 convolutions at stage transition, and thus also displays variance resets.
24

---

Published as a conference paper at ICLR 2021
APPENDIX G NEGATIVE RESULTS
G.1
FORWARD MODE VS DECOUPLED WS
Parameterization methods like Weight Standardization (Qiao al.] 2019). Weight Normalization (Saliaans & Kingmal 2016). and Spectral Normalization (Miyato et al. 2018) are typically pro- posed as "foward mode" modifications applied to parameters during the forward pass of network. This has two consequences: first, this means that the gradients with respect to the underlying param- eters are influenced by the parameterization, and that the weights which are optimized may differ substantially from the weights which are actually plugged into the network.
One alternative approach is implement "decoupled" variants of these parameterizers, by applying them as projection step in the optimizer. For example, "Decoupled Weight Standardization" can be implemented atop any gradient based optimizer by replacing W with the normalized W after the update step. Most papers proposing parameterizations (including the above) argue that the pa- rameterization's gradient influence is helpful for learning, but this is typically argued with respect to simply ignoring the parameterization during the backward pass, rather than with respect to a strategy such this.
Using a Forward- Mode parameterization may result in interesting interactions with moving averages or weight decay. For example, with WS, if one takes a moving average of the underlying weights, then applies the WS parameterization to the averaged weights, this will produce different results than if one took the EMA of the Beight-Standarrdize parameters. Weight decay will have similar phe- nomenon: if one is weight decaying a parameter which is actually a proxy for a weight-standardized parameter, how does this change the behavior of the regularization?
We experimented with Decoupled WS and found that it reduced sensitivity to weight decay (pre- sumably because of the strength of the projection step) and often improved the accuracy of the EMA weights early in training, but ultimately led to worse performance than using the originally proposed moodwadd-moee formulation. We emphasize that our experiments in this regime were only cursory, and suggest that future work might seek to analyze these interactions in more depth.
We also tried applying Scaled WS as a regularizer ("Soft WS") by penalizing the mean squared error between the parameter W and its Scaled WS parameterization, W. We implemented this as direct addition to the parameters following Loshchilov & Hutter (2019) rather than as a differentiated loss, with a scale hyperparameter controlling the strength of the regularization. We found that this scale could not be meaningfully decreased from its maximal value without drastic training instability, indicating that relaxing the WS constraint is better done through other means, such as the affine gains and biases we employ.
G.2 MISCELLANEOUS
For SPPs, we initially explored plotting activation mean (ap mmp. (h)) instead of the average squared channel mean, but found that this was less informative.
We also initially explored plotting the average pixel norm: the Frobenius norm of each pixel (reduced across the C' axis) then averaged across the NHW axis, mp. mean (np. lipalinggg... norm (h, axis=-1) )). We found that this value did not add any information not already contained in the channel or residual variance measures, and was harder to interpret due to it varying with the channel count.
We explored Ne-ResNet variants which maintained constant signal variance, rather than mimicking Batch-Normalized ResNets with signal growth + resets. The first of two key components in this approach was making use of "rescaled sum junctions," " where the sum junction in residual block was rewritten to downscale the shortcut path as y -+)())== which is approximately norm-preserving f [ax) orthogonal to x (which we observed to generally hold in practice). Instead of Scaled WS, this variant employed SeL.U (Klambauer et al., (017)) activations, which we found to work as-advertised in encouraging centering and good scaling.
While these networks could be made to train stably, we found tuning them to be difficult and were not able to casily recover the performance of BN-ResNets as we were with the approach ultimately presented in this paper.
25

---

Published as a conference paper at ICLR 2021
APPENDIX H EXPERIMENTS WITH ADDITIONAL TASKS
H.1 SEMANTIC SEGMENTATION ON PASCAL VOC
NE-ResNets (ours)
Ne-ResNets
mloU
mloU
ResNet-50
74.4
75.4
101
76.7
77.0
ResNet-152
77.6
77.9
ResNet-200
78.4
78.0
Table 6: Results on Pascal VOC Semantic Segmentation.
We present additional results investigating the transferability of our normalizer-fiet free models down- stream tasks, beginning with the Pascal VOC Semantic Segmentation task. We use the FCN archi- tecture (Long et al. (015)) following He et al. (2020) and Grill al. (2020) We take the ResNet backbones of each variant and modify the 3x3 convolutions in the final stage to have dilation 2 and stride of I, then add two extra 3x3 convolutions with dilation 6, and final Ixl convolution for classification. We train for 30000 steps at batch size 16 using SGD with Nesterov Momentum of 0.9, a learning rate of 0.003 which is reduced by a factor of 10 at 70% and 90% of training, and weight decay of 5e-5. Training images horizontal flips, and random crops. Results in mean Intersection over Union (mloU) are reported in Table 6 on the val2012 set using single 513 pixel center crop. We do not add any additional regularization such as stochastic depth or dropout. NF-ResNets obtain comparable performance to their BN-ResNet counterparts across all variants.
augmented with random scaling in the range [0.5, 2.0]), random
H.2 DEPTH ESTIMATION ON NYU DEPTH v2
Higher better
Lower Better
pet. < 1.25
pct. < 1.25²
pet. < 1,253
rms
rel
NF-ResNet-50
81.9
95.9
98.9
0.572
0.141
BN-ResNet-50
81.7
95.7
98.8
0.579
0.141
NF-ResNett101
82.7
96.4
99.0
0.564
0.136
BN-ResNeet101
83.4
96.2
98.9
0.563
0.132
Nf-ResNet-1152
83.2
96.4
99.1
0.558
0.135
BN-ResNet-152
81.6
96.0
98.9
0.579
0.140
NF- NF-ResNet-200
84.0
96.7
99.2
0.552
0.130
BN-ResNet-200
84.6
96.6
99.1
0.548
0.125
Table 7: Results on NYUv2 Depth Estimation.
We next present results for depth estimation on the NYU v2 dataset (Silberman et al.) 2012) using the protocol from (Laina et al.) 2016). We downsample the images and center crop them to [304, 228] pixels, then randomly flip and apply several color augmentations: grayscale with probability 30%, brightness with maximum difference of 0255, saturation with random factor picked from [0.5, 1.5], and Hue with adjustment factor picked [-0.2, 0.2]. We take the features from the final residual stage and feed them into the up projeeectioon blocks from (Silberman et al. 2012), then train with a reverse Huber loss (Laina et al. [0166] Owen (007). We train for 7500 steps at batch size 256 using SGD with Nesterov Momentum of 0.9, learning rate of 0.16, and cosine annealing. We report results in Table7 using five metrics commonly used for this task: the percentage of pixels where the magnitude of the relative error (taken the ratio of the predicted depth and the ground truth, where the denominator is whichever is smaller) is below certain threshold, as well as root- mean-squared and relative error (rms and rel), As with semantic segmentation, we do not apply any additional regularization, and find that our normalizer-Fe ResNets attain comparable performance across all model variants.
26


Reviews:
[{"review_id": "oskMJqMxDSC", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe authors propose a set of visualization tools named SPPs for better monitoring the key indicators, including Average Channel Squared Mean, Average Channel Variance, etc., of hidden activations. With SPPs in hand, they find two key failure modes at initialization in deep ResNets and then develop Normalizer-Free ResNets using Scaled Weight Standardization, achieving competitive performance on ResNet-288 and Efficient-Net.\n\nStrength:\n--The overall idea makes sense and the proposed method of removing batch normalization can reduce computational resources and speed up computing greatly.\n--The visualization of the key indicators may be helpful for understanding how batch normalization works and how to remove batch normalization.  \n--The experiment results seem that the proposed method achieves good performance in large-scale ResNets.\n\nWeakness:\n-- There is a lack of accuracy comparison with other removing batch normalization works. \n-- It will be better to prove the effectiveness of the proposed method in more tasks, not only the classification task. \n\nComments:\n(1)\tThe batch size you used for training is 1024. As you know, decreasing batch size is good for online training on portable devices. Can the proposed method decrease the batch size to a small value like 2/4/8 after removing batch normalization? And if it is difficult, can you give an explanation on how batch size affects the performance after removing batch normalization with the proposed method. \n(2)\tYou have visualized the key indicators of the proposed initialization method. Can you give the visualization of the same indicators of other initialization methods like Fixup initialization and make some comparison? I think it’s within your ability. \n(3)\tIt’s better to add some accuracy comparisons with other removing batch normalization works.\n\n------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\nUPDATE: The author has addressed most of my concerns, but regarding the motivations and the benefits for the community, I still keep my score.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "2XcdWUJ7qmS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a novel, and mathematically well motivated initialization scheme for deep ResNet like models. \nThis is used to train batchnorm free Resnets that compare to their baseline.\nThey also train RegNet like models that are fairly close to EfficientNets in terms of performance.\n\nComments: \n\nYou list problems with batchnorm:\n1) 'it breaks independence between training examples'\n   Why is this a problem? You don't include any experiments showing your method works well in the small batch regime where batchnorm is problematic.\n2) 'it is suprisingly expensive to compute'\n    Batchnorm was widely adopted as it made networks train much faster. And there is no computational overhead at test time. Does your method allow models to be trained more quickly, i.e. how does accuracy compare after $N$ minutes of training time?\n3) 'it often results in unexpected bugs'\n   I don't understand this point. Batchnorm has lead to many state-of-the-art results. In contrast you have only managed to get good results by using large amounts of prior knowledge that was obtained from experiments on the same validation set and using batchnorm.\n\nSection 5.1 Why is the training collapse of the 288 layer models not reflected in Table 1?\n\nRegarding Figure 3: Why compare your method **with** data augmentation and EfficientNets **without**? That seems very misleading to me. Getting close to proper EfficientNet performance without batchnorm is a decent achievement. Why include an inferior, incomparable baseline?\n\n--update---\nI am upgrading to 7: Good paper accept; as my concerns have been addressed by the additional experiments.\nThe proposed method is not yet a drop in replacement for BatchNorm in general, but it can be useful in specific circumstances, i.e. small batch-size training.", "extra_scores": {}, "extra_text": {}}, {"review_id": "00ddVX6k5ny", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes the signal propagation plot (spp) which is a tool for analyzing residual networks and analyzes ResNet with/without BN. Based on the investigation, the authors first provide ResNet results without normalization with the proposed scaled weight standardization. Furthermore, the authors provide a bunch of models that are competitive to EfficientNets based on RegNetY-400MF,  which seem to be highly tuned in terms of architecture design.\n\nPros)\n\n1. This paper is well written and easy to follow.\n\n2. The proposed SPP seems to be a good tool for analyzing a model.\n\n\nCons)\n\n1. The authors seem to have failed to provide any reasons for needing normalization free ResNet over the original BN-ResNet. \n\n2. It is not clear that the trained model without NF with SWS can be used as a backbone that can be directly applied to downstream tasks (e.g., object detection). \n\n3. I don't think it was necessary to show competitive results with EfficientNets using the other baseline - RegNet.  Especially, the proposed models (which are compared with EfficientNets) are highly-tuned trying to surpass EfficientNets' accuracy. This type of paper would be better to be focused on investigating the characteristics of a network.\n\nComments)\n1. The major problem of this paper is none of the advantages of NF with SWS are highlighted over BN, so it is hard to find any reasons for replacing BN with NF-SWS. I mean non of the disadvantages of BN are addressed by the proposed method.\n\n2. SPP could enlighten that an unusual ReLU-BN-Conv ordering would have some benefits but why should a network without normalization mimick the SPP trend of ResNet? It is unnatural that a network without normalization should follow the ResNet's behavior.\n\n3. From the equation x_{l+1}=x_l+ a * f (x_l/b_l) at p.4, the proposed approach eventually normalizing the feature even if a or b_l is fixed. Moreover, gamma in eq. (3) additionally scaling up or down the weight which ultimately gives an effect on the computed feature.\n\n4. How did the authors compute gamma in eq.(3) in a training phase? Why the provided code contains a learnable gamma?\n\n5. Why \"zero padding\" at p.5 affects the variance decay in the rightmost graph in Figure 2?\n\n6. Please clarify why RegNetY-400MF chose it as the baseline. It is not clear that the authors pick RegNet and tune it highly and compared with EfficientNets. If the authors decided to use RegNets, then it is natural to use NF-SWS-RegNets are compared with the original RegNets without any big modifications as shown in the model details in the Appendix.\n\n7. Did the authors use trained ResNetV2-600 or randomly-initialized model?\n\n8. How does SPP goes on with the post-activation network which uses the original bottleneck block consists of Conv-BN-ReLU? \n\n9. How can the resolution downsampling block in a ResNet affect averaged channel mean and variance?\n\n10. Why the ResNet experiments are done with weight decay of 5e-5? The common knowledge of training ResNet is with weighing decay of 1e-4, so one may unconvinced the result because of the tuning.\n\n11. Comparing EfficientNets that are trained without CutMix and Mixup (but used randaug or autoaug) with the proposed models with cutmix and mixup seems to be not fair.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper analyses the signal propagation through residual architectures; then suggests a scaling method which, together with weight standardization, allows to train such networks to high accuracy with batch-norm; it demonstrates that the method performs better than previous methods (Fixup, SkipInit), and can be used on more advanced architectures. 

The reviewers initially had several concerns, but after the author's revision, these concerns were addressed and most reviewers recommended acceptance. One reviewer did not respond, but I think these concerns were addressed. I think it will help to further convince the readers on the usefulness of the method readers if the authors would check the sensitivity to the learning rate with the current method and compare with other methods (SkipInit, Fixup, BN). The reason I'm suggesting this is that I think one of the main reasons BN is still in popular use is that it commonly tends to make training more robust to changes in hyper-parameters, such as the learning rate (while other methods, like SkipInit and Fixup, require more hyper-parameter tuning).

Overall the analysis and the suggested method seem useful, especially at a small batch size and the writing is mostly clear, so I recommend acceptance. 


Decision: Accept (Poster)