Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs
Authors: Jonathan Frankle
David J. Schwab
Ari S. Morcos
Venue: ICLR 2021 Poster (2021)
Keywords: affine parameters
random features
batchnorm

Abstract: A wide variety of deep learning techniques from style transfer to multitask learning rely on training affine transformations of features. Most prominent among these is the popular feature normalization technique BatchNorm, which normalizes activations and then subsequently applies a learned affine transform. In this paper, we aim to understand the role and expressive power of affine parameters used to transform features in this way. To isolate the contribution of these parameters from that of the learned features they transform, we investigate the performance achieved when training only these parameters in BatchNorm and freezing all weights at their random initializations. Doing so leads to surprisingly high performance considering the significant limitations that this style of training imposes. For example, sufficiently deep ResNets reach 82% (CIFAR-10) and 32% (ImageNet, top-5) accuracy in this configuration, far higher than when training an equivalent number of randomly chosen parameters elsewhere in the network. BatchNorm achieves this performance in part by naturally learning to disable around a third of the random features. Not only do these results highlight the expressive power of affine parameters in deep learning, but - in a broader sense - they characterize the expressive power of neural networks constructed simply by shifting and rescaling random features.

Published as a conference paper at ICLR 2021
TRAINING BATCHNORM AND ONLY BATCHNORM: ON THE EXPRESSIVE POWER OF RANDOM FEATURES IN CNNS
Jonathan Frankle* MIT CSAIL
David J. Schwab CUNY Graduate Center, ITS Facebook AI Research ds @schwabe fb. com
Ari Morcos Facebook Al Research arimorcos@ fb. com
Frankleemit..es edu
ABSTRACT
A wide variety of deep learning techniques from style transfer to multitask learning rely on training affine transformations of features. Most prominent among these is the popular feature normalization technique BatchNorm, which normalizes activations and then subsequently applies learned affine transform. In this paper, aim understand the role and expressive power of affine parameters used to transform features in this way. To isolate the contribution of these parameters from that of the learned features they transform, when training only these parameters BatchNorm and freezing all weights at their random initializations. Doing so leads to surprisingly high performance considering the significant limitations that this style of training imposes. For example, sufficiently deep ResNets reach 82% (CIFAR-10 10) and 32% (ImageNet, top-5) accuracy in this configuration, far higher than when training an equivalent number of randomly chosen parameters elsewhere in the network. BatchNorm achieves this performance in part by naturally learning to disable around third of the random features. Not only do these results highlight the expressive power of affine parameters in deep learning, but-in broader sense- they characterize the expressive power of neural networks constructed simply by shifting and rescaling random features.
investigate the performance achieved
INTRODUCTION
Throughout the literature deep learning, wide variety of techniques rely learning affine transformations of features-mulin muutulls-muiiiiiinng each feature by a learned coefficient and adding a learned bias 3. This includes everything from multi-task learning (Mudrakarta al. 2019) style transfer and generation (e.g.. Dumoulin et al. 2017; Huang & Belongie 2017 Karras el al. 2019). One of the most common examples of these affine parameters are in feature normalization techniques like BatchNorm (loffe & Szegedy 2015). Considering their practical importance and their presence in nearly all modern neural networks, we know relatively little about the role and expressive power of affine parameters used to transform features in this way.
To gain insight into this question, we focus on the y and parameters in BatchNorm. BatchNorm is nearly ubiquitous in deep convolutional neural networks (CNNs) for computer vision, meaning these affine parameters are present by default numerous models that researchers and practitioners train every day. Computing BatchNorm proceeds in two steps during training (see Appendix A for full details). First, each pre-activation is normalized according to the mean and standard deviation across the mini- These normalized pre-activations are then scaled and shifted by trainable per-feature coefficient and bias 8.
One fact we do know about and /3 in BatchNorm is that their presence has meaningful effect the performance of ResNets, improving accuracy by 0.5% to 2% on CIFAR- (Kriheeskyy et al. 2009) and 2% on ImageNet (Deng et al., 2009) (Figure Dp. These improvements large enough
Work done while an intern and student researcher at Facebook Al Research. He al. (016)) find better accuracy when using BatchNorm before activation rather than after in ResNets.

---

Published as a conference paper at ICLR 2021
96
95
Top-1 Test Accurary (%)
78
04
74
Teat
70
90
14
32
58
1100
218
4.4
866
89
14-1
142
3444
14-5
14-16
14-32
18
34
50
101
200
CIFAR-10 Besttet
CIPAR-10 WRN
Imageleet Bestle:
All Parameters Trainable
All Parameters Trainable (y and & Disabled)
Figure Accuracy when training deep (left) and wide (center) ResNets for CIFAR-10 and deep ResNets for ImageNet (right) as described in Table [ when all parameters are trainable (blue) and all parameters except ? and 3 are trainable (purple). Training with y and / enabled results in accuracy 0.5% to 2% (CIFAR-10) and 2% (ImageNet) higher than with y and 3 disabled.
that, were y and / proposed as a new technique, it would likely see wide adoption. However, they are small enough that it difficult isolate the specific role y and 3 play these improvements.
More generally, the central challenge of scientifically investigating per-feature affine parameters is distinguishing their contribution from that of the features they transform. all practical contexts, these affine parameters are trained jointly with (as in the case of BatchNorm) or after the features themselves (Mudrakarta et al.) 2019 Dumoulin et al. 2017). In order to study these parameters in isolation, we instead train them on network composed entirely of random features. Concretely, freeze all weights at initialization and train only the yy and parameters in BatchNorm.
Although the networks still retain the same number of features, only small fraction of parameters (at most 0.6%) are trainable. This experiment forces all learning to take place in 7 and 3, making possible assess the expressive power of network whose only degree of freedom is scaling and shifting random features. We emphasize that our goal is scientific nature: to assess the performance and the mechanisms by which networks this limited capacity represent meaningful functions; we neither intend nor expect this experiment to reach SOTA accuracy. We make the following findings:
When training only and 3, sufficiently deep networks (e.g., ResNet-866 and ResNet-200) reach surprisingly high (although SOTASOTA) accuracy: 82% on CIFAR-10 10 and 32% top-5 on ImageNet. This demonstrates the expressive power of the affine BatchNorm parameters.
Training an equivalent number of randomly-selected parameters per channel performs far worse (56% on CIFAR-IO and 4% top-5 on ImageNet). This demonstrates that y and / have particularly significant expressive power per-feature coefficients and biases.
When training only BatchNorm, 7 naturally learns to disable between a quarter to half of all channels by converging to values close zero. This demonstrates that y and / achieve this accuracy in part by imposing per-feature sparsity.
When training all parameters, deeper and wider networks have smaller y values but few features are outright disabled. This hints the role ? may play in moderating activations in settings where disabling 7 and |3 leads to lower accuracy (the right parts of the plots in Figure D..
In summary, we find that 7 and have noteworthy expressive power in their own right and that this expressive power results from their particular position per-feature coefficient and bias. Beyond offering insights into affine parameters that transform features, this observation has broader implications for our understanding of neural networks composed of random features. By freezing all other parameters at initialization, we are training networks constructed by learning shifts and rescalings of random features. In this light, our results demonstrate that the random features available at initialization provide sufficient raw material to represent high-accuracy functions for image classification. Although prior work considers models with random features and a trainable linear output layer (e.g., Rahimi & Recht, 2009) Jaeger] 2003 Maass et al. 2202),, we reveal the expressive power of networks configured such that trainable affine parameters appear after each random feature.
RELATED WORK
BatchNorm. BatchNorm makes it possible to train deeper networks (He et al. 2015a) and causes SGD to converge sooner (Toffe & Szegedy 2015). However, the underlying mechanisms by which it
2

---

Published as a conference paper at ICLR 2021
does are debated. The original authors argue it reduces internal covariate shift (ICS), in which "the distribution of each layer's s inputs changes during training..rreuurriin lower learning rates" (Lofte) & Szegedy, 2013). (Santurarr al. (2018) cast doubt this explanation by artificially inducing ICS after BatchNorm with little change in training times. Empirical evidence suggests BatchNorm makes the optimization landscape smoother (Santurkar et al. (2018): is "safety precaution" against exploding activations that lead to divergence (Bjorck et al. 2018); and allows the network to better utilize neurons (Balduzzi al., 2017 Morcos et al. 2018). Theoretical results suggest Batch- Norm decouples optimization of weight magnitude and direction (Kohler et al. 2019) as weight normalization (Salimans Kingma 2016) does explicitly; that causes gradient magnitudes to reach equilibrium (Yang et 2019);; and that it leads to a novel form of regularization (Luo et al. 2019)
We focus on the role and expressive power of the affine parameters in particular, whereas the aforementioned work addresses the overall effect of BatchNorm on the optimization process. In service of this broader goal, related work generally emphasizes the normalization aspect of Batch- Norm, in some cases eliding one (Kohler et al., 2019) or both of r and 3 (Santurkar et al. 2018 Yang et al. 2019). Other work treats BatchNorm as a black-box without specific consideration for Y and B (Santurkar et al.) 2018 Bjorck et al. 2018, 8 Morcos et al. 2018 Balduzzi et al., 2017). One
notable exception is the work of Luo et al. (2019). who show theoretically that BatchNorm imposes a
? decay-a data-dependent 1.2 penalty on y; we discuss this work further in Section 5
Exploiting the expressive power of affine transformations. There are many techniques in the deep learning literature that exploit the expressive power of affine transformations of features. Mudrakartal et al. (2019) develop parameter-efficient approach to multi-task learning with shared network backbone (trained on a particular task) and separate sets of per-task BatchNorm parameters (trained on a different task while the backbone is frozen). Similarly, Rebuffi et al. (2017) allow a network backbone trained on one task adapt to others by way of residual models added onto the network comprising BatchNorm and convolution. Perezz et al. (017)) perform visual question-answering by using a RNN receiving text input to produce coefficients and biases that are used to transform the internal features of a CNN on visual data. Finally, work on neural style transfer and style generation uses affine transformations of normalized features to encode different styles Dumoulin et al. 2017
Huang& & Belongie 2017 Karras et al. 2019).
Training only BatchNorm. Closest to our work, Rosenfeld & Tsotsos (2019) explore freezing various parts of networks at initialization; in doing so, they briefly examine training only y and B. However, there are several important distinctions between this paper and our work. They conclude only that it is generally possible to "successfully train[] mostly-random networks," while we find that BatchNorm parameters have greater expressive power than other parameters (Figure green).
In fact, their experiments cannot make this distinction. They train only BatchNorm in just two CIFAR-10 networks (DenseNet and an unspecified Wide ResNet) for just ten epochs (vs. the standard 100+), reaching 61% and 30% accuracy. For comparable parameter-counts, we reach 80% and 70%. These differences meaningfully affect our conclusions: they allow us to determine that training only BatchNorm leads to demonstrably higher accuracy than training an equivalent number of randomly chosen parameters. The accuracy in Rosenteld & Tsotsos is too low to make any such distinction.
Moreover, we go much further in terms of both scale of experiments and depth of analysis. We study much wider range of networks and, critically, show that training only BatchNorm can achieve impressive results even for large-scale networks on ImageNet. We also investigate how the BatchNorm parameters achieve this performance by examining the underlying representations.
We also note that Mudrakarta et al. 2019) train only BatchNorm and a linear output layer on a single, randomly initialized MobileNet (in the context of doing so on many trained networks for the purpose of multi-task learning); they conclude simply that "it quite striking" that this configuration "can achieve non-trivial accuracy."
Random features. There is a long history of building models from random features. The perceptron Block 1962) learns linear combination of associators, each the inner product of the input and random vector. More recently, Rahimi & Recht (209)) showed theoretically and empirically that linear combinations of random features perform nearly as well as then-standard SVMs and Adaboost.
Reservoir computing (Schrauwen et al. 2007). also known as echo state networks (Jaeger (203) or liquid state machines (Maass et al.) 2002), learns linear readout from a randomly connected recurrent neural network; such models can learn useful functions of sequential data. To theoretically
3

---

Published as a conference paper at ICLR 2021
Family
ResNet for CIEAR-10
Wide ResNe: (WRN) for CIFAR-UU LO
ResNet for ImageNet
Depth
14
32
56
110
218
434
866
14
14
14
14
14
18
34
50
101
200
Width Seale
Total
175K
467K
856K
1.73M
3.4M
6,96M
14.0M]
175K
696K
2.78M
11.1M
44.5M
177M
11.7M
21.8M
25
6M
94.6
64.7M
BaschNorm
L12K
2,46K
4.26K
8.29K
16.4K
32.5K
64.7K]
1,1K
2.24
4,8K
8.96K
17.9K
35.8K
9.6K
17.0K
53.1K
105K
176K
Ouppr
650
650
650
650
650
650
650
650
1,29K
2,57K
5.13K
10.3K
20.5K
S13K
513K
2,05M
205M
2055
Shorscut
2,56K
2,56K
10.3K
685K
2.62M
172K
2.77M
BuschNorm
0.64%
0.53%
0,5015
0.48%
0.7%
0.40%
0.6498
0.32%
0.16%
0.08%
0.0%%
000%
0.00%%
0,08%6
0.20%
0.22%
Ougut
0.0WEE
0.04%E
0.00%
0.01%
0.09%
0.5%%
0.01%
4.39%
4.60%
3.17%
Shuricut
1.46%
0.55%
0.30%
0.15%
0.07% 0.04% 0.02%
1.45%
1.47%
L.47%
1.48%
1.8%%
1.8%%
1.47%
0.79%
10.83%
6.22% 4.28%
Table I: ResNet specifications and parameters in each part of the network. ResNets are called ResNet-D, where D is the depth. Wide ResNets are called WRN-D-W, where W the width scale. ResNet- 18 and 34 have a different block structure than deeper ImageNet ResNets (He et al. 2015a).
study SGD on overparameterized networks, recent work uses two layer models with the first layer wide enough that it changes little during training (c.g.. Du et al. 2019); in the limit, the first layer can be treated as frozen its random initialization (Jacot et al. 2018 8 Ychudai & Shamir 2019).
In all cases, these lines of work study models composed of a trainable linear layer on top of random nonlinear features. In contrast, our models have affine trainable parameters throughow the network after each random feature in each layer. Moreover, due to the practice of placing BatchNorm before the activation function (He et al. 2016), our affine parameters occur prior to the nonlinearity.
Freezing weights at random initialization. Neural networks are initialized randomly (He et al. 2015b Glorot & Bengio 2010), and performance with these weights is no better than chance However, is still possible to reach high accuracy while retaining some or all of these weights. Zhang al. (2019a) show that many individual layers in trained CNNs can be reset to their random i.i.d. initializations with little impact on accuracy. Zhou et al. (2019) and RRamanujan et al. (2019) reach high accuracy on CIFAR-10 10 and ImageNet merely by learning which individual weights to remove.
METHODOLOGY
ResNet architectures. We train convolutional networks with residual connections (ResNets) on CIFAR-10 and ImageNet. We focus ResNets because they make it possible to add features by arbitrarily (a) increasing depth without interfering with optimization and (b) increasing width without parameter-counts becoming large that training is infeasible. Training deep ResNets generally requires BatchNorm, so it is a natural setting for our experiments. In Appendix C
we run the same experiments for non-residual VGG-style network for CIFAR-10, 10, finding qualitatively similar results.
We use the ResNets for CIFAR-10 10 and ImageNet designed by He et al. 201 F We scale depth according to He et al. 2013a) and scale width by multiplicatively increasing the channels per layer. As depth increases, networks maintain the same number of shortcut and output parameters, but deeper networks have more features and, therefore, more BatchNorm parameters. As width increases, the number of BatchNorm and output parameters increases linearly, and the number of convolutional and shortcut parameters increase quadratically because the number of incoming and outgoing channels both increase. The architectures we use are summarized in Table 1 (full details in Appendix B).
BatchNorm. We place BatchNorm before activation, which He al. (2016) find leads to better performance than placing it after activation. We initialize 3 to 0 and sample 7 uniformly between 0 and 1, although we consider other initializations Appendix D
Replicates. All experiments are shown as the mean across five (CIFAR-10) three (ImageNet) runs with different initializations, data orders, and augmentation. Error bars for one standard deviation from the mean are present in all plots; in many cases, error bars are too small to be visible.
"CIFAR-10 and ImageNet ResNets different architecture families with different widths and block designs.
4

---

Published as a conference paper at ICLR 2021
100
Test Acctucc (8)
70
70
60
60
50
50
40
14
32
56
110
218
434
886
144
142
14-4
14-8
14-16
14:32
GFAR-10 ResNet
CIFAR-10 WRN
100
Top-S Test-ccuro 89
100
90
90
80
80
20
20
10
50
ImageNet RecNet
ImageNet Rectet
Params Trainable
BatchNorm
Random Params Per Channel
BatchNorm Output
Output
Chance
Figure 2: Accuracy of ResNets for CIFAR-I0 (top left, deep; top right, wide) and ImageNet (bottom left, top-1 accuracy; bottom right, top-5 accuracy) with different sets of parameters trainable.
4 TRAINING ONLY BATCHNORM
In this section, we study freezing all other weights initialization and train only y and B. These parameters comprise no more than 0.64% all parameters in networks for CIFAR-10 and 0.27% % in networks for ImageNet. Figure shows the accuracy when training only y and in red for the families of ResNets described in Table I We also include two baselines: training all parameters (i.e., training normally) in blue and chance performance (i.e., random guessing on the test set) in gray.
Case study: ResNet-110. We first consider ResNet- 10 on CIFAR-IO. When all 1.7M parameters are trainable (blue), the network reaches 93.3% test accuracy. Since CIFAR-10 has ten classes, chance performance is 10%. When training just the 8.3K (0.48% afline parameters that can only shift and rescale random features, the network achieves surprisingly high test accuracy of 69.5%, suggesting that these parameters have noteworthy representational capacity.
While our motivation is study the role of the affine parameters, this result also has implications for the expressive power of neural networks composed of random features. All of the features in the network (i.e., convolutions and linear output layer) are fixed random initializations; the affine parameters can only shift and scale the normalized activation maps that these features produce in each layer. In other words, this experiment can be seen as training neural networks parameterized by shifts and rescalings of random features. In this light, these results show that it is possible to reach high accuracy on CIFAR-10 10 using only the random features that were available at initialization.
Increasing available features by varying depth and width. From the lens of random features, the expressivity of the network will be limited by the number of features available for the affine parameters combine. If we increase the number of features, expect that accuracy will improve. We can do so two ways: increasing the network's 's depth or increasing its width.
Figure 2 presents the test accuracy when increasing the depth (top left) and width (top right) of CIFAR- 10 ResNets and the depth of ImageNet ResNets (bottom). As expected, the accuracy of training only BatchNorm improves we deepen or widen the network. ResNet-14 achieves 48% accuracy on CIFAR-10 10 when training only BatchNorm, but deepening the network to 866 layers or widening it by factor of 32 increases accuracy 82% and 73%. Snrectively.Similaaa 55
achieves 17% top-5 accuracy on ImageNet, but deepening to 200 layers increases accuracy to 32%, [
is possible that, since ImageNet has 1000 classes, accuracy artificially constrained when freezing the linear output layer because the network cannot learn fine-graines distinctions between classes. To examine this possibility, we made the 0.5M to M output parameters trainable (Figure 2 purple).
Training the output layer alongside BatchNorm improves top-5 accuracy by about 25 percentage points to a maximum value of 57% and top-1 accuracy by 12 to 19 percentage points to a maximum value of 32%. The affine parameters essential for this performance: training outputs alone
In Appendix E we find that changing the BatchNorm initialization improves accuracy by a further 2-3 percentage points (CIFAR-10) and five percentage points (ImageNet top-5).
5

---

Published as a conference paper at ICLR 2021
85
Incressing Depth
Test Axcurary (N
Incressing withh
65
55
500e #lw09966 p=2.885:05
0.5
1
16
32
64
Thousands of BatchNorm Parameters llo scale!
Figure 3: The relationship between BatchNorm parameter count and accuracy when scaling depth and width of CIFAR-10 ResNets.
Distribution ResNes CIFAR-10
Distr bution of Restict 101 for ImageMet
18
15
Al Parame Trainahl
Dordtr
BateWWem
1.2
9,.
90.5
Figure 4: The distribution of for ResNet-110 and ResNet-100 aggregated from five CIFAR-10) or three replicates (ImageNet). Distributions of and 3 for networks are Appendix H
achieves just 2.7% top-5 and 0.8% top-1 accuracy for ResNet-200 (yellow). The same modification makes little difference on CIFAR-10 10, which has only ten classes.
Finally, note that accuracy is percentage points higher for ResNet-434 than for WRN- 14-32 although both have similar numbers of BatchNorm parameters (32.5K vs. 35.8K). This raises further question: for a fixed budget of BatchNorm parameters (and, thereby, a fixed number of random features), performance always better when increasing depth rather than increasing width? Figure plots the relationship between number of BatchNorm parameters (x-axis) and test accuracy on CIFAR-10 (y-axis) when increasing depth (blue) and width (red) from the common starting point of ResNet- 14. In both cases, accuracy increases linearly as BatchNorm parameter count doubles. The trend is 18% steeper when increasing depth than width, meaning that, for the networks consider, increasing depth leads to higher accuracy than increasing width for a fixed BatchNorm parameter budget
Are affine parameters special? Is the accuracy of training only BatchNorm a product of the unusual position of y and B as scaling and shifting entire features, or is it simply due to the fact that, in aggregate, substantial number of parameters are still trainable? For example, the 65K BatchNorm parameters in ResNer-866 are a third of the 175K parameters in all of ResNet-14; perhaps any arbitrary collection of this many parameters would lead to equally high accuracy.
To assess this possibility, we train two random parameters in each convolutional channel as substitutes for y and (Figure green) Should accuracy match that of training only BatchNorm, would suggest our observations are not unique to and 9 and simply describe training an arbitrary subset of parameters as suggested by [Rosenfeld
to percentage points lower CIFAR-I0 10 and never exceeds 4% top-5 on ImageNet. This result suggests that y and have greater impact on accuracy than other kinds of parameters 6 In other words, it appears more important to have coarse-graines control over entire random features than to learn small axis-aligned modifications of the features themselves.
T'sossos 2019). Instead, accuracy
Summary. Our goal was to study the role and expressive power of the affine parameters y and in -stthoo the presence of trained features to transform. We found that training only these parameters in ResNets with BatchNorm leads to surprisingly high accuracy (albeit lower than training all parameters). By increasing the quantity of these parameters and the random features they
Wee expect accuracy will eventually saturate and further expansion will have diminishing returns. We begin saturation for ResNet-866. which is below the regression line.
'ee also tried distributing these parameters randomly throughout the layer. Accuracy was the same or lower. F find that it is necessary to train between 8 and 16 random parameters per channel the CIFAR-I0 10 ResNets to match the performance training only the 2 afline parameters per channel.
'In Appendix
'In Appendix D we investigate whether it is better to have small number of dense trainable features or to learn to scale and shift large number of random features. To do so, compare the performance of training only BatchNorm training all parameters ResNets with an similar number of trainable parameters.

---

Published as a conference paper at ICLR 2021
70
60
of (2C
40
Frectoon of
Tractin
30
10
20
20
10
130
218
434
856
1.41
14:2
144
1648
14-16
14-12
34
50
101
200
CIFAR-10 Resblet
CIFA-110 WAN
Imagellet Reshit
All Parame Trinnblc, 0-001
1-0.5
(-6.3
BatchNerm.
0.00
(-
0.05
2-0.1
Figure 5: Fraction of y parameters for which |y| is smaller than various thresholds.
Change Top Accuracy PM J 5
Chargein Accuracy 060
1
L
Change
218 434
144
14.8 CIFAR-10 WAN
14
CIFAR Restitt
ImageNet Restict
All Params Trainable, 00.01
bn 00.05
|v
BatchNorm, [v] 0.01
IM <.0.5
Figure 6: Accuracy change when clamping Y values with hy| below various thresholds 0.
combine, we found that we can further improve this accuracy. This accuracy is not simply due to the raw number of trainable parameters, suggesting that y and 3 have partieular expressive power per-feature coefficient and bias.
5 EXAMINING THE VALUES OF AND B
In the previous section, we showed that training just and 3 leads to surprisingly high accuracy. Considering the severe restrictions placed on the network by freezing all features their random initializations, we are interested in how the network achieves this performance. In what ways do the values and role ry and /3 change between this training regime and when all parameters are trainable?
Examining As an initial case study, we plot the y values learned by ResNet- 10 for CIFAR- 10 and ResNet-100 101 for ImageNet when all parameters are trainable (blue) and when only y and are trainable (red) in Figure (distributions for all networks and for 33 are in Appendix H.. When training all parameters, the distribution of y for ResNet-1 110 roughly normal with a mean of 0.27; the standard deviation of 0.21 is such that 95% of values are positive. When training only BatchNorm, the distribution of y has a similar mean (0.20) but a much wider standard deviation (0.48), meaning that 25% of y values negative. For ResNet- 101, the mean value of y similarly drops from 0.14 to 0.05 and the standard deviation increases from 0.14 to 0.26 when training only BatchNorm.
Most notably, the BatchNorm-onke y values have spike at 0: 27% (ResNet-1 110) and 33% (ResNet- 101) of all y values have a magnitude < 0.01 (compared with 4% and 5% when training all parameters). By setting y close zero, the network seemingly learns to disable between quarter and third of all features. Other than standard weight decay for these architectures, we take no additional steps induce this sparsity; occurs naturally when we train this fashion. This behavior indicates that an important part of the network's ' representation is the set of random features that it learns to ignore. When all parameters are trainable, there is much smaller spike 0, suggesting that disabling features is a natural behavior of y, although it is exaggerated when only and | are trainable.
The same behavior holds across all depths and widths, seemingly disabling a large fraction of features. When training only BatchNorm, | < 0.01 in between quarter and a third of cases (Figure 5 green). In contrast, when all parameters are trainable (Figure blue), this occurs for just 5% values in all but the deepest ResNets. Values of y tend to be smaller for deeper and wider networks, especially when all parameters are trainable. For example, the fraction of In| < 0.05 increases from 3% for Re to 48% for ResNet-866. We hypothesize that y values become smaller to prevent exploding activations: this might explain why disabling y and 3 particularly hurts the accuracy of deeper and wider CIFAR- 10 networks in Figure
Small values of y disable features. Just because values of y are close to zero does not necessarily mean they disable features and can be set equal to zero; they may still play important role in the
7

---

Published as a conference paper at ICLR 2021
45
% Acduvaations Disates
40
Activatorn Diaables
25
35
25
30
20
15
13
218
434
866
1141
24.4
14-5 14:16 WAN
CIFAR ResNet
CIFAR
Imagelctt ResNet
Trainable
BetchNorm
Figure 7: The fraction of ReLU activations for which Pr Pr/activation = 0] > 0.99.
representation. To evaluate the extent to which small values of y are, in fact, removing features, we explicitly set these parameters zero and measure the accuracy of the network that results (Figure Clamping all values of |n| < 0.01 to zero does not affect accuracy, suggesting that these features are indeed expendable. This is true both when all parameters are trainable and when only BatchNorm is trainable; in the latter case, this means between 24% to 38% of features can be disabled. This confirms our hypothesis that values closest zero reflect features that are irrelevant the network's representation. Results are mixed for a threshold of 0.05: when all parameters are trainable, accuracy remains close its full value for all but the deepest and widest networks (where we saw spike in the fraction of parameters below this threshold).
Training only BatchNorm sparsifies activations. So far, we have focused solely on the role of y. However, works collaboratively with 3 to change the distribution of normalized are-activations. It challenging to describe the effect of 3 due to its additive role; for example, while many values are also close to zero when training only BatchNorm (Appendix HH). this does not necessarily disable features. To understand the joint role of y and B3, we study the behavior of the activations themselves. In Figure 7 we plot the fraction of ReLU activations for which Prlactivation = 0j > 0.99 across all test examples and all pixels in the corresponding activation maps When training only BatchNorm, 28% to 39% of activations In contrast, when all parameters are trainable, no more than 10% (CIFAR-10) and 2% (ImageNet) of activations are disabled according this heuristic. These results support our hypothesis of the different roles that small values of 7 play in these two training settings. When only training BatchNorm, we see small values of and entire activations disabled. However, when all parameters are trainable, few activations are disabled even though large fraction of y values are small in deeper and wider networks, suggesting that these parameters still play role in the learned representations.
disabled, meaning Y and | indeed sparsify activations in practice.
Context in the literature. Mehta et al. 2019) also find that feature-level sparsity emerges in CNNs when trained with certain combinations of optimizers and regularization. They measure this sparsity in manner similar to ours: the fer-feature activations and y values. Mehta et al. only study standard training (not training only BatchNorm). In this context, they find much higher levels of sparsity (sometimes higher than 50%) than we do (less than 5% in all cases we consider), despite the fact that our threshold for y to represent a pruned feature is 10x higher (i.e., we sparsify more features). This may indicate that the behaviors Mehta et al. observe in their "BasicNet" setting (a 7-layer ConvNet for CIFAR) and VGG setting (it appears they use the ImageNet version of VGG, 500x larger than ResNet-20 and very overparameterized for CIFAR- 10) may not hold in general.
In their theoretical analysis of BatchNorm, Luo et al. (2019) find that one of its effects is y decay a data-dependent 1.2 penalty the 7 terms in BatchNorm. This observation may provide insight into why we find per-feature sparsity when training only BatchNorm in Section (5) by freezing all other parameters at their initial values, it is possible that our networks behave according to Luo et al.ss assumptions and that decay may somehow encourage sparsity (although it is not subject to sparsity-inducing LI regularization).
Summary. In this section, we compared the internal representations of the networks when training all parameters and training only the affine parameters 7 and in BatchNorm. When training only BatchNorm, we found 7 to have a larger variance and a spike at 0 and that ? was learning to disable entire features. When all parameters were trainable, we found that y values became smaller in wider
'For some features, this is also the fraction of batch normmloorallized pre-activations that are S 0 (i.e., that will he eliminated by the ReI.U). However, the end of residual block, the hatch-normalized pre-activations are added to the skip connection before the ReI.U, even if 7 = 0, the activation may be non-zero.
8

---

Published as a conference paper at ICLR 2021
and deeper networks but activations were not disabled, which implies that these parameters still play a role in these networks.
6 DISCUSSION AND CONCLUSIONS
Our results demonstrate that it is possible to reach surprisingly high accuracy when training only the affine parameters associated with BatchNorm and freezing all other parameters their original initializations. To answer our research question, we conclude that affine parameters that transform features have substantial expressive power their own right, even when they are not paired with learned features. We make several observations about the implications of these results.
BatchNorm. Although the research community typically focuses on the normalization aspect of BatchNorm. our results emphasize that the affine parameters are remarkable in their own right. Their presence tangibly improves performance, especially in deeper and wider networks (Figure [. behavior we connect to our observation that values of y are smaller as the networks become deeper. On their own, y and | create surprisingly high-accurace networks, even compared to training other subsets of parameters, despite (or perhaps due to) the fact that they disable more than a quarter of activations.
Random features. From different perspective, experiment is novel way of training networks constructed out of random features. While prior work (e.g.. Rahimi & Recht] 2009) considers training only linear output layer top of random nonlinear features, we distribute affine parameters throughout the network after each feature in each layer. This configuration appears to give the network greater expressive power than training the output layer alone (Figure (2. Empirically, we see our results as further evidence (alongside the work of et al. (2019) and Ramanujan et al. (2019)) that the raw material present random initialization sufficient create performant networks. would also be interesting to better understand the theoretical capabilities of our configuration.
Unlike Rahimi & Recht our method does not provide practical reduction in training costs; it is still necessary to fully backpropagate to update the deep BatchNorm parameters. However, our work could provide opportunities to reduce the cost of storing networks at inference-time. rather than needing to store all of the parameters in the network, we could store the random seed necessary to generate the network's 's weights and the trained BatchNorm parameters, We could even do so in multi-task fashion similar to Mudrakarta et al. (2019). storing a single random seed and multiple sets of BatchNorm parameters for different tasks. Finally. if we are indeed able to develop initialization schemes that produce random features that lead higher accuracy on specific tasks, could store a per-task distribution and random seed alongside the per-task BatchNorm parameters.
particular,
Limitations and future work. There are several ways to expand study to improve the confidence and generality of our results. We only consider ResNets trained on CIFAR-10 and ImageNet, and it would be valuable to consider other architecture families and tasks (e.g.. Inception computer vision and Transformer on NLP). In addition, we use standard hyperparameters and do not search for hyperparameters that specifically perform well when training only BatchNorm.
In follow-up work, we are interested in further studying the relationship between random features and the representations learned by the affine parameters. Are there initialization schemes for the convolutional layers that allow training only the affine parameters to reach better performance than using conventional initializations? (See Appendix P for initial experiments on this topic.) possible to rejuvenate convolutional filters that are eliminated by (in manner similar to Co[Cohen al. (2016)) to improve the overall accuracy of the network? Finally, can we better understand the role of these affine parameters outside the context of BatchNorm? That is, can distinguish the expressive power of the affine parameters from the normalization itself? For example, we could add these parameters when using techniques that train deep networks without normalization, such as WeightNorm (Salimans & Kingma 2016) and FixUp initialization (Zhang et al. 2019b).
9

---

Published as a conference paper at ICLR 2021
REFERENCES
David Balduzzi, Marcus Frean, Lennox Leary, JP Lewis, Kurt Wan-Duo Ma, and Brian McWilliams. The shattered gradients problem: If resnets are the answer, then what the question? In Proceedings of the 34th International Conference on Machine Learning-Vor Volume 70, pp. 342-350.
JMLR. org, 2017.
Nils Bjorek, Carla Gomes, Bart Selman, and Kilian Q Weinherger. Understanding batch normalization. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 7694-7705. Curran Associates, Inc., 2018. URL http:/ httt::/ppapers wwappers..iii.oo wwwp.cc/paaa...eeemmee 7996 understanding batch norma Lizztioon.oe pdf
Hans-Dieter Block. The perceptron: A model for brain functioning, i. Reviews of Modern Physics, 34(1):123, 1962.
Joseph Paul Cohen, Henry Z. Lo, and Wei Ding. Randomout: Using convolutional gradient norm win the filter lottery. CoRR, abs/ abs/1602.05931, 2016. URL
05931
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 245-255. leee, 2009.
Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. 2019.
Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. A learned representation for artistic style. In International Conference on Learning Representations, 2017.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference artificial intelligence and statistics, pp. 249-256, 2010.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2015a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645. Springer, 2016.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization. In Proceedings of the IEEE International Conference Computer Vision, pp. 1501-1510, 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32Nd International Conference on International Conference on Machine Learning Volume 37, ICML 15, pp. 448-456. JMLR.org, 3045167
2015. bttp://t acm. http://ciaatoo...eeeeeeeee...eee
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in neural information processing systems, pp. 8571-8580, 2018
Herbert Jaeger. Adaptive nonlinear system identification with echo state networks. In Advances in neural information processing systems, pp. 609-616, 2003.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
10

---

Published as a conference paper at ICLR 2021
Jonas Kohler, Hadi Daneshmand, Aurelien Lucchi, Thomas Hofmann, Ming Zhou, and Klaus Neymeyr. Exponential convergence rates for batch normalization: The power of length- length-direction decoupling in non-convex optimization. In Kamalika Chaudhuri and Masashi Sugiyama (eds.), Proceedings of Machine Learning Research, volume 89 of Proceedings of Machine Learning Research, pp. 806-815, PMLR, 16-18 Apr 2019. URL httphhttppppcceeeeeeee ngs. ml press/ wa9/Kohler19a, 1ttm
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In International Conference on Learning Representations, 2019. URL
Ping Luo, Xinjiang Wang, Wenqi Shao, and Zhanglin Peng. Towards understanding regularization in batch normalization. In International Conference on Learning Representations, 2019. URL
net/ Horummid
Wolfgang Maass, Thomas Natschläger, and Henry Markram. Real-time computing without stable states: A new framework for neural computation based on perturbations. Neural computation, 14 (11):2531-2560, 2002.
Dushyant Mehta, Kwang In Kim, and Christian Theobalt. On implicit filter level sparsity in convolu- tional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 520-528, 2019.
Ari Morcos, David GT Barrett, Neil C Rabinowitz, and Matthew Botvinick. On the importance of single directions for generalization. In Proceeding of the International Conference on Learning Representations, 2018
Pramod Kaushik Mudrakarta, Mark Sandler, Andrey Zhmoginov, and Andrew Howard. K for the price of Parameter efficient multi-task and transfer learning. In International Conference on Learning Representations, 2019, httttttts://peeetet
eveiw net. meee://oorumm2dd=JJXXe
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. arXiv preprint 170v00170..0787., 2017.
Ali Rahimi and Benjamin Recht. Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning. In Advances in neural information processing systems, pp. 1313- 1320, 2009.
Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Rastegari. What's hidden in randomly weighted neural network?, 2019.
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30, pp. 506- 516. Curran Associates, Inc., 2017. URL httpttps//prooeenggggg www.nnneu.... cc/paper/ Paper..pepppp.p.
Amir Rosenfeld and John K Tsotsos. Intriguing properties of randomly weighted networks: Gener- alizing while learning next to nothing. In 2019 16th Conference on Computer and Robot Vision (CRV), pp. 9-16. IEEE, 2019.
Tim Salimans and Durk Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In Advances in neural information processing systems, pp. 901-909, 2016.
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch normalization help optimization? In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Desa-Bianchh, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 2483-2493. Curran Associates, Inc 2018. | http:://tpppeeeett hwtp://appess..// hwtppppwwwsssps...p.pppppppppaap cttpps.cc//aee ctt.ppssccppppr// / 7515-howe does bww.haattchtae zat ion he opt zat wwt.ii..aano..........o pd

---

Published as a conference paper at ICLR 2021
Benjamin Schrauwen, David Verstraeten, and Jan Van Campenhout. An overview of reservoir computing: theory, applications and implementations. In Proceedings of the 15th european symposium artificial neural networks. p. 471-487 2007, pp. 471-482,2000 2007.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409, 1556, 2014.
Greg Yang, Jeffrey Pennington, Vinay Rao, Jascha Sohl-Dickstein, and Samuel S. Schoenholz. A mean field theory batch normalization. International Conference on Learning Representations, 2019.
net/ http://wwww.....oommmoaaaaaaa/aa
Gilad Yehudai and Ohad Shamir. On the power and limitations of random features for understanding neural networks. Advances in Neural Information Processing Systems, pp.
2019.
Chiyuan Zhang, Samy Bengio, and Yoram Singer. Are all layers created equal? 2019a.
Hongyi Zhang, Yann N. Dauphin, and Tengyu Ma. Residual learning without normalization via better initialization. In International Conference on Learning Representations, 2019b. URL (ooormmmii-1gs3333000 -
http::::///peeeeeee ew net/
Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing lottery tickets: Zeros, signs, and the supermask. In Advances in Neural Information Processing Systems, 2019.
12

---

Published as a conference paper at ICLR 2021
TABLE OF CONTENTS FOR SUPPLEMENTARY MATERIAL
In these appendices, we include additional details about our experiments, additional data that did not fit in the main body of the paper, and additional experiments. The appendices are as follows:
Appendix A. formal re-statement of the standard BatchNorm algorithm.
Appendix B. The details of the ResNet architectures and training hyperparameters
Appendix C. The experiments from the main body of this paper performed on the VGG family of architectures for CIFAR-10, which do not have residual connections. The results match those described in the main body.
Appendix D. Comparing the performance of training only BatchNorm to training small ResNets with an equivalent number of trainable parameters.
Appendix E. The effect of varying the initializations of both the random features and the BatchNorm parameters. We find that initializing 3 to improves the performance of training only BatchNorm.
Appendix F. Further experiments on training a random number of parameters per channel: (I) determining the number of random parameters necessary to reach the same performance as training and | and (2) training random parameters and the output layer.
Appendix G. Examining the role of making shortcut parameters trainable.
Appendix H. The distributions of y and for all networks presented in PectionSSoor ResNet-110 and 10sNet-101.
Appendix I. Verifying that values of y that close zero be set 0 without affecting accuracy, meaning these features are not important to the learned representations.
Appendix J. The frequency with which activations are disabled for all ResNets that we study (corresponding to the activation experiments in Section
13

---

Published as a conference paper at ICLR 2021
A FORMAL RESTATEMENT OF BATCHNORM
The following is the batch normalization algorithm proposed by loffe & Szegedy (2015).
1. Let a0)), : 1.- (n) be the pre-activations for a particular unit a neural network for inputs through in mini-batch.
2. Let μ1 = - x0)
3. Let o² - 11)²
4. The batch-normalized pre-activation .00) 7 20
V +
where y and are trainable
parameters.
5. The activations are foeef(at ((((t) where f is the activation function.
B DETAILS OF RESNETS
B.1 CIFAR-10 RESNETS
ResNet architectures. We use the ResNets for CIFAR-10 described by He et al. (2015a). Each network has an initial 3x3 convolutional layer from the three input channels to 16W channels (where W is the width scaling factor). Afterwards, the network contains 3N residual blocks.
Each block has two 3x3 convolutional layers surrounded by shortcut connection with the identity function and trainable parameters. The first of N blocks have 16W filters, the second set of N blocks have 32W filters, and the third set of N blocks have 64W filters. The first layer in each set blocks downsamples by using stride of 2; the corresponding shortcut connection has 1x1 convolution that also downsamples. If there is a 1x1 convolution on the shortcut connection, it undergoes BatchNorm separately from the second convolutional layer of the block; the values are added together after normalization.
After the convolutions, each remaining channel undergoes average pooling and a fully-connected layer that produces ten output logits. Each convolutional layer is followed by batch normalization before the activation function is applicd (He et al., 2016))
The depth is computed by counting the initial convolutional layer (1), the final output layer (1), and the layers in the residual blocks (3N blocks layers per block). For example, when N = 5, there are 32 layers (the initial convolutional layer, the final output layer, and 30 layers in the blocks).
When the width scaling factor 1, we refer to the network as ResNet-depth, e.g., ResNet-32.
When the width scaling factor W 1, refer the network as WRN-depth-W,ec c.g., WRN-14-4
Hy plypepparameters We initialize all networks using He normal initialization (He et al. 2015b), although we experiment with other initializations in Appendix The 7 parameters of BatchNorm are sampled uniformly from [0, and the 3 parameters are set to O. We train for 160 epochs with SGD with momentum (0.9) and batch size of 128. The initial learning rate is 0.1 and drops by 10x at epochs 80 and 120. We perform data augmentation by normalizing per-pixel, randomly flipping horizontally, and randomly translating by up to four pixels in each direction. We standard weight decay of le-4 on all parameters, including BatchNorm.
B.2 IMAGENET RESNETS
ResNet architectures. We use the ResNets for ImageNet as described by He al. (2015a). Each model has an initial 7x7 convolutional layer from three input channels to three output channels. Afterwards, there is 3x3 max-pooling layer with stride of 2.
Afterwards, the network has four groups of blocks with N1. NN.. Ny. and N₂ blocks in each group. The first group of blocks has convolutions with 64 channels, the second group of blocks has convolutions with 128 channels, the third group of blocks has convolutions with 256 channels, and the fourth group of blocks has convolutions with 512 channels.
14

---

Published as a conference paper at ICLR 2021
After the convolutions, each channel undergoes average pooling and fully- cully-connected layer that produces one thousand output logits. Each convolutional layer is followed by batch normalization before the activation function is applied (He et al., 2016).
The structure of the blocks differs; ResNet-18 and ResNet-34 have one block structure (a basic block) and ResNet-50, ResNet- 101, and ResNet-200 have another block structure (a bottleneck block). These different block structures mean that ResNet-18 18 and ResNet-34 have a different number of output and shorteut parameters than the other ResNets. The basic block is identical to the block in the CIFAR-I0 ResNets: two 3x3 convolutional layers (each followed by BatchNorm and a ReLU activation). The bottleneck block comprises a 1xl convolution. a 3x3 convolution, and final 1x1 convolution that increases the number of channels by 4x the first Ixl convolution in the next block decreases the number of channels by 4x back to the original value. In both cases, if the block downsamples the number of filters, it does so by using stride 2 on the first 3x3 convolutional layer and adding a 1x1 convolutional layer to the skip connection.
The depth is computed just with the CIFAR-10 ResNets: by counting the initial convolutional layer (I), the final output layer (I), and the layers in the residual block. We refer to the network as ResNet-depth, e.g.. ResNet-50.
The table below specifies the values of N,, N2. N3, and N4 for each of the ResNets we use. These the same values specified by He et al. 2015a).
Name
N1
N2
N3
N4
ResNet- 18
2
2
2
2
ResNet-34
3
4
6
3
ResNet-50
3
4
6
3
ResNet-101
3
4
23
3
ResNet-220
3
24
36
3
Hyperparameters. We initialize all networks using He normal initialization (He et al. 2015b). The Y parameters of BatchNorm are sampled uniformly from [o, 1] and the / parameters are set to 0. We train for 90 epochs with SGD with momentum (0.9) and batch size of 1024. The initial learning rate 0.4 and drops by 10x at epochs 30, 60, and 80. The learning rate linearly warms up from to 0.4 over the first 5 epochs. We perform data augmentation by normalizing per-pixel, randomly flipping horizontally, and randomly selecting a crop of the image with a scale between 0.1 and 1.0 and an aspect ratio of between 0.8 and 1.25. After this augmentation, the image is resized to 224x224. We use standard weight decay of le-4 on all parameters, including BatchNorm.
C RESULTS FOR VGG ARCHITECTURE
In this Section, we repeat the major experiments from the main body of the paper for VGG-style neural networks (Simonyan & Zisserman 2014) for CIFAR-10. The particular networks
adapted for CIFAR-TO by Liu et al. (2019). ). The distinguishing quality of these networks is that they do not have residual connections, meaning they provide a different style of architecture in which to explore the role of the BatchNorm parameters and the performance of training only BatchNorm.
C.1 ARCHITECTURE AND HYPERPARAMETERS
Architecture. We consider four VGG networks: VGG-11, VGG-13, VGG-16. and VGG-19. Each of these networks consists of a succession 3x3 convolutional layers (each followed by BatchNorm) and max-pooling layers with stride that downsample the activation maps. After some max-pooling layers, the number of channels per layer sometimes doubles. After the final layer, the channels are combined using average pooling and linear output layer produces ten logits.
The specific configuration of each network is below. The numbers are the number of channels per layer, and M represents max-pooling layer with stride 2.
15

---

Published as a conference paper at ICLR 2021
Name
Configuration
VGG-11
64, M.. 128, M. 256, 256, M, 512, 512, M, 512.
VGG-13
64, 64, M, 128, 128, M, 256, 256, M, 512, 512, 512,
VGG-16
64, 64, M, 128, 128, M, 256, 256, 256, M. 512, 512, 512, 512, 512, 512
VGG-19
64, 64, M, 128, 128, M. 256, 256, 256, 256, M. 512, 512, 512, 512, M, 512, 512, 512, 512
Hyperparameters. We initialize all networks using He normal initialization. The yy parameters of BatchNorm sampled uniformly from [0, 1] and the 3 parameters set to O. We train for 160 epochs with SGD with momentum (0.9) and batch size of 128. The initial learning rate is 0.1 and drops by 10x at epochs 80 and 120. We perform data augmentation by normalizing per-pixel, randomly flipping horizontally, and randomly translating by up to four pixels in each direction. We use standard weight decay of le-4 on all parameters, including BatchNorm.
Parameter-count Below are the number of parameters in the entire network, the BatchNorm layers. and the output layer. This table corresponds to Table
Family
VGG for CIFAR-I0
Depth
11
16
19
Width Scale
1
Total
9.23M
9.42M
14.73M
20.04M
BatchNorm
5.5K
5.89K
8.45K
11.01K
Output
5.13K
5.13K
5.13K
5.13K
BatchNorm
0.06%
0.06%
0.06%
0.05%
Output
0.06%
0.05%
0.03%
0.03%
C.2 RESULTS
In this subsection, we compare the behavior of the VGG family of networks to the major results in the main body of the paper. Unlike the ResNets, disabling 7 and has no effect on performance of the VGG networks when all other parameters are trainable (Figure corresponding to Figure I the main body).
When training only BatchNorm (Figure 9 corresponding to Figure 2 in the main body), the VGG networks reach between 57% (VGG-11) and 61% (VGG-19) accuracy, lower than full performance (92% to 93%) but higher than chance (10%). Since these architectures do not have skip connections, we are not able to explore as wide range of depths as we do with the ResNets. However we do see that increasing the number of available features by increasing depth results in higher accuracy when training only BatchNorm. Making the output layer trainable improves performance by more than percentage points for VGG-11 1 but less than 2 percentage points for VGG- 16 and VGG-19, the same limited improvements as in the CIFAR-10 10 ResNets. Training the output layer alone performs far worse: 41% % accuracy for VGG-11 dropping down to 27% for VGG-19.
The performance of training only BatchNorm remains far higher than training two random parameters per channel. Accuracy is higher by between 13 and 14 percentage points, again suggesting that y and 3 have particular expressive power as per-channel coefficient and bias.
As we observed Section for the ResNets, training only BatchNorm results in a large number ? parameters close to zero (Figure 21). Nearly half (44% for VGG-11 and 48% for VGG-19) of all parameters have magnitudes less than 0.01 when training only BatchNorm as compared between 2% and 3% when all parameters are trainable.
D COMPARING TRAINING ONLY BATCHNORM TO SMALL RESNETS
As another baseline to contextualize our findings, we compare training only BatchNorm to training all parameters in small ResNets (Figure 10). This experiment assesses whether, for fixed budget of trainable parameters, it is better train large number of random features small number of learnable features. To make this comparison, we perform grid search over depths and widths to find ResNets whose total parameter-counts are similar the BatchNorm parameter-counts in
16

---

Published as a conference paper at ICLR 2021
networks. The small ResNets indeed outperform training only BatchNorm, at best by 5 to 10 percentage points | outperforms shifting and scaling random features with BatchNorm. (We emphasize that intend this experiment only to contextualize our earlier findings; our goal is to study the role and expressive power of y and B, not find the most performant way allocate fixed parameter budget.)
In other words, for a fixed trainable parameter budget, learning features indeed
E VARYING INITIALIZATION
E.1 FEATURE INITIALIZATION
In the main body of the paper, we show that ResNets comprising shifts and rescalings of random features reach high accuracy. However, we have only studied one class of random features: those produced by He normal initialization (He et al. 2013b). It is possible that other initializations may produce features that result in higher accuracy. We explored initializing with uniform distribution, binarized weights, and samples from a normal distribution that were orthogonalized using SVD, however, doing so had no discernible effect on accuracy.
E.2 BATCHNORM INITIALIZATION
Similarly, our BatchNorm initialization scheme (y 010 1], B 0) was designed with training all parameters in mind. is possible that a different scheme may be more suitable when training only BatchNorm. We studied three alternatives: another standard practice for BatchNorm (7 1, 3 = 0), centering y around (y ~ 11, 3 = 0), and ensuring a greater fraction of normalized features pass through the ReLU activation (7 1. 3 1). The first two schemes did not change performance.
The third, where 3 = 1, increased the accuracy of the BatchNorm-only experiments by to percentage points across all depths and widths (Figure IT). improving ResNet-866 to 84% accuracy, WRN-14-32 to 75%, and ResNet-200 to 37% top-5 accuracy. Interestingly, doing so lowered the accuracy when all parameters trainable 3.5% WRN-14-33 and 0.5% (top-5) ResNet-200; on the deepest networks for CIFAR-IO, it caused many runs to fail entirely.
We conclude that (1) ideal initialization schemes for the BatchNorm parameters in the BatchNorm- only and standard scenarios appear to be different and (2) the standard training regime indeed sensitive the choice of BatchNorm initializations.
F FREEZING PARAMETERS
tried training two random parameters per channel in place of y and B to study the extent to which the BatchNorm-onlly performance was due merely to the aggregate number of trainable parameters. We found that training random parameters per channel resulted in far lower accuracy, suggesting that y and 3 indeed have particular expressive power as a product of their position as per-feature coefficient and bias.
In Section 4
we study the number of number of random parameters that must be made trainable per- channel in order to match the performance of training only BatchNorm. (Note: only collected this data for the CIFAR-10 10 Networks.) For the shallower ResNets, we must train 8 random parameters per channel (4x many parameters the number of BatchNorm parameters) to match the performance of training only and B. For the deeper ResNets, we must train 16 random parameters per channel (8x as many parameters as the number of BatchNorm parameters) match the performance of training only 7 and B. These results further support the belief that y and 3 have greater expressive power than other parameters in the network.
In Figure 13
In Figure 14
we study training both two random parameters per channel and the output layer. On CIFAR- 10, doing so matches or underperforms training only BatchNorm. On ImageNet, doing so outperforms training only BatchNorm in shallower networks and matches the performance of training only BatchNorm in deeper networks. In all cases, it far underperforms training BatchNorm and the output layer.
These networks separate into two accuracy strata based on width. The lower stratum has width scale 1/8 and is deeper, while the higher stratum has width scale 1/4 to 1/2 and is shallower.
17

---

Published as a conference paper at ICLR 2021
G MAKING SHORTCUTS TRAINABLE
is possible to train deep ResNets due to shortcut connections that propagate gradients to the earlier layers (He al., 2015a Balduzzi et al. 2017). Nearly all shortcuts use the identity function and have no trainable parameters. However, the shortcuts that downsample use 1xl convolutions. It is possible that, by freezing these parameters, we have inhibited the ability our networks to propagate gradients to lower layers and take full advantage of the BatchNorm parameters.
To evaluate the impact of this restriction, we enable training for the shorteut and output layer parameters in addition to the BatchNorm parameters (Figure 12). For ResNet-1 1 10, making these additional 3.2k (0.38%) parameters trainable improves accuracy by five points to 74.6%, suggesting that freezing these parameters indeed affected performance. However, on deeper networks, the returns from making shortcut and output parameters trainable diminish, with no improvement in accuracy on ResNet-8666. If freezing these parameters were really impediment to gradient propagation, would expect the deepest networks to benefit most, so this evidence does not support our hypothesis.
As alternate explanation, we propose that accuracy improves simply due the presence of more trainable parameters. As evidence for this claim, the shallowest networks for which shortcut and output parameters make up larger proportion of weights weights--bnefit benefit most when these parameters trainable. Doing so quadruples the parameter-count of ResNet-14, which improves from 48% to 63% accuracy, and adds further 2.6M parameters (315x the number of BatchNorm parameters) WRN- 14-32, which improves from 73% to 87% accuracy. We see a similar effect for the ImageNet networks: the top-5 accuracy of ResNett101 improves from 25% 72%, but the number of parameters also increases 26x from 105K to 2.8M. Finally, if we freeze BatchNorm and train only the shortcuts and outputs, performance of shallower networks is even better than training just BatchNorm, reaching 49% (vs. 48%) for ResNet- 14 and 35% (vs. 25%) top-5 for ResNet-101.
H BATCHNORM PARAMETER DISTRIBUTIONS FOR ALL NETWORKS
In Section 5 we plot the distributions of y and 3 for ResNet-1 10 for CIFAR-10 and ResNet- 101 for ImageNet. For all other networks, we include a summary plot (Figure showing the fraction of y parameters below various thresholds. In this Appendix, we plot the distributions for the deep
ResNets for CIFAR-I0 (Figures 15 and (6]. the wide ResNets for CIFAR-10 (Figures 17 and 18).
ImageNet ResNets (Figures 19 and 20 and the VGG networks (Figures 21 and 22
I FREQUENCY OF ZERO ACTIVATIONS
In Section we plot the number of ReLUs for which the Prjactivation 0) >> 0.99 for each ResNet. In Figure (deep ResNets for CIFAR-10), Figure 24 (wide ResNets for CIFAR- 10), and Figure 255 (ImageNet ResNets), we plot histogram of the values of Prfactivation 0] for the ReLUs in each network. In general, training only BatchNorm leads to many activations that are either always off or always on. In contrast, training all parameters leads many activations that some of the time (10% to 70%).
18

---

Published as a conference paper at ICLR 2021
Test Accuracy (66
90
CIFAR- VGG
All Parameters Trainable
All Parameters Trainable (y and 8 Disabled)
Figure Accuracy when training VGG networks. Accuracy does not differ when training with y and disabled.
90
Test Accurasy
70
60
50
40
30
20
10
11
13
16
19
CIFAR-10
All Paramp Trainable
BatchNorm
Random Params Per Channel
BatchNorm Output
Output
Chance
Figure 9: Accuracy of VGG networks for CIFAR-10 when making certain parameters trainable.
85
Text Accurecy (S)
85
55
0.5
Thousands Trainable Parametors (log scale!
Figure 10: Comparing training only BatchNorm with training all parameters in small ResNets found via grid search.
90
Test Accuracy (66)
70
70
60
60
50
50
40
40
30
Test
30
20
20
10
10
14
32
56
110
218
434
866
14-1
14-2
14-4
14-6
14-16
14-32
CIFAR-10 ResNet
CIFAR-10
Top-1 Tex Accuracy (96)
90
90
80
80
70
70
60
60
50
50
40
40
30
30
20
20
10
34
101
101
Ima geNet ResNet
ImageNet ResNet
All Params Trainable
All Params Trainable, Alternat
BatchNorm
BatchNorm, Alternaee
Chance
Figure 11: Accuracy of ResNets for CIFAR-10 (top left, deep; top right, wide) and ImageNet (bottom left, top-1 accuracy; bottom right, top-5 accuracy) with the original BatchNorm initialization (7 ~ (0, 1]. 3 = 0) and an alternate initialization (y = 1, B = 1).
19

---

Published as a conference paper at ICLR 2021
90
90
Text Accurny (16)
80
80
70
70
60
60
50
50
40
40
30
30
20
20
14
56
210
434
866
14-1
14-2
14-4
14-16
14-32
CIFAR-10
Top-1 Text Accuracy (%)
100
100
90
90
80
80
70
70
60
60
50
50
40
40
30
30
Imagelet ResNet
ImageNeet ResNet
Params Trainable
BatchNorm
BatchNorm Shortcuts
Output
Shortcuts Output
Chance
Figure 12: Accuracy of ResNets for CIFAR-10 (top left, deep; top right, wide) and ImageNet (bottom left, top-1 accuracy; bottom right, top-5 accuracy) when making output and shortcut layers trainable addition BatchNorm.
Text Accuracy (%)
70
Test Accuracy
70
60
60
50
50
40
40
30
30
20
20
10
14
32
56
11.
218
434
B66
14-1
14-2
14-4
14-8
14-16
14-32
CIFAR-10 ResNet
All Params Trainable
BatchNorm
Randem Params Channel: 2
Chance
Figure 13: Accuracy of ResNets for CIFAR-10 (left, deep; right, wide) when training only a certain number of randomly-selected parameters per convolutional channel. When training two random parameters per-channel, we are training the same number of parameters as when training only BatchNorm.
20

---

Published as a conference paper at ICLR 2021
100
100
90
90
Text Accuracy (%)
80
80
70
70
20
30
Test
20
10
10
14
56
218
434
866
14-1
14-2
14-4
14-8
14-16
14-32
CIFAR-10 ResNet
CIFAR-10 WRN
Top-1 Text Acauncy (No
100
100
90
90
80
80
70
40
[4
40
30
30
20
20
10
1OP
10
18
34
50
101
200
18
34
50
101
200
Ima geNet ResNet
ImageNet ResNet
All Params Trainable
BatchWorrm
BatchNorm + Cutput
Two Random Params per Channel Output
Chance
Figure 14: Accuracy of ResNets for CIFAR-10 (top left, deep; top right, wide) and ImageNet (bottom left, top-1 accuracy; bottom right, top-5 accuracy) when making two random parameters per channel and the output layer trainable.
21

---

Published as a conference paper at ICLR 2021
Distribution of ResNet-14 for CIFAR-10
Distribution of ResNet CIFAR-10
All Params Trainable Batrhkorm
Paramss Trainable
Danaty
1.5
Density
1.0
0.5
0.0
1.5
-100
-0.5
1.0
1.5
2.0
2.5
215
-1.0
-0.5
1.5
2.0
2.5
Distribution of y Re5Net-56 for CIFAR-10
Distribution of y- ResNet- for CIFAR-10
Paroass
BatchNorm
BatchNorm
91.0
9.0
Value
Value
Distribution of ResNet-221 for CIFAR-10
Distribution
ResNet-434 CIFAR-10
10.0
Al Porams Trainable Batch Norm
All Po omm Trainoale
Denaty
Oeeaityy
7.5
Batchtarm
5.0
2.5
-8.75
-0.50
-0.25
0.25
0.75
1.00
1.25
0.6
-0.50
-0.25
0.00
0.25
0.50
0.75
1.00
Value
Value
Distribution of ResNet-866 CIFAR-10
Al Perams Trainable
Donsity
Batch Norm
-8.75
-0.50
-0.25
0.00
0.25
0.50
0.75
1.00
Value
Figure 15: The distributions of for the deep CIFAR-10 ResNets.
22

---

Published as a conference paper at ICLR 2021
Distribution of ş ResNet-14 for CIFAR-10
Distribution of ş ResNet-32 for CIFAR-10
All Params Trainalle BatchNorm
All Params Trainable
BatchNorm
Density
Density
91.5
-1.0
0.5
0.0
1.0
1.5
2.0
2.5
9.5
-1.0
-0.5
1.0
1.5
2.0
2.5
Vallue
Value
Distribution B ResNet-56 CIFAR-10
Distribution ResNet-110 CIFA-110
AllParams Trainable
Params Trainable
Density
Density
91.0
-0.5
0.0
1.0
1.5
2.0
-1.0
-0.5
0.0
0.5
1.5
Distribution of $ ResNet-2118 for CIFAR-10
Distribution of & ResNet-434 CIFAR-10
Params
Donaity
Batch Norm
Donsity
Batch-Norm
-8.75
-0.50
-0.25
0.50
1.25
-8.75
-0.25
0.25
Distribution of 5 Restett- 856 for CIFAR-10
Params Trainable ButchNamm
Density
5.0
2.5
0.8.75
0.00
Figure 16: The distributions of 33 for the deep CIFAR-10 ResNets.
23

---

Published as a conference paper at ICLR 2021
Distribution of y- WRN-14-1 for CIFAR-10
Distribution of WRN-14-2 for CIFAR-10
2.0
All Parama Trainable BatchNorm
Dendit
Density
1.0
0.5
-1.5
-1.0
-0.5
0.0
0.5
1.0
1.5
2.0
2.5
1.5
-1.0
-0.5
1.0
1.5
2.0
255
Value
Value
Distribution of W for CIFAR-10
Distribution of WRN-14-8 for CIFAR-10
All Params Trannalee BatchNorm
All Parama Trainable
Dersity
Density
BotchNorm
91.0
-0.5
0.0
0.5
1.0
1.5
2.0
1
-0.5
D.D
D.5
1.0
1.5
Distribution of y- WAN-14-16 CIFAR-10
Distribution of y- WRN-14-32 for CIFAR-10
Denaly
Norm
Density
-8.75
-8.75
-0.50
-0.25
0.25
0.50
0.75
1.00
1.25
-0.50
-0.25
0.00
0.25
0.50
1.00
Figure 17: The distributions of y for the wide CIFAR-I0 ResNets.
Distribution of 8 WRN-14-1 for CIFAR-10
Distribution of 8 WRN-14-2 for CIFAR-10
Paromss Trainale BatchNorm
All Porams Trainable
Density
Density
BatchNorm
91.5
-1.0
-0.5
1.0
1.5
2.0
2.5
9.5
-1.0
-0.5
1.0
2.0
2.5
Value
Value
Distribution of WRN-14-4 for CIFAR-10
Distribution of WRN-14-8 for CIFAR-10
All Paroms Trainable BatchNorm
All Paroms Trainable
Density
Density
BanchNorm
91
-0.5
0.0
0.5 Value
1.0
1.5
2.0
9.
-0.5
0.D
0.5
1.0
1.5
Value
Distribution of WRN-14-16 CIFAR-10
Distribution of f WRN-14-22 for CIFAR-10
10.0
Trainaale
Densty
Norn
Denskty
7.5
-8.75
-0.50
-0.25
0.00
0.25
0.50
0.75
1.00
1.25
0.6.75
Figure 18: The distributions of 3 for the wide CIFAR-10 ResNets.
24

---

Published as a conference paper at ICLR 2021
Distribution y ResNet-18 ImageNet
y
ImageNet.
All Params Trainable Beichburm
10
All Parama Trainable
Doneity
Densty
Batchhorm
0.55
2.0
-6.50
-0.25
0.00
0.25
0.50 Value
0.75
Value
Distribution of y ResNett 50 for ImageNet
Distribution of y ResNet 101 for ImageNet
10.0
7.5
All Params Trainable Batchoorm
Daraty
Density
5.0
2.5
0.3
Value
Value
Distribution of ResNet-200 for ImageNet
Params Trainable
Density
BatchNorm
-0.4
-0.2
0.0
0.2
0.4
0.6
0.8
1.0
Value
Figure 19: The distributions of y for the ImageNet ResNets.
Distribution of ResNet-18 18 ImageNet
Distribution of ResNet-34 for ImageNet
20
Denoty
15
All Parama Trainable
suschhorm
Density
15
All Params Trainable
BatchNorm
10
Value
Value
Distribution of 8- ResNeet-50 for ImageNet
Distribution of # ResNet-101 for ImageNet
15
Params Trainatle
Parama Trainatle
Danaby
10
Beichtorm
Denaby
25
Betch/oomm
10
%.3
1.0
Value
Value
Distributian of 8- 2esNet-200 for ImageMet
10
Params Tranabie Batchtormm
Density
-0.4
-0.2
0.0
Value
Figure 20: The distributions of B for the ImageNet ResNets.
25

---

Published as a conference paper at ICLR 2021
Distribution
Distribution y
Denaity
Density
Batch
-8.50
-0.25
0.00
0.25
0.50
0.75
1.00
1.25
-8.50
-0.25
0.00
0.25
0.50
0.75
1.00
1.25
Value
Distribution y
Distribution y
Denstt
Norm
Density
-8.50
-0.25
0.00
0.25
0.50
0.75
1.00
1.25
-8.50
-0.25
0.00
0.25
0.50
0.75
1.00
1.25
Figure 21: The distributions of for the VGG networks.
Distribution p-
Densiyy
Batchlicrm
Density
Batchiiom
0
0
-0.5
0.0
0.5
1.0
-0.5
0.0
0.5
1.0
Value
Value
8
Distribution of VGG 19 for CIFAR-10
10
Al Params Trainatle Bitchhiorm
Density
Batchtom
Density
0
-0.5
0.0
0.5
1.0
100
-0.5
0.0
0.5
1.0
Value
Figure 22: The distributions of 3 for the VGG networks.
26

---

Published as a conference paper at ICLR 2021
Histogram of Prjactivation 0] ResNet for CIFAR-16 10
Histogram of Prjactivation 0] ResNet 32 for CIFAR
All Params Tranable Tatchhoomm
All Params Trainalle
Batt Natch
Density
Density
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Prfactivation
Prfactivation
Histogram of Prjactivation 0j ResNet-56 for CIFAR-10
Histogram of Praactivation 0] ResNet-1110 for CIFAR-10
All Parami Tranable Batchticrm
All Params Tranabic
Batchliom
Density
Density
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Prfactivation
Prfactivation
Histogram of Pri activation
ResKet-218 for CIFAR-10
Histogram of Pracivvaaioon
ResNet-433 for CIFAR-10
All Params Tranable
All Params Tranable
Densiyy
Density
0.8
Prfactivation
Prfactivation
Histogram of Pi(activation ResNet-866 for CIFAR-10
Trainable
Density
Prfactivation 0]
Figure 23: Per-ReLU activation freqencies for deep CIFAR-10 ResNets.
27

---

Published as a conference paper at ICLR 2021
Histogram Prlactivation 0)- WRN-14-1 for CIFAR-10
Histogram of Prlactivation 8)- WRN-14-2 for CIFAR-10
All Paramo Tranable Batchtiorm
All Params Tranabic Batchtiom
Density
Density
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Pr(activation
Prfactivation
Histogram of Prjactivation oj WRN-14-4 for CIFAR-10
Histogramn of Pr(activation 0) WRN-14-8 for CIFAR-10
10.0
Parems Trainable
10.0
All Parama Trainable
Batchhom
BatchNerm
Denalty
Density
2.5
2.5
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Priactivation
Prfactivation
Histogram of Prjectivation 0] WRN-14-16 for CIFAR-10
Histogram of Pr(activation 0] 1R-N-14-22 for CIFAR-10
10.0
Parsens Trainable BatchWorm
Params Trainable BatchNorm
Dendity
7.5
Dendity
7.5
5.0
2.5
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
Prlactivation 0)
Pr(activation
Figure 24: Rer- activation fregencies for CIFAR-10 WRNs.
Histogram of Pr(activation 0] ResNet-18 for Ima ge
Histogram of Prjactivation 0] ResNet-34 for ImageNtt
All Paromss Trainable Bachhoom
All Params Trainable BatchNerm
Denalyy
7.5
Densiy
7.5
0.0
0.2
0.4
0.6
1.0
0.0
0.2
0.4
0.6
1.0
Prjactivation
Prfactivation
Histogram of Prfactivation 0] ResNett1101 for ImageNet
Histogram of Pr[activation
ResNet-50 for emageN
All Parerrs Trainable BatchNorm
All Params Tranable
BatchNorm
Denaity
7.5
Density
5.0
2.3
0.0
0.0
0.2
0.4
0.6
0.8
1.0
Prfactivation
Prfactivation
Histogram Prfactivation 0) ResNet-200 for ImageNet
All Params Tranable
Density
0.2
0.6
0.8
Prfactivation 0]
Figure 25: Per-ReLU activation freqencies for ImageNet ResNets.
28


Reviews:
[{"review_id": "w8CUUArI5nN", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the expressive power of batchnorm parameters by training only these parameters while fixing other randomly initialized parameters. With experiments on different datasets and models, the authors show that batchnorm parameters are consistently more expressive than other parameters. The authors also try to explain such phenomenon by examining the values of parameters and activations, showing that training BN only can lead to sparse values. \n\nOverall, I am leaning toward accepting. The following are the pros and cons of this paper.\n\nStrengths:\n1. The paper is well written, and it is easy to follow. \n2. Understanding the mechanism behind batchnorm(BN) is a very important problem. \n3. The paper provides comprehensive experiments, demonstrating the expressive power of BN parameters from small models to large models (controlled by both depth and width). \n4. To explain the effect, the paper provides some interesting results from training BN parameters only: there is sparsity in the BN parameters, and sparsity in ReLU activations. \n\nWeaknesses:\n1. My major concern is that the expressive power of BN parameters has already been discussed in several papers [1,2] on multi-task learning or transfer learning. For example, [1] showed training different BN parameters for different tasks can greatly enhance the models' performance for multiple tasks without significantly increasing the model size. I think this paper should take the literature into the discussion and better pose the contribution of this paper.  \n2. I think the paper could be stronger if it could provide more implication of its findings. More discussions on how can we make use of these findings, (one example would be like in [1] we can build a compact model for multiple tasks) would greatly enhance the impact.\n3. This paper focuses on batchnorm. Actually, layernorm (in transformer), instance norm, group norm and so on, all have such parameters. It would be interesting to see whether the results generalize to other normalization types. \n4. It would also be interesting to discuss more on the difficulty of training BN parameters only compared with the difficulty of training whole networks. Does it become more sensitive or less sensitive to learning rate, batch size?\n\n \nComments:\n\n1. Introduction, second paragraph, 'in the time since batchnorm was first proposed' -> 'since batchnorm was first proposed'.\n\n[1] Mudrakarta, Pramod Kaushik and Sandler, Mark and Zhmoginov, Andrey and Howard, Andrew. K for the Price of 1: Parameter-efficient Multi-task and Transfer Learning, ICLR 2019.\n[2] S.-A. Rebuffi, H. Bilen, and A. Vedaldi. Learning multiple visual domains with residual adapters. In NIPS, 2017.\n\n\n***********************\nAfter rebuttal: I maintain my rating and think this paper is on the borderline. I think the paper is interesting, but the novelty and significance is limited by previous works along the direction.", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZaFXjtcQ5F7", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the effect of training BN parameters on training deep neural networks. The conclusion is striking: learning only BN parameters is enough when increasing the depth of the network. Authors have done extensive experiments to understand the effect of increasing the depth and width of the network. To stress the important role of BN parameters, the same number of parameters are chosen randomly and trained. Yet, it is observed BN parameters can obtain far better accuracy. Furthermore, an interesting observation is conducted on the distribution of BN parameters: when training only these parameters, a sparsity pattern is observed on the optimal parameters. While learning all parameters does not reach such a sparse pattern for BN parameters. The sparsity pattern indicates that an efficient network only needs to have a particular ground-truth connection between different units and the choice of weights is not important. This shows that random features imposed by neurons can create a very interesting function class when they are connected in a proper way. \n\nI have a question to understand the results betters: When authors are talking about optimizing only BN parameters, do they mean that they did not optimize parameters of the last linear layer in the network (the one that is connected to outputs).\n\nI have one concern about the way that the key message delivered here: BN parameters are very important and needed to be trained. Yet, the effectiveness of these parameters is tied to the randomness of the weights. Let me explain more about this. Recent studies show that a BN network with random weights (according to the standard initialization scheme for neural networks) provides more distinguishing features compare to vanilla networks (see for example \"Understanding batch normalization\" and \"Batch Normalization Provably Avoids Rank Collapse\").  This may be the reason why training only BN parameter is sufficient as the interesting features that are created by random weights. This hypothesis can be checked with a simple experiment: initialize the weights with a non-zero mean distribution (or any other non-standard initialization) then see whether still the training only BN parameters is effective. My (academic) guess is that you can easily find an initialization scheme for weights which makes the training only BN parameters ineffective. This result will provide more intuition about the coupling of the weights and BN parameters. \n\n* I have read the authors' response and also other reviewers. In my view, this paper provides novel insights into batch normalization.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "JBmue2Sv81g", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n-------------\nThe authors explore the representational power of BatchNorm's affine parameters (scale $\\gamma$ and bias $\\beta$). For that, they freeze the randomly-initialized parameters of different versions of ResNet and VGG, and only train the affine transformations. They also compare the expressiveness of BatchNorm coefficients with respect to the same amount of neural net parameters.  The main conclusions of this work are that BatchNorm coefficients have a greater discriminative power than the rest of network parameters. Moreover, in random networks, $\\gamma$ seems to disable non-useful features, disabling more than 25% of the channels, and in non-random networks it may prevent overshooting. They also show how these coefficients interact with networks of different depth and width, concluding that deeper random networks achieve better performance than wider random networks with the same amount of BatchNorm parameters.\n\nOverall Review\n--------------------\nUnderstanding the principles that make BatchNorm so successful is key to design better neural network architectures. This makes this submission interesting for the research community. On the other hand, it is known that $\\gamma$ and $\\beta$ are highly expressive [1,2] and that they produce sparsity [3]. In summary, in its current state, it is difficult to see what the impact of this submission in future works will be and I cannot recommend it for acceptance. Thus, I encourage the authors to increase the depth of their study (see weaknesses).\n\nStrengths\n-------------\n* Understanding BatchNorm is an important and interesting topic.\n* The paper is well-organized and easy to read.\n* The proposed experiments are sound.\n\nWeaknesses\n-----------------\n* It is known that BatchNorm coefficients have a high expressive power. For instance, FILM [1] layers modulate neural networks by changing these coefficients. In fact, these coefficients are not only important for BatchNorm, but also for other normalizations such as PixelNorm as shown in StyleGAN [2]. The authors acknowledge this in \"Limitations and Future Work\". Thus, this work does not really provide much insight on BatchNorm itself but on random features and conditioning, which makes the title, abstract and introduction misleading.\n* It is also known that BatchNorm coefficients can produce sparsity [3]. I suggest the authors to update the text to account for this fact.\n* Besides the main experiment of training BatchNorm only vs different alternatives, it would have been interesting if the authors had provided the results for alternative transformations. For instance, what would happen if only the PReLU parameters were trained? and if only Squeeze and Excitation blocks were trained [7]? Does the expressiveness of the affine transform change if using pre-activation or post-activation blocks [5]? \n* Although the authors cite [8], they do not provide any insight if the \"activation prunning\" performed by $\\gamma$ could be helping to find lottery tickets [9, 10]. It would be interesting if the authors added some result or new insight about that.\n* Questions and Clarifications:\n  * For the experiment where 2 random params per channel are trained, are these parameters \"pixels\" of convolutional filters? In this case, it may be normal that an affine transformation works better since the number of random parameters of the convolution will overwhelm the number of non-random ones. To be more fair, you could train a depth-wise separable convolution [6], and change an equivalent number of parameters as BatchNorm. \n  * What happens with BatchNorm and its parameters when two parameters per channel are learned? Are they randomly initialized and frozen?\n  * Does the standardization performed by BatchNorm boost the performance that can be obtained by the affine transformation?\n\nTypos\n--------\nPage 3: \"train[]\"\n\n\n[1] Perez, Ethan, et al. \"Film: Visual reasoning with a general conditioning layer.\" arXiv preprint arXiv:1709.07871 (2017).\n\n[2] Karras, Tero, Samuli Laine, and Timo Aila. \"A style-based generator architecture for generative adversarial networks.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2019.\n\n[3] Mehta, Dushyant, Kwang In Kim, and Christian Theobalt. \"On implicit filter level sparsity in convolutional neural networks.\" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019.\n\n[4] He, Kaiming, et al. \"Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.\" Proceedings of the IEEE international conference on computer vision. 2015.\n\n[5] He, Kaiming, et al. \"Identity mappings in deep residual networks.\" European conference on computer vision. Springer, Cham, 2016.\n\n[6] Chollet, François. \"Xception: Deep learning with depthwise separable convolutions.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.\n\n[7] Hu, Jie, Li Shen, and Gang Sun. \"Squeeze-and-excitation networks.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.\n\n[8] Zhou et al. (2019) and Ramanujan et al. (2019) in the submission.\n\n[9] Frankle, Jonathan, and Michael Carbin. \"The lottery ticket hypothesis: Finding sparse, trainable neural networks.\" arXiv preprint arXiv:1803.03635 (2018).\n\n[10] Malach, Eran, et al. \"Proving the Lottery Ticket Hypothesis: Pruning is All You Need.\" arXiv preprint arXiv:2002.00585 (2020).\n\n\n-----------------------\nAfter Rebuttal\n============\nMy main concern was that it is already known that BatchNorm has a great expressive power, and thus the authors should have gone broader and deeper in their search for valuable insights to explain what makes BatchNorm so special. I proposed different ways to do that, such as comparing with other parametric transformations like squeeze-and-excitation, or trying to understand what is the role of normalization. My concerns were shared with Reviewer 4, who proposed an interesting experiment that involves comparing multiple normalization functions such as LayerNorm.\n\nThe authors have agreed with some of these points, and updated the text to reflect the discussion. However, they believe all these suggestions belong to future work and, although the form of the paper has changed, the content is still the same.\n\nOverall, I think this work is interesting and I think studies like this are necessary (although more in depth). Thus,  I have raised my score to borderline. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "ABv5Kr3_xGj", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper investigates the role of shit and bias parameters in BatchNorm. The authors find that by training BatchNorm parameters with other parameters fixed, the networks can already achieve a good performance. They also show a connection between BatchNorm parameters and disabling features.\n\nPros:\n- The topic is interesting. Understanding the important role of gamma and beta offers great help to many relevant tasks since BatchNorm has been widely used in various networks.\n- The paper is well written and organized.\n- Extensive experiments validate the proposed findings.\n\nCons:\n- The paper is more like a technical report. Although it draws many interesting conclusions about the BatchNorm parameters, the authors do not provide how these findings contribute to the community. I expect that the authors can show some promising results on applications. If not, some theoretical analyses may need to be provided.\n- Some observations are not novel. For example, Mehta et al. [1] also found the connection between gamma and sparsity. The role of gamma has also been proved by Luo et al. [2]\n\nIn conclusion, this paper provides some insights into understanding the role of BatchNorm parameters. However, some observations are not novel and it is unclear how these findings help the community, which prevents this work to be accepted.\n\n[1] On implicit filter level sparsity in convolutional neural networks. CVPR, 2019.\n\n[2] Towards understanding regularization in Batch Normalization. ICLR, 2019.\n\n\n****After rebuttal\n\nI would appreciate the authors' detailed response to my concerns. It clearly clarifies their contributions compared with the previous work. Overall, the paper conducts comprehensive experiments and shows the expressive power of BN parameters. In particular, an interesting observation about the random feature and affine parameters is made. In the rebuttal, the author also gives some implications of their findings. I would like to increase my rating to 6.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper provides an astonishingly simple experiment: the parameters in the network are fixed, but only the parameters in the BatchNorm (taking less than 1% of the total number of parameters) are trained and also the last linear layer is trained. 
 The resulting networks provide better accuracies than training a random subset of the network. Another part of this work is the study of the effect of $\beta$ and $\gamma$ when doing full training. 

Pros: - All the reviewers agree this is an interesting and important observation.  
          - Contribution is clear and paper is well-written
          - In future, better understanding of different parameters may 

Cons: A concern has been raised by one of the reviewers that it is more like a technical report
           Some previous work which studies the effect of $\gamma$ was not mentioned. 

I think, the most interesting part is training only $\beta$ and $\gamma$. It will provide a ground for theoretical investigations of the properties of deep neural network models, and maybe lead to more efficient training algorithms.



Decision: Accept (Poster)