Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs

1 INTRODUCTION . Throughout the literature on deep learning , a wide variety of techniques rely on learning affine transformations of features—multiplying each feature by a learned coefficient γ and adding a learned bias β . This includes everything from multi-task learning ( Mudrakarta et al. , 2019 ) to style transfer and generation ( e.g. , Dumoulin et al. , 2017 ; Huang & Belongie , 2017 ; Karras et al. , 2019 ) . One of the most common examples of these affine parameters are in feature normalization techniques like BatchNorm ( Ioffe & Szegedy , 2015 ) . Considering their practical importance and their presence in nearly all modern neural networks , we know relatively little about the role and expressive power of affine parameters used to transform features in this way . To gain insight into this question , we focus on the γ and β parameters in BatchNorm . BatchNorm is nearly ubiquitous in deep convolutional neural networks ( CNNs ) for computer vision , meaning these affine parameters are present by default in numerous models that researchers and practitioners train every day . Computing BatchNorm proceeds in two steps during training ( see Appendix A for full details ) . First , each pre-activation1 is normalized according to the mean and standard deviation across the mini-batch . These normalized pre-activations are then scaled and shifted by a trainable per-feature coefficient γ and bias β . One fact we do know about γ and β in BatchNorm is that their presence has a meaningful effect on the performance of ResNets , improving accuracy by 0.5 % to 2 % on CIFAR-10 ( Krizhevsky et al. , 2009 ) and 2 % on ImageNet ( Deng et al. , 2009 ) ( Figure 1 ) . These improvements are large enough ∗Work done while an intern and student researcher at Facebook AI Research . 1He et al . ( 2016 ) find better accuracy when using BatchNorm before activation rather than after in ResNets . that , were γ and β proposed as a new technique , it would likely see wide adoption . However , they are small enough that it is difficult to isolate the specific role γ and β play in these improvements . More generally , the central challenge of scientifically investigating per-feature affine parameters is distinguishing their contribution from that of the features they transform . In all practical contexts , these affine parameters are trained jointly with ( as in the case of BatchNorm ) or after the features themselves ( Mudrakarta et al. , 2019 ; Dumoulin et al. , 2017 ) . In order to study these parameters in isolation , we instead train them on a network composed entirely of random features . Concretely , we freeze all weights at initialization and train only the γ and β parameters in BatchNorm . Although the networks still retain the same number of features , only a small fraction of parameters ( at most 0.6 % ) are trainable . This experiment forces all learning to take place in γ and β , making it possible to assess the expressive power of a network whose only degree of freedom is scaling and shifting random features . We emphasize that our goal is scientific in nature : to assess the performance and the mechanisms by which networks use this limited capacity to represent meaningful functions ; we neither intend nor expect this experiment to reach SOTA accuracy . We make the following findings : • When training only γ and β , sufficiently deep networks ( e.g. , ResNet-866 and ResNet-200 ) reach surprisingly high ( although non-SOTA ) accuracy : 82 % on CIFAR-10 and 32 % top-5 on ImageNet . This demonstrates the expressive power of the affine BatchNorm parameters . • Training an equivalent number of randomly-selected parameters per channel performs far worse ( 56 % on CIFAR-10 and 4 % top-5 on ImageNet ) . This demonstrates that γ and β have particularly significant expressive power as per-feature coefficients and biases . • When training only BatchNorm , γ naturally learns to disable between a quarter to half of all channels by converging to values close to zero . This demonstrates that γ and β achieve this accuracy in part by imposing per-feature sparsity . • When training all parameters , deeper and wider networks have smaller γ values but few features are outright disabled . This hints at the role γ may play in moderating activations in settings where disabling γ and β leads to lower accuracy ( the right parts of the plots in Figure 1 ) . In summary , we find that γ and β have noteworthy expressive power in their own right and that this expressive power results from their particular position as a per-feature coefficient and bias . Beyond offering insights into affine parameters that transform features , this observation has broader implications for our understanding of neural networks composed of random features . By freezing all other parameters at initialization , we are training networks constructed by learning shifts and rescalings of random features . In this light , our results demonstrate that the random features available at initialization provide sufficient raw material to represent high-accuracy functions for image classification . Although prior work considers models with random features and a trainable linear output layer ( e.g. , Rahimi & Recht , 2009 ; Jaeger , 2003 ; Maass et al. , 2002 ) , we reveal the expressive power of networks configured such that trainable affine parameters appear after each random feature . 2 RELATED WORK . BatchNorm . BatchNorm makes it possible to train deeper networks ( He et al. , 2015a ) and causes SGD to converge sooner ( Ioffe & Szegedy , 2015 ) . However , the underlying mechanisms by which it does so are debated . The original authors argue it reduces internal covariate shift ( ICS ) , in which “ the distribution of each layer ’ s inputs changes during training ... requiring lower learning rates ” ( Ioffe & Szegedy , 2015 ) . Santurkar et al . ( 2018 ) cast doubt on this explanation by artificially inducing ICS after BatchNorm with little change in training times . Empirical evidence suggests BatchNorm makes the optimization landscape smoother ( Santurkar et al. , 2018 ) ; is a “ safety precaution ” against exploding activations that lead to divergence ( Bjorck et al. , 2018 ) ; and allows the network to better utilize neurons ( Balduzzi et al. , 2017 ; Morcos et al. , 2018 ) . Theoretical results suggest BatchNorm decouples optimization of weight magnitude and direction ( Kohler et al. , 2019 ) as weight normalization ( Salimans & Kingma , 2016 ) does explicitly ; that it causes gradient magnitudes to reach equilibrium ( Yang et al. , 2019 ) ; and that it leads to a novel form of regularization ( Luo et al. , 2019 ) . We focus on the role and expressive power of the affine parameters in particular , whereas the aforementioned work addresses the overall effect of BatchNorm on the optimization process . In service of this broader goal , related work generally emphasizes the normalization aspect of BatchNorm , in some cases eliding one ( Kohler et al. , 2019 ) or both of γ and β ( Santurkar et al. , 2018 ; Yang et al. , 2019 ) . Other work treats BatchNorm as a black-box without specific consideration for γ and β ( Santurkar et al. , 2018 ; Bjorck et al. , 2018 ; Morcos et al. , 2018 ; Balduzzi et al. , 2017 ) . One notable exception is the work of Luo et al . ( 2019 ) , who show theoretically that BatchNorm imposes a γ decay—a data-dependent L2 penalty on γ ; we discuss this work further in Section 5 . Exploiting the expressive power of affine transformations . There are many techniques in the deep learning literature that exploit the expressive power of affine transformations of features . Mudrakarta et al . ( 2019 ) develop a parameter-efficient approach to multi-task learning with a shared network backbone ( trained on a particular task ) and separate sets of per-task BatchNorm parameters ( trained on a different task while the backbone is frozen ) . Similarly , Rebuffi et al . ( 2017 ) allow a network backbone trained on one task to adapt to others by way of residual models added onto the network comprising BatchNorm and a convolution . Perez et al . ( 2017 ) perform visual question-answering by using a RNN receiving text input to produce coefficients and biases that are used to transform the internal features of a CNN on visual data . Finally , work on neural style transfer and style generation uses affine transformations of normalized features to encode different styles ( e.g. , Dumoulin et al. , 2017 ; Huang & Belongie , 2017 ; Karras et al. , 2019 ) . Training only BatchNorm . Closest to our work , Rosenfeld & Tsotsos ( 2019 ) explore freezing various parts of networks at initialization ; in doing so , they briefly examine training only γ and β . However , there are several important distinctions between this paper and our work . They conclude only that it is generally possible to “ successfully train [ ] mostly-random networks , ” while we find that BatchNorm parameters have greater expressive power than other parameters ( Figure 2 , green ) . In fact , their experiments can not make this distinction . They train only BatchNorm in just two CIFAR-10 networks ( DenseNet and an unspecified Wide ResNet ) for just ten epochs ( vs. the standard 100+ ) , reaching 61 % and 30 % accuracy . For comparable parameter-counts , we reach 80 % and 70 % . These differences meaningfully affect our conclusions : they allow us to determine that training only BatchNorm leads to demonstrably higher accuracy than training an equivalent number of randomly chosen parameters . The accuracy in Rosenfeld & Tsotsos is too low to make any such distinction . Moreover , we go much further in terms of both scale of experiments and depth of analysis . We study a much wider range of networks and , critically , show that training only BatchNorm can achieve impressive results even for large-scale networks on ImageNet . We also investigate how the BatchNorm parameters achieve this performance by examining the underlying representations . We also note that Mudrakarta et al . ( 2019 ) train only BatchNorm and a linear output layer on a single , randomly initialized MobileNet ( in the context of doing so on many trained networks for the purpose of multi-task learning ) ; they conclude simply that “ it is quite striking ” that this configuration “ can achieve non-trivial accuracy. ” Random features . There is a long history of building models from random features . The perceptron ( Block , 1962 ) learns a linear combination of associators , each the inner product of the input and a random vector . More recently , Rahimi & Recht ( 2009 ) showed theoretically and empirically that linear combinations of random features perform nearly as well as then-standard SVMs and Adaboost . Reservoir computing ( Schrauwen et al. , 2007 ) , also known as echo state networks ( Jaeger , 2003 ) or liquid state machines ( Maass et al. , 2002 ) , learns a linear readout from a randomly connected recurrent neural network ; such models can learn useful functions of sequential data . To theoretically study SGD on overparameterized networks , recent work uses two layer models with the first layer wide enough that it changes little during training ( e.g. , Du et al. , 2019 ) ; in the limit , the first layer can be treated as frozen at its random initialization ( Jacot et al. , 2018 ; Yehudai & Shamir , 2019 ) . In all cases , these lines of work study models composed of a trainable linear layer on top of random nonlinear features . In contrast , our models have affine trainable parameters throughout the network after each random feature in each layer . Moreover , due to the practice of placing BatchNorm before the activation function ( He et al. , 2016 ) , our affine parameters occur prior to the nonlinearity . Freezing weights at random initialization . Neural networks are initialized randomly ( He et al. , 2015b ; Glorot & Bengio , 2010 ) , and performance with these weights is no better than chance . However , it is still possible to reach high accuracy while retaining some or all of these weights . Zhang et al . ( 2019a ) show that many individual layers in trained CNNs can be reset to their random i.i.d . initializations with little impact on accuracy . Zhou et al . ( 2019 ) and Ramanujan et al . ( 2019 ) reach high accuracy on CIFAR-10 and ImageNet merely by learning which individual weights to remove .

Summary: The authors explore the representational power of BatchNorm's affine parameters (scale $\gamma$ and bias $\beta$). For that, they freeze the randomly-initialized parameters of different versions of ResNet and VGG, and only train the affine transformations. They also compare the expressiveness of BatchNorm coefficients with respect to the same amount of neural net parameters.  The main conclusions of this work are that BatchNorm coefficients have a greater discriminative power than the rest of network parameters. Moreover, in random networks, $\gamma$ seems to disable non-useful features, disabling more than 25% of the channels, and in non-random networks it may prevent overshooting. They also show how these coefficients interact with networks of different depth and width, concluding that deeper random networks achieve better performance than wider random networks with the same amount of BatchNorm parameters.