Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper studies the effect of training BN parameters on training deep neural networks. The conclusion is striking: learning only BN parameters is enough when increasing the depth of the network. Authors have done extensive experiments to understand the effect of increasing the depth and width of the network. To stress the important role of BN parameters, the same number of parameters are chosen randomly and trained. Yet, it is observed BN parameters can obtain far better accuracy. Furthermore, an interesting observation is conducted on the distribution of BN parameters: when training only these parameters, a sparsity pattern is observed on the optimal parameters. While learning all parameters does not reach such a sparse pattern for BN parameters. The sparsity pattern indicates that an efficient network only needs to have a particular ground-truth connection between different units and the choice of weights is not important. This shows that random features imposed by neurons can create a very interesting function class when they are connected in a proper way. 

I have a question to understand the results betters: When authors are talking about optimizing only BN parameters, do they mean that they did not optimize parameters of the last linear layer in the network (the one that is connected to outputs).

I have one concern about the way that the key message delivered here: BN parameters are very important and needed to be trained. Yet, the effectiveness of these parameters is tied to the randomness of the weights. Let me explain more about this. Recent studies show that a BN network with random weights (according to the standard initialization scheme for neural networks) provides more distinguishing features compare to vanilla networks (see for example "Understanding batch normalization" and "Batch Normalization Provably Avoids Rank Collapse").  This may be the reason why training only BN parameter is sufficient as the interesting features that are created by random weights. This hypothesis can be checked with a simple experiment: initialize the weights with a non-zero mean distribution (or any other non-standard initialization) then see whether still the training only BN parameters is effective. My (academic) guess is that you can easily find an initialization scheme for weights which makes the training only BN parameters ineffective. This result will provide more intuition about the coupling of the weights and BN parameters. 

* I have read the authors' response and also other reviewers. In my view, this paper provides novel insights into batch normalization.

review_title: This is a thought-provoking experimental result that shows optimizing only BN parameters is very effective for deep neural networks 

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper investigates the role of shit and bias parameters in BatchNorm. The authors find that by training BatchNorm parameters with other parameters fixed, the networks can already achieve a good performance. They also show a connection between BatchNorm parameters and disabling features.

Pros:
- The topic is interesting. Understanding the important role of gamma and beta offers great help to many relevant tasks since BatchNorm has been widely used in various networks.
- The paper is well written and organized.
- Extensive experiments validate the proposed findings.

Cons:
- The paper is more like a technical report. Although it draws many interesting conclusions about the BatchNorm parameters, the authors do not provide how these findings contribute to the community. I expect that the authors can show some promising results on applications. If not, some theoretical analyses may need to be provided.
- Some observations are not novel. For example, Mehta et al. [1] also found the connection between gamma and sparsity. The role of gamma has also been proved by Luo et al. [2]

In conclusion, this paper provides some insights into understanding the role of BatchNorm parameters. However, some observations are not novel and it is unclear how these findings help the community, which prevents this work to be accepted.

[1] On implicit filter level sparsity in convolutional neural networks. CVPR, 2019.

[2] Towards understanding regularization in Batch Normalization. ICLR, 2019.


****After rebuttal

I would appreciate the authors' detailed response to my concerns. It clearly clarifies their contributions compared with the previous work. Overall, the paper conducts comprehensive experiments and shows the expressive power of BN parameters. In particular, an interesting observation about the random feature and affine parameters is made. In the rebuttal, the author also gives some implications of their findings. I would like to increase my rating to 6.

review_title: Review of Paper 310

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary
-------------
The authors explore the representational power of BatchNorm's affine parameters (scale $\gamma$ and bias $\beta$). For that, they freeze the randomly-initialized parameters of different versions of ResNet and VGG, and only train the affine transformations. They also compare the expressiveness of BatchNorm coefficients with respect to the same amount of neural net parameters.  The main conclusions of this work are that BatchNorm coefficients have a greater discriminative power than the rest of network parameters. Moreover, in random networks, $\gamma$ seems to disable non-useful features, disabling more than 25% of the channels, and in non-random networks it may prevent overshooting. They also show how these coefficients interact with networks of different depth and width, concluding that deeper random networks achieve better performance than wider random networks with the same amount of BatchNorm parameters.

Overall Review
--------------------
Understanding the principles that make BatchNorm so successful is key to design better neural network architectures. This makes this submission interesting for the research community. On the other hand, it is known that $\gamma$ and $\beta$ are highly expressive [1,2] and that they produce sparsity [3]. In summary, in its current state, it is difficult to see what the impact of this submission in future works will be and I cannot recommend it for acceptance. Thus, I encourage the authors to increase the depth of their study (see weaknesses).

Strengths
-------------
* Understanding BatchNorm is an important and interesting topic.
* The paper is well-organized and easy to read.
* The proposed experiments are sound.

Weaknesses
-----------------
* It is known that BatchNorm coefficients have a high expressive power. For instance, FILM [1] layers modulate neural networks by changing these coefficients. In fact, these coefficients are not only important for BatchNorm, but also for other normalizations such as PixelNorm as shown in StyleGAN [2]. The authors acknowledge this in "Limitations and Future Work". Thus, this work does not really provide much insight on BatchNorm itself but on random features and conditioning, which makes the title, abstract and introduction misleading.
* It is also known that BatchNorm coefficients can produce sparsity [3]. I suggest the authors to update the text to account for this fact.
* Besides the main experiment of training BatchNorm only vs different alternatives, it would have been interesting if the authors had provided the results for alternative transformations. For instance, what would happen if only the PReLU parameters were trained? and if only Squeeze and Excitation blocks were trained [7]? Does the expressiveness of the affine transform change if using pre-activation or post-activation blocks [5]? 
* Although the authors cite [8], they do not provide any insight if the "activation prunning" performed by $\gamma$ could be helping to find lottery tickets [9, 10]. It would be interesting if the authors added some result or new insight about that.
* Questions and Clarifications:
  * For the experiment where 2 random params per channel are trained, are these parameters "pixels" of convolutional filters? In this case, it may be normal that an affine transformation works better since the number of random parameters of the convolution will overwhelm the number of non-random ones. To be more fair, you could train a depth-wise separable convolution [6], and change an equivalent number of parameters as BatchNorm. 
  * What happens with BatchNorm and its parameters when two parameters per channel are learned? Are they randomly initialized and frozen?
  * Does the standardization performed by BatchNorm boost the performance that can be obtained by the affine transformation?

Typos
--------
Page 3: "train[]"


[1] Perez, Ethan, et al. "Film: Visual reasoning with a general conditioning layer." arXiv preprint arXiv:1709.07871 (2017).

[2] Karras, Tero, Samuli Laine, and Timo Aila. "A style-based generator architecture for generative adversarial networks." Proceedings of the IEEE conference on computer vision and pattern recognition. 2019.

[3] Mehta, Dushyant, Kwang In Kim, and Christian Theobalt. "On implicit filter level sparsity in convolutional neural networks." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019.

[4] He, Kaiming, et al. "Delving deep into rectifiers: Surpassing human-level performance on imagenet classification." Proceedings of the IEEE international conference on computer vision. 2015.

[5] He, Kaiming, et al. "Identity mappings in deep residual networks." European conference on computer vision. Springer, Cham, 2016.

[6] Chollet, François. "Xception: Deep learning with depthwise separable convolutions." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.

[7] Hu, Jie, Li Shen, and Gang Sun. "Squeeze-and-excitation networks." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.

[8] Zhou et al. (2019) and Ramanujan et al. (2019) in the submission.

[9] Frankle, Jonathan, and Michael Carbin. "The lottery ticket hypothesis: Finding sparse, trainable neural networks." arXiv preprint arXiv:1803.03635 (2018).

[10] Malach, Eran, et al. "Proving the Lottery Ticket Hypothesis: Pruning is All You Need." arXiv preprint arXiv:2002.00585 (2020).


-----------------------
After Rebuttal
============
My main concern was that it is already known that BatchNorm has a great expressive power, and thus the authors should have gone broader and deeper in their search for valuable insights to explain what makes BatchNorm so special. I proposed different ways to do that, such as comparing with other parametric transformations like squeeze-and-excitation, or trying to understand what is the role of normalization. My concerns were shared with Reviewer 4, who proposed an interesting experiment that involves comparing multiple normalization functions such as LayerNorm.

The authors have agreed with some of these points, and updated the text to reflect the discussion. However, they believe all these suggestions belong to future work and, although the form of the paper has changed, the content is still the same.

Overall, I think this work is interesting and I think studies like this are necessary (although more in depth). Thus,  I have raised my score to borderline. 
review_title: Review of "Training BatchNorm and Only BatchNorm"

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper studies the expressive power of batchnorm parameters by training only these parameters while fixing other randomly initialized parameters. With experiments on different datasets and models, the authors show that batchnorm parameters are consistently more expressive than other parameters. The authors also try to explain such phenomenon by examining the values of parameters and activations, showing that training BN only can lead to sparse values. 

Overall, I am leaning toward accepting. The following are the pros and cons of this paper.

Strengths:
1. The paper is well written, and it is easy to follow. 
2. Understanding the mechanism behind batchnorm(BN) is a very important problem. 
3. The paper provides comprehensive experiments, demonstrating the expressive power of BN parameters from small models to large models (controlled by both depth and width). 
4. To explain the effect, the paper provides some interesting results from training BN parameters only: there is sparsity in the BN parameters, and sparsity in ReLU activations. 

Weaknesses:
1. My major concern is that the expressive power of BN parameters has already been discussed in several papers [1,2] on multi-task learning or transfer learning. For example, [1] showed training different BN parameters for different tasks can greatly enhance the models' performance for multiple tasks without significantly increasing the model size. I think this paper should take the literature into the discussion and better pose the contribution of this paper.  
2. I think the paper could be stronger if it could provide more implication of its findings. More discussions on how can we make use of these findings, (one example would be like in [1] we can build a compact model for multiple tasks) would greatly enhance the impact.
3. This paper focuses on batchnorm. Actually, layernorm (in transformer), instance norm, group norm and so on, all have such parameters. It would be interesting to see whether the results generalize to other normalization types. 
4. It would also be interesting to discuss more on the difficulty of training BN parameters only compared with the difficulty of training whole networks. Does it become more sensitive or less sensitive to learning rate, batch size?

 
Comments:

1. Introduction, second paragraph, 'in the time since batchnorm was first proposed' -> 'since batchnorm was first proposed'.

[1] Mudrakarta, Pramod Kaushik and Sandler, Mark and Zhmoginov, Andrey and Howard, Andrew. K for the Price of 1: Parameter-efficient Multi-task and Transfer Learning, ICLR 2019.
[2] S.-A. Rebuffi, H. Bilen, and A. Vedaldi. Learning multiple visual domains with residual adapters. In NIPS, 2017.


***********************
After rebuttal: I maintain my rating and think this paper is on the borderline. I think the paper is interesting, but the novelty and significance is limited by previous works along the direction.
review_title: This paper studies the expressive power of batchnorm parameters by training only these parameters while fixing other randomly initialized parameters. 

Meta-review: The paper provides an astonishingly simple experiment: the parameters in the network are fixed, but only the parameters in the BatchNorm (taking less than 1% of the total number of parameters) are trained and also the last linear layer is trained. 
 The resulting networks provide better accuracies than training a random subset of the network. Another part of this work is the study of the effect of $\beta$ and $\gamma$ when doing full training. 

Pros: - All the reviewers agree this is an interesting and important observation.  
          - Contribution is clear and paper is well-written
          - In future, better understanding of different parameters may 

Cons: A concern has been raised by one of the reviewers that it is more like a technical report
           Some previous work which studies the effect of $\gamma$ was not mentioned. 

I think, the most interesting part is training only $\beta$ and $\gamma$. It will provide a ground for theoretical investigations of the properties of deep neural network models, and maybe lead to more efficient training algorithms.



Decision: Accept (Poster)