Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes the signal propagation plot (spp) which is a tool for analyzing residual networks and analyzes ResNet with/without BN. Based on the investigation, the authors first provide ResNet results without normalization with the proposed scaled weight standardization. Furthermore, the authors provide a bunch of models that are competitive to EfficientNets based on RegNetY-400MF,  which seem to be highly tuned in terms of architecture design.

Pros)

1. This paper is well written and easy to follow.

2. The proposed SPP seems to be a good tool for analyzing a model.


Cons)

1. The authors seem to have failed to provide any reasons for needing normalization free ResNet over the original BN-ResNet. 

2. It is not clear that the trained model without NF with SWS can be used as a backbone that can be directly applied to downstream tasks (e.g., object detection). 

3. I don't think it was necessary to show competitive results with EfficientNets using the other baseline - RegNet.  Especially, the proposed models (which are compared with EfficientNets) are highly-tuned trying to surpass EfficientNets' accuracy. This type of paper would be better to be focused on investigating the characteristics of a network.

Comments)
1. The major problem of this paper is none of the advantages of NF with SWS are highlighted over BN, so it is hard to find any reasons for replacing BN with NF-SWS. I mean non of the disadvantages of BN are addressed by the proposed method.

2. SPP could enlighten that an unusual ReLU-BN-Conv ordering would have some benefits but why should a network without normalization mimick the SPP trend of ResNet? It is unnatural that a network without normalization should follow the ResNet's behavior.

3. From the equation x_{l+1}=x_l+ a * f (x_l/b_l) at p.4, the proposed approach eventually normalizing the feature even if a or b_l is fixed. Moreover, gamma in eq. (3) additionally scaling up or down the weight which ultimately gives an effect on the computed feature.

4. How did the authors compute gamma in eq.(3) in a training phase? Why the provided code contains a learnable gamma?

5. Why "zero padding" at p.5 affects the variance decay in the rightmost graph in Figure 2?

6. Please clarify why RegNetY-400MF chose it as the baseline. It is not clear that the authors pick RegNet and tune it highly and compared with EfficientNets. If the authors decided to use RegNets, then it is natural to use NF-SWS-RegNets are compared with the original RegNets without any big modifications as shown in the model details in the Appendix.

7. Did the authors use trained ResNetV2-600 or randomly-initialized model?

8. How does SPP goes on with the post-activation network which uses the original bottleneck block consists of Conv-BN-ReLU? 

9. How can the resolution downsampling block in a ResNet affect averaged channel mean and variance?

10. Why the ResNet experiments are done with weight decay of 5e-5? The common knowledge of training ResNet is with weighing decay of 1e-4, so one may unconvinced the result because of the tuning.

11. Comparing EfficientNets that are trained without CutMix and Mixup (but used randaug or autoaug) with the proposed models with cutmix and mixup seems to be not fair.
review_title: Review

final_score: rating: 7: Good paper, accept
review_content: Summary:

The authors propose a set of visualization tools named SPPs for better monitoring the key indicators, including Average Channel Squared Mean, Average Channel Variance, etc., of hidden activations. With SPPs in hand, they find two key failure modes at initialization in deep ResNets and then develop Normalizer-Free ResNets using Scaled Weight Standardization, achieving competitive performance on ResNet-288 and Efficient-Net.

Strength:
--The overall idea makes sense and the proposed method of removing batch normalization can reduce computational resources and speed up computing greatly.
--The visualization of the key indicators may be helpful for understanding how batch normalization works and how to remove batch normalization.  
--The experiment results seem that the proposed method achieves good performance in large-scale ResNets.

Weakness:
-- There is a lack of accuracy comparison with other removing batch normalization works. 
-- It will be better to prove the effectiveness of the proposed method in more tasks, not only the classification task. 

Comments:
(1)	The batch size you used for training is 1024. As you know, decreasing batch size is good for online training on portable devices. Can the proposed method decrease the batch size to a small value like 2/4/8 after removing batch normalization? And if it is difficult, can you give an explanation on how batch size affects the performance after removing batch normalization with the proposed method. 
(2)	You have visualized the key indicators of the proposed initialization method. Can you give the visualization of the same indicators of other initialization methods like Fixup initialization and make some comparison? I think it’s within your ability. 
(3)	It’s better to add some accuracy comparisons with other removing batch normalization works.

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
UPDATE: The author has addressed most of my concerns, but regarding the motivations and the benefits for the community, I still keep my score.


review_title: review

final_score: rating: 7: Good paper, accept
review_content: The paper proposes a novel, and mathematically well motivated initialization scheme for deep ResNet like models. 
This is used to train batchnorm free Resnets that compare to their baseline.
They also train RegNet like models that are fairly close to EfficientNets in terms of performance.

Comments: 

You list problems with batchnorm:
1) 'it breaks independence between training examples'
   Why is this a problem? You don't include any experiments showing your method works well in the small batch regime where batchnorm is problematic.
2) 'it is suprisingly expensive to compute'
    Batchnorm was widely adopted as it made networks train much faster. And there is no computational overhead at test time. Does your method allow models to be trained more quickly, i.e. how does accuracy compare after $N$ minutes of training time?
3) 'it often results in unexpected bugs'
   I don't understand this point. Batchnorm has lead to many state-of-the-art results. In contrast you have only managed to get good results by using large amounts of prior knowledge that was obtained from experiments on the same validation set and using batchnorm.

Section 5.1 Why is the training collapse of the 288 layer models not reflected in Table 1?

Regarding Figure 3: Why compare your method **with** data augmentation and EfficientNets **without**? That seems very misleading to me. Getting close to proper EfficientNet performance without batchnorm is a decent achievement. Why include an inferior, incomparable baseline?

--update---
I am upgrading to 7: Good paper accept; as my concerns have been addressed by the additional experiments.
The proposed method is not yet a drop in replacement for BatchNorm in general, but it can be useful in specific circumstances, i.e. small batch-size training.
review_title: ConvNets without normalization

Meta-review: This paper analyses the signal propagation through residual architectures; then suggests a scaling method which, together with weight standardization, allows to train such networks to high accuracy with batch-norm; it demonstrates that the method performs better than previous methods (Fixup, SkipInit), and can be used on more advanced architectures. 

The reviewers initially had several concerns, but after the author's revision, these concerns were addressed and most reviewers recommended acceptance. One reviewer did not respond, but I think these concerns were addressed. I think it will help to further convince the readers on the usefulness of the method readers if the authors would check the sensitivity to the learning rate with the current method and compare with other methods (SkipInit, Fixup, BN). The reason I'm suggesting this is that I think one of the main reasons BN is still in popular use is that it commonly tends to make training more robust to changes in hyper-parameters, such as the learning rate (while other methods, like SkipInit and Fixup, require more hyper-parameter tuning).

Overall the analysis and the suggested method seem useful, especially at a small batch size and the writing is mostly clear, so I recommend acceptance. 


Decision: Accept (Poster)