Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper conducts extensive experiments to study batch normalization, a very popular technique for training a deep convolutional network and its relationship with learning rate and batch size. In addition, the authors also propose a new initialization scheme, “ZeroInit”, to train a deep ResNet for better test accuracy. This is a very empirical study and the authors also show extensive experimental results. However, I do not see any novel findings in this study. Mostly this paper confirms the results of previous studies. The experimental results do not show much advantage of ZeroInit either. Overall, it is unclear what is the major novel contribution in this paper.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The name "ZeroInit" is very confusing, because that is how FixUp was called initially https://openreview.net/forum?id=H1gsz30cKX , perhaps the authors should consider a different name. I will call it "NewZeroInit" in my review to avoid confusion.

The paper focuses on training image classification networks without batch normalization. The authors claim that effectiveness of batch normalization, and methods which attempt to eliminate it, should be tested on a wide range of learning rates. On experiments performed on CIFAR they find that batch normalization is able to achieve high accuracy even with very high learning rates, in line with Goyal et al. 2017. Based on this, they propose a simplification of FixUp for image classification, in which they remove the need in progressive scaling of initialization, and propose to remove weight decay regularization, while adding dropout on the last layer. This "NewZeroInit" is tested on ImageNet and compares favorably to batch normalization and FixUp.

The closest studies are FixUp and Goyal et al. 2017, with the difference that FixUp studies both image classification ResNet and seq2seq approaches in the absence of batch normalization, and Goyal et al. show a wide range of large scale experiments on full scale ImageNet, whereas "NewZeroInit" studies small scale CIFAR dataset. It is thus unclear if "NewZeroInit" transfers to seq2seq.
There is also "Bag of Tricks for Image Classification with Convolutional Neural Networks" by He et al 2018 (missing citation) which evaluates a similar set of tricks on ImageNet ResNet-50 with batch normalization. In particular, they show that removing weight decay from BN bias and setting scaling gamma to 0 initially significantly improves the results.

On page 4 the authors say "Given access to sufficient hardware, this will enable practitioners to dramatically reduce wallclock time of training (Goyal et al.)". It is not clear what they mean, since Goyal et al. already enabled the reduction by increasing learning rate and minibatch size on ImageNet, whereas the results authors show are on small CIFAR dataset.

On page 5 the authors mention that they introduce bias to each convolution and classification layer, which is surprising because it is a standard way to composing a convolutional network.

Overall, the most significant contributions of the paper are:
 - a study of minibatch size on CIFAR
 - removing weight decay from FixUp on ImageNet

Also, I am interested in the following results:
 - clear comparison of FixUp with "NewZeroInit" for image classification
 - ImageNet ResNet-50 results with dropout regularization in the final layer
 - ImageNet ResNet-50 results with FixUp, dropout regularization and no weight decay.
 - (optionally) seq2seq with NewZeroInit instead of FixUp.

Without these results it hard to judge the novelty and contributions of the paper, so I propose reject.
review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: This paper conducts extensive experiments to verify two practical benefits of batch normalization. i) It increases the final test accuracy and the largest stable learning rate; ii) it enables efficient training with larger batches and a larger learning rate. In addition, the authors propose a new initialization scheme, “ZeroInit”, to train a deep ResNet to improve the test accuracy. My detailed comments are as follows.


Positive points:

1. The experiments are sufficient. In this paper, the authors conduct extensive experiments to explore the benefits of batch normalization, and verifies the effectiveness of the proposed “ZeroInit”.

2. The method is effective in some cases. Specifically, the proposed “ZeroInit” outperforms batch normalization when the batch size is small, and it is competitive with batch normalization when the batch size is not too large.

Negative points:

1. The importance and novelty of the empirical study should be emphasized. The practical benefits of batch normalization can be also found in other papers. For the first benefit, most studies (Bjorck et al. 2018) have found that batch normalization is able to improve the test accuracy. For the second benefit, batch normalization requires a large batch size and a large learning rate (Santurkar et al., 2018). Therefore, what is the difference between this paper and others? More critically, it is necessary to explain why batch normalization has these benefits. It would be better to provide empirical or theoretical justifications to support these. 

2. The motivation of the proposed “ZeroInit” is not clear. (Balduzzi et al., 2017) states that “the correlations can be preserved by initializing deep networks close to linear functions”. It is not clear how “ZeroInit” preserves the correlations? 

3. Why initialize the scalar multiplier and biases to zero? What are the benefits of the zero initialization? Actually, the scalar multiplier and biases can be randomly initialized. When they are randomly initialized, what is the performance of the initialization? It is an important baseline to justify the effectiveness of the proposed initialization method. 

4. The technical details of “ZeroInit” are not clear. It would be better to express the proposed initialization “ZeroInit” in the mathematical formulation.

5. The proposed initialization “ZeroInit” is designed for deep ResNets. How to extend it to the other deep neural networks?

6. This paper states that “the empirical success of batch normalization …improves the conditioning of the loss landscape. However, our results conclusively demonstrate that this is not the case”. Does it mean that batch normalization does not improve the conditioning of the loss landscape? However, the empirical results cannot justify this statement and explain the success of batch normalization.

7. Some results of the figures are missing. In Figure 1, the experimental results of w/o batch norm with varying batch sizes (2^0 ~ 2^5) are missing. Similarly, Figure 2 also has missing results. Please provide more discussions about these missing results.

review_title: Official Blind Review #1

Meta-review: The paper is rejected based on unanimous reviews.
Decision: Reject