Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper performs an empirical study of four batch-normalization improvements and proposes a new normalization technique for small batch sizes, based on group and batch normalizations. Among others, the authors address the inconsistency between the train and the test stages and the problem of small batch sizes. The authors conducted an empirical ablation study of the four techniques and proposed an intuition when each method should be used.

Concerns:
(1) The comparison with baselines in section 4.2 seems to be unclear. Fig.5 shows the performance of normalization methods for different batch sizes. Batch Normalization, however, has the same performance for all batch sizes. The authors refer to this baseline as “idealized Batch Normalization”. Additional elaboration on what does this means is required. 
(2) It would also be beneficial to see the comparison with the original Ghost Batch Normalization in the final evaluation (section 4.2), since this method, according to section 3.2, was capable of the significant improvement for Caltech-256 dataset.
(3) In section 3.1, the authors provide an intuition of why can the discrepancy between test and train phases hurts the performance of a model. The empirical evaluation of this effect is needed to justify this intuition.

Overall, the newly proposed method is a minor update, and novelty is limited. However, the thorough empirical study of existing improvement techniques would be a good addition to the conference.

Minor comments:
1. share the y-axis in Fig.4 between different ghost batch sizes.

I would also recommend authors to include the following papers to the related work section:
1. Riemannian approach to batch normalization [https://arxiv.org/abs/1709.09603]

----------

Respond to the rebuttal. Clarification on the concern (3):

I agree that, in general, a discrepancy between training and testing can hurt a model. The paper showed that the output of a batch normalization layer is theoretically unbounded during testing. However, it would be beneficial to see numerically if it indeed the case on a real test set (the output range is wider than the one during training).


review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: The paper introduces four techniques to improve the deep network model through modifying Batch Normalization (BN). The inspirations are from the gaps between train&test and between batches in multi-gpu training, comparison to other normalization methods, and weight decay in regularizing convolution weights training. The paper studies each techniques with the support from experiments. The paper is easy to follow. The techniques seem effective.

The paper mentions "theory" multiple times, but lacks sufficient justification to support these "theories". So one suggestion is to replace "theory" with a soft word.

Experimental evidence seems sufficient and there are some theoretical derivations, but it looks incremental that the paper presents some techniques in improving Batch Normalization only. In general, the paper is of values to the community.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The authors discuss four techniques to improve Batch Normalization, including inference example weighing, medium batch size, weight decay, the combination of batch and group normalization. Equipped with the proposed techniques, the authors obtain promising results when training deep models with various batch sizes. However, the novelty of this paper seems very limited and more experiments are required.

Please see my detailed comments below.

Positive points：
1. The proposed inference example weighing method yields promising results and does not require any re-training.
2. The combination of batch and group normalization makes it possible to train deep models with very small batch size.
3. By combining all the techniques, the proposed method yields promising performance when training deep models with different batch sizes.

Negative points:

1. Some notations are very confusing. For example, the authors use B to represent the size of a minibatch. However, why do the authors only consider B-1 samples in Eq. (2), i.e., selecting the minimum possible output among x_1, …, x_{B-1}?

2. The proposed inference example weighing method seems very similar to Batch Renormalization. Both methods seek to use a linear function to combine the batch statistics and the moving statistics. What is the essential difference between these methods?

3. What model do the authors use in the experiment of Figure 2? Why do the authors conduct experiments on different datasets in Section 3.1 (including ImageNet) and Section 3.2 (excluding ImageNet)? It would be stronger to provide ImageNet results in Section 3.2.

4. The authors draw different conclusions about the usage of weight decay from a recent work (He et al, CVPR2019). The CVPR paper reports that training \gamma and \beta without weight decay on ResNet-50 yields significant performance improvement. However, this paper shows that training ResNet-50 with weight decay improves the performance. Please comment on the differences in the conclusions.

Reference: "Bag of tricks for image classification with convolutional neural networks." CVPR, 2019.

5. The authors only report ImageNet results of the proposed inference example weighing method. However, all the experiments in Section 4 are performed on three small datasets. It is necessary and important to provide ImageNet results to show the effectiveness of the other three techniques in Section 4. 

6. Note that training deep models with non-i.i.d. minibatches is a typical case to evaluate normalization methods, e.g., Batch Renormalization. Specifically, examples in a minibatch are not sampled independently. What would happen if the authors apply the proposed techniques to the non-i.i.d. case?

7. Some closely related work should be discussed in the paper, such as

[1] "Decorrelated Batch Normalization." CVPR, 2018.
[2] "Double Forward Propagation for Memorized Batch Normalization." AAAI, 2018.
[3] "Differentiable Dynamic Normalization for Learning Deep Representation." ICML, 2019.
[4] "Iterative Normalization: Beyond Standardization towards Efficient Whitening." CVPR, 2019.

Minor issues:
1. In Section 1, the third contribution is not a complete sentence.

2. There are many typos in the paper.
(1) In Section 2, “Layer Normalization, which has found use in many natural language processing tasks.” Should “which has found use” be “which has been used”?
(2) In Section 3.1, “Batch Normalization has a disparity in function between training inference”. “between training inference” should be “between training and inference”.
(3) In Section 3.1, “we need only figure out …” should be “we only need to figure out …”


review_title: Official Blind Review #1

Meta-review: This paper proposes techniques to improve training with batch normalization. The paper establishes the benefits of these techniques experimentally using ablation studies. The reviewers found the results to be promising and of interest to the community. However, this paper is borderline due in part due to the writing (notation issues) and because it does not discuss related work enough. We encourage the authors to properly address these issues before the camera ready.
Decision: Accept (Poster)