Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes a novel Cross-Iteration Batch Normalization (CBN) to address the limitation of BN in the case of small mini-batch sizes. Different from existing methods, CBN exploits the statistics cross different iterations to obtain more accurate estimates of the data statistics. Specifically, the proposed CBN uses Taylor polynomials to approximate the statistics using the information of multiple recent iterations. The experiments on both image classification and object detection tasks demonstrate the effectiveness of the proposed method.

Please see my detailed comments below.

Positive points:

1. The authors propose a novel method that exploits the information from different iterations to estimate the normalization statistics more accurately. 

2. Unlike existing methods, the authors use Taylor polynomials to cast the information of previous iterations into the current iteration. The proposed method is theoretically sound.

3. The experiments on both image classification and object detection tasks show the superiority of the proposed CBN over the considered baseline methods.

Negative points:

1. The authors only conduct experiments on a single image classification model ResNet-18. More deep architectures should be considered in the experiments, e.g., DenseNet, MobileNet, etc.

2. Several state-of-the-art normalization methods should be compared in the experiment of " sensitivity to batch size", including SN [1] and DN [2].

3. One limitation of the proposed CBN method is that it takes much more computational complexity and memory consumption than the baseline BN method (See Table 6). It is not clear whether the performance improvement comes from the increased computational cost.

4. It would be stronger to compare the inference time of different normalization methods.

5. This paper considers the case of small mini-batch sizes. However, the authors only investigate 5 different batch sizes, such as (32, 16, 8, 4, 2). What would happen if the authors set the batch size to 1?

6. The differences from a closely related work [3] should be discussed. This paper exploits the similar idea of computing the statistics of the current iteration by exploiting the statistics of multiple recent iterations.

7. Some typos

(1) In the experiment of Comparison of feature normalization methods, “Similar as the results ...” should be “Similar to the results ...”

(2) In the section of Effect of compensation, “The train curve ... “ should be “The training curve ...”

Some References

[1] "Differentiable Learning-to-normalize via Switchable Normalization." ICLR, 2019.
[2] "Differentiable Dynamic Normalization for Learning Deep Representation." ICML, 2019.
[3] "Double Forward Propagation for Memorized Batch Normalization." AAAI, 2018.


review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The paper tackles the problem of batch normalization (BN) instability when using small batch sizes. As the fix, authors propose the following modifications to BN:
 1) Use previous iterations statistics to virtually increase batch size
 2) Correction via Taylor series linearization of of the previous iterations to compensate weight changes

 
Paper shows in experiments on classification (ImageNet), detection (COCO) and segmentation (COCO), that the proposed method works on par with other normalization methods. Paper also shows that the proposed method does not work well for the beginning of the training and use "burn-in" period when standard BN is used instead of proposed CBN.


Questions: 

 1) Is it possible to use proposed method with batch size = 1? This could be a killer feature.
 2) Why the CBN/BN memory footprints ratios for ImageNet and COCO (Table 6) are so different? (2.75 x for ImageNet vs 1.09 for COCO).
 It seems that for ImageNet there is no benefit of using CBN, because it takes 3x more memory than BN (0.66 vs 0.24 Gb), so one can just increase batch size 3 times for standard BN. 
 3) Paper reports results for "validation set", yet "hyper-parameters were set by cross-validation". Could you please specify, how was cross-validation done? 
 4) There is a practice of gradient accumulation:  doing multiple forward-backward passes, then apply optimization step once. Could you please comment, how such practice may possible interfere with proposed weight compensation? 
 Wouldn`t it be benefitial to do similar practice for cross-iteration BN, so that no weight compensation be needed for such case?
5) What are potential use cases of CBN compared to use  BN for big batches and GN for small batches, even bs = 1?
 

Overall I think that paper is OK, but don`'t see practical applications where it is beneficial to use CBN instead of BN (for big batches) or GN (for small batches, even bs = 1).  I may be willing to increase my score, if my questions would be addressed. 


####

My concerns were addressed in the rebuttal and I am happy to increase my rating to weak accept. 

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: Paper summary: This paper proposes a new normalization technique specially designed for settings with small mini-batch sizes (where previous methods like BatchNorm are known to suffer). The approach aggregates mean/variance statistics from previous iterations, weighted based on the Taylor expansion, to get a better estimate of population statistics. The authors evaluate their approach on ImageNet classification, and object detection and instance segmentation on the COCO dataset.



The paper is clearly written and explores an interesting idea---aggregating mini-batch statistics across iterations. That being said, I am not convinced by the utility of the proposed approach since it doesn’t offer over significant benefits over prior approaches designed for the small mini-batch setting (e.g., Group Normalization)---either in terms of empirical performance, implementation complexity, or lower computational/memory requirements.

Specific comments/questions:

1. Why do the authors not include the Kalman Normalization baseline in the paper? Based on my understanding, it is also designed for the low-sample regime (and the original paper also conducts experiments on ImageNet/COCO). Also the BRN baseline is included in Table 3 for ImageNet but is missing from the COCO experiments. It is important to thoroughly compare to other normalization techniques specifically designed for this regime to clearly highlight relative benefits of the proposed approach.

2. The authors mention that they average performance across 5 trials but omit confidence intervals in their Figures/Tables. Since the difference in performance between the different approaches compared is small, I think confidence intervals should be reported to see whether improvements are within statistical error margins. In fact, for CIFAR-10 based on Table 7 in the Appendix, performance of batch renormalization, group normalization and CBN is essentially the same.

3. In Table 3, how many iterations was the BRN aggregation performed over? Was it also chosen to ensure effective number of samples was not less than 16 (like CBN). What do Figures 3 and 4 look like for BRN?

4. One thing I find interesting (the authors do not discuss this specifically) about Figure 5 in the Appendix is that both for CIFAR and COCO (the two datasets for which plots are provided), CBN helps as long as it is used before the final learning rate drop. Specifically, choosing a burn-in period as large as 120 for CIFAR and 8 for COCO is fine (in fact, probably the best) as long as you turn on CBN before the final learning rate drop. This makes me curious whether BN in the low-sample regime only suffers in terms of generalization performance in the final stages of training (compared to low-sample alternatives like GN/CBN, etc).



The authors should include other recent benchmarks (Kalman normalization and BRN in the omitted Tables) and error bars to make the change in performance clearer. It would also be interesting to see whether (and probably make the paper stronger if) alternatives like GN/KN benefit from being combined with the proposed scheme to aggregate statistics over time.

Overall, while the proposed approach is novel, its performance is comparable to prior approaches, with the disadvantage of an additional computational/memory footprint. Thus, I am not yet convinced about how useful/interesting this approach would be to the community.

review_title: Official Blind Review #1

Meta-review: This paper proposes cross-iteration batch normalization, which is a strategy for maintaining statistics across iterations to improve the applicability of batch normalization on small batches of data. 

The reviewers pointed out some strong points but also some weak points about the paper. The paper was judged to be novel and theoretically sound, and the paper was judged to be well-written. 

However, there were some doubts regarding the relevance and significance of the work. Reviewers commented on being unconvinced by the utility of the approach, it being unclear when the proposed method is beneficial, and the relative small magnitude of the empirical improvement. 

On the balance, the paper seems decent but not completely convincing. This means that with the current high competitiveness and selectivity of ICLR I unfortunately cannot recommend the manuscript for acceptance. 
Decision: Reject