Cross-Iteration Batch Normalization
Authors: Zhuliang Yao
Yue Cao
Shuxin Zheng
Gao Huang
Stephen Lin
Jifeng Dai
Venue:  (2020)
Keywords: batch normalization
small batch size

Abstract: A well-known issue of Batch Normalization is its significantly reduced effectiveness in the case of small mini-batch sizes. When a mini-batch contains few examples, the statistics upon which the normalization is defined cannot be reliably estimated from it during a training iteration. To address this problem, we present Cross-Iteration Batch Normalization (CBN), in which examples from multiple recent iterations are jointly utilized to enhance estimation quality. A challenge of computing statistics over multiple iterations is that the network activations from different iterations are not comparable to each other due to changes in network weights. We thus compensate for the network weight changes via a proposed technique based on Taylor polynomials, so that the statistics can be accurately estimated and batch normalization can be effectively applied. On object detection and image classification with small mini-batch sizes, CBN is found to outperform the original batch normalization and a direct calculation of statistics over previous iterations without the proposed compensation technique.

Under review as a conference paper at ICLR 2020
CROSS-ITERATION BATCH NORMALIZATION
Anonymous authors
Paper under double-blind review
ABSTRACT
A well-known issue of Batch Normalization is its significantly reduced effec- tiveness in the case of small mini-batch sizes. When mini-batch contains few examples, the statistics upon which the normalization is defined cannot be reliably estimated from it during a training iteration. To address this problem, we present Cross-Iteration Batch Normalization (CBN), in which examples from multiple recent iterations are jointly utilized enhance estimation quality. A challenge of computing statistics over multiple iterations is that the network activations from different iterations are not comparable to each other due to changes in network weights. We thus compensate for the network weight changes via a proposed technique based on Taylor polynomials, so that the statistics can be accurately estimated and batch normalization can be effectively applied. On object detection and image classification with small mini-batch sizes, CBN is found to outperform the original batch normalization and a direct calculation of statistics over previous iterations without the proposed compensation technique.
INTRODUCTION
Batch Normalization (BN) (Ioffe & Szegedy, 2015) has played a significant role in the success of deep neural networks. It was introduced to address the issue of internal covariate shift, where the distribution of network activations changes during training iterations due to the updates of network parameters. This shift is commonly believed to be disruptive to network training, and BN alleviates this problem through normalization of the network activations by their mean and variance, computed over the examples within the mini-batch at each iteration. With this normalization, network training can be performed at much higher learning rates and with less sensitivity to weight initialization.
In BN, it is assumed that the distribution statistics for the examples within each mini-batch reflect the statistics over the full training set. While this assumption is generally valid for large batch sizes, breaks down in the small batch size regime (Peng et al., 2018; Wu & He, 2018; Ioffe, 2017), where noisy statistics computed from small sets of examples can lead to a dramatic drop in performance. This problem hinders the application of BN to cemmory-consuming tasks such object detection (Ren et al., 2015; Dai et al., 2017), semantic segmentation (Long et al., 2015; Chen et al., 2017) and action recognition (Wang et al., 2018b), where batch sizes are limited due memory constraints.
Towards improving estimation of statistics in the small batch size regime, alternative normalizers have been proposed. Several of them, including Layer Normalization (LN) (Ba al., 2016), Instance Normalization (IN) (Ulyanov et al., 2016), and Group Normalization (GN) (Wu & He, 2018), compute the mean and variance over the channel dimension, independent of batch size. Different channel-wise normalization techniques, however, tend to be suitable for different tasks, depending on the set of channels involved. On the other hand, synchronized BN (SyneBN) (Peng et al., 2018) yields consistent improvements by processing larger batch sizes across multiple GPUs. These gains in performance come the cost of additional overhead needed for synchronization across the devices.
A seldom explored direction for estimating better statistics is to compute them over the examples from multiple recent training iterations, instead of from only the current iteration as done previous techniques. This can substantially enlarge the pool of data from which the mean and variance are obtained. However, there exists an obvious drawback to this approach, in that the activation values from different iterations are not comparable to each other due to the changes in network weights. As shown in Figure 1, directly calculating the statistics over multiple iterations, which we refer to as Naive CBN, results in lower accuracy.

---

Under review as a conference paper at ICLR 2020
Temporal window size [k) CBN
Figure I: Top-1 classification accuracy batch sizes per iteration. The base model is a ResNet-18 (He et al. 2016) trained on ImageNet (Rus- sakovsky et al.. 2015). BN (loffe & Szegedy, 2015)'s s accuracy drops rapidly when the batch size is reduced. GN (Wu & He, 2018) exhibits stable performance but underperforms BN on adequate batch sizes. CBN compensates for the redueed batch size per GPU by exploiting approx- imated statistics from recent ilerations (Temporal window size denotes how many recent iters are utilized for statistics computation). CBN shows relatively stable performance over different batch sizes. Naive CBN does not work well, which directly calculates statistics from recent iterations without compensation.
70
Top-1 accuracy
69
68
67
BN
GN
66
Nekva CBN CBN
66
32
16
batch size per iter
In this paper, we present method that compensates for the network weight changes among iterations, so that examples from preceding iterations can be effectively used improve batch normalization. Our method, called Cross- Batch Normalization (CBN), is motivated by the observation that network weights change gradually, instead of abruptly, between consecutive training iterations, thanks to the iterative nature of Stochastic Gradient Descent (SGD). As result, the mean and variance of examples from recent iterations can be well approximated for the current network weights via low-order Taylor polynomial, defined gradients of the statisties with respect the network weights. The compensated means and variances from multiple recent iterations are averaged with those of the current iteration to produce better estimates of the statisties.
In the small batch size regime, CBN leads to appreciable performance improvements over the original BN, as exhibited in Figure 1. The superiority of our proposed approach is further demonstrated through more extensive experiments on ImageNet classification and object detection on COCO. These gains are obtained with negligible overhead, as the statistics from previous iterations have already been computed and Taylor polynomials are simple to evaluate. With this work, it is shown that cues for batch normalization can successfully be extracted along the time dimension, opening a new direction for investigation.
2 RELATED WORK
The importance of normalization in training neural networks has been recognized for decades (LeCun et al., 1998). In general, normalization can be performed on three components: input data, hidden activations, and network parameters. Among them, input data normalization is used most commonly because of its simplicity and effectiveness (Sola & Sevilla, 1997; LeCun et al., 1998).
After the introduction of Batch Normalization (loffe & Szegedy, 2015), the normalization of activa- tions has become nearly as prevalent. By normalizing hidden activations by their statistics within each mini-batch, BN effectively alleviates the vanishing gradient problem and significantly speeds up the training of deep networks. To mitigate the mini-batch size dependency of BN, number of variants have been proposed, including Layer Normalization (LN) (Ba et al., 2016), Instance Normalization (IN) (Ulyanov et al., 2016), Group Normalization (GN) (Wu & He, 2018), and Batch Instance Normalization (BIN) (Nam & Kim, 2018). The motivation of LN is to explore more suitable statistics for sequential models, while IN performs normalization in a manner similar to BN but with statistics only for each instance. GN achieves a balance between IN and LN, by dividing features into multiple groups along the channel dimension and computing the mean and variance within each group for normalization. BIN introduces a learnable method for automatically switching between normalizing and maintaining style information, enjoying the advantages of both BN and IN on style transfer tasks. Cross- CrUss-GPU Batch Normalization (CGBN or SyncBN) (Peng et al., 2018) extends BN across multiple GPUs for the purpose of increasing the effective batch size. Though providing higher accuracy, it introduces synchronization overhead to the training process. Kalman Normalization (KN) (Wang et al., 2018a) presents a Kalman filtering procedure for estimating the statistics for a network layer from the layer's 's observed statistics and the computed statistics of previous layers.
Batch Renormalization (BRN) (Ioffe, 2017) is the first attempt to utilize the statistics of recent iterations for normalization. It does not compensate for the statistics from recent iterations, but rather it weights the importance of statistics from distant iterations. This nown-weighting heuristic, however, does not make the resulting statistics "correct", the statisties from recent iterations not of the current network weights. BRN can be deemed as a special version of our Naive CBN baseline (without Taylor polynomial approximation), where distant iterations are dein-weig.
2

---

Under review as a conference paper at ICLR 2020
Iteration
Iteration
Iteration
BN
Normalize, Afline transform
Normalize, Affine transform
Normalize, Afline transform
moan, variance (H-22.0.. 6
mean, variance (Hh-1. 6_11)
meen, varience 8,)
Compensated variance ( 8
Compensated mean, variance (Ra-1 θm1))
Compensated mean, variance (2, 8,)
Normalize, Affine transform
Normalize, Affine transform
Normalize, Aftine transform
CBN
Figure 2:
Illustration of BN and the proposed Cross-Iteratin Batch Normalization (CBN).
Recent work have also investigated the normalization of network parameters. In Weight Normaliza- tion (WN) (Salimans & Kingma, 2016), the optimization of network weights is improved through a reparameterization of weight vectors into their length and direction. Weight Standardization (WS) (Qiao et al., 2019) instead reparameterizes weights based on their first and second moments for the purpose of smoothing the loss landscape of the optimization problem. To combine the advantages of multiple normalization techniques, Switchable Normalization (SN) (Luo et al., 2018) and Sparse Switchable Normalization (SSN) (Shao et al., 2019) make use of differentiable learning to switch among different normalization methods.
The proposed CBN takes an activation normalization approach that aims to mitigate the mini-batch dependency of BN. Different from existing techniques, it provides way to effectively aggregate statistics across multiple training iterations.
3 METHOD
3.1 REVISITING BATCH NORMALIZATION
The original batch normalization (BN) (Ioffe & Szegedy, 2015) whitens the activations of each layer by the statistics computed within a mini mini-batch. Denote 6, and (6,) as the network weights and the feature response of certain layer for the i-th example in the 1-th mini-batch. With these values, BN conducts the following normalization:
xr,(0,) (e)
(e)
V Or (0.)2
where (e) is the whitened activation with zero mean and unit variance, E is a small constant added for numerical stability, and (0.)) and (,()) are the mean and variance computed for all the examples from the current mini-batch, i.e.,
(0,)
m Σ (e).
(2)
(0))
(x, (0)))
())) V ( (000))-((00)))) (0))
(3)
where v, - - 1 x,, and denotes the number of examples in the current mini-batch The whitened activation (.) further undergoes linear transform with learnable weights, to increase its expressive power:
e,) = (0) +B,
(4)
where y and B are the learnable parameters (initialized to y= 1 and B 0 in this work).
When the batch size m is small, the statistics μ. (0,) and (0,)) become noisy estimates of the training set statistics, thus degrading the effects of batch normalization. In the ImageNet classification task
3

---

Under review as a conference paper at ICLR 2020
for which the BN module was originally designed, batch size of 32 is typical. However, for other tasks requiring larger models and/or higher image resolution, such as object detection, semantic segmentation and video recognition, the typical batch size may be as small as 1 or 2 due to GPU memory limitations. The original BN becomes considerably less effective such cases.
3.2 LEVERAGING STATISTICS FROM PREVIOUS ITERATIONS
To address the issue of BN with small mini-batches, naive approach is to compute the mean and variance over the current and previous iterations. However, the statistics (4-r(0,-e) and (0-) computed under the network weights 0,-r, making them obsolete for the
of the (1- ---th iteration current iteration. As a consequence, directly aggregating statistics from multiple iterations produces inaccurate estimates of the mean and variance, leading significantly worse performance.
We observe that the network weights change smoothly between consecutive iterations, due to the nature of gradient-based training. This allows us to approximate (4-((0) and Vi-((0,) from the readily available 44-((8,r) and (0-= via a Taylor polynomial, i.c.,
M-r = 4-r d -
(0, 0,-r) + O(10, 0-rl1
(5)
j6,
dv; (0-- r)
(0- (0,-))
(0. 0,-c) + 0(1 0--r1
(6)
d
are gradients of the statistics with respect to the network weights, and O 0 =1= denotes higher-order terms of the Taylor polynomial, which can be omitted since the first-order term dominates when (6, 0--r) is small.
where
and
In Eq. (5) and Eq. (6), the gradients (0-=))000 and dv; +)/d0,-r cannot be precisely determined at a negligible cost because the statistics // (0-=) and v/_ (0,-r) for node at the I-th network layer depend on all the network weights prior to the /-th layer, i.e., d (0_==))))//00----0 + and d - (0-+)) =) + O for r I, where 0- denotes the network weights at the r-th layer. Only when / can these gradients be derived in closed form efficiently.
aul(a) and (v)
Empirically, we find that as the layer index decreases (r S l), the partial gradients rapidly diminish. These reduced effects of network weight changes at earlier layers on the activation distributions in later layers may perhaps be explained by the reduced internal covariate shift of BN Motivated by this phenomenon, which is studied in Appendix C, we propose to truncate these partial gradients at layer I.
Thus, we further approximate Eq. (5) and Eq. (6) by
duf + (
(0-z) (0/ -
(0) = (0,-))
00
(v
v v/
(0) 07-))
20
A naive implementation of 0
and 0v involves computational over- head of OiC' x C' x K), where C2 and ct-l denote the channel dimension of the /-th layer and the 1)-th layer, respectively, and K denotes the kernel size of e/_.- Here we find that the operation can be implemented efficiently in OCC x ct-1 x K), thanks to the averaging over feature responses of μı and v. See Appendix for the details.
3.3 CROSS-ITERANE
BATCH NORMALIZATION
After compensating for network weight changes, we aggregate the statistics of the k I most recent iterations with those of the current iteration to obtain the statistics used in CBN:
k-1 4/ (0),
(e)
Σ
(9)
0-
(0))
Σ
max [v/- - (V),
(10)
r=0
4

---

Under review as a conference paper at ICLR 2020
batch size per iter
#examples for statistics
Norm axis
IN
#bs/GPU #GPU
(spatial)
LN
#bs/GPU #GPU
(channel, spatial)
GN
#bs/GPU #GPU
(channel group, spatial)
BN
#bs/GPU #GPU
#bs/GPU
(batch, spatial)
syncBN
#bs/GPU #GPU
#Ps/GPU #GPU
(batch, spatial, GPU)
CBN
#bs/GPU * #GPU
#Ps/GPU * temporal window (batch, spatial, iteration)
Table I: Comparison of different feature normalization methods. #bs/GPU denotes batch size per GPU.
(0)) = / (+) p. (0))
(11)
where (0,) and (0.) are computed from Eq. (7) and Eq. (8). In Eq. (10), (0,) ) is determined
from the maximum of (0,) and (0,) in each iteration because (0,) u (0,) should hold for valid statistics but may be violated by Taylor polynomial approximations in Eq. (7) and
Eq. (8). Finally, P (0)) and (0.) are applied normalize the corresponding feature responses
(xt, (O)) at the current iteration:
(0,) = -1,(0)))1(x(()
(12)
V (0,)
With CBN, the effective number of examples used to compute the statistics for the current iteration is k times large as that for the original BN. In training, the loss gradients are backpropagated to the network weights and activations at the current iteration, i.e., 6 and (0)) Those of the previous iterations are fixed and do not receive gradients. Hence, the computation cost of CBN in back-propagation is the same as that of BN.
Replacing the BN modules in network by CBN leads to only minor increases computational overhead and memory footprint. For computation, the additional overhead mainly comes from computing the partial derivatives (04==((0==//00 and (0)= in relation to the overhead of the whole network. For memory, the module requires access to the statistics (0,-))) and (v==+(0.-=)} and the gradients (00m=1(0_2)/8))_11
which is insignificant
and (dv--(0--))de))_ M computed for the most recent compared to the rest of the memory consumed in processing the input examples. The additional computation and memory of CBN is reported for our experiments in Table 6.
iterations, which is also minor
A key hyper-parameter in the proposed CBN is the temporal window size, k, of recent iterations used for statistics estimation. A broader window enlarges the set of examples, but the example quality becomes increasingly lower for more distant iterations, since the differences in network parameters 6, and 0 become more significant and are compensated less well using low-order Taylor polynomial. Empirically, we found that CBN is effective with window size up to k = 8 in variety of settings and tasks. The only trick is that the window size should be kept small at the beginning of training, when the network weights change quickly. Thus, introduce burn-in period of length Thum-in for the window size, where k = and CBN degenerates to the original BN. In our experiments, the burn-in period is set 25 epochs on ImageNet image classification and 3 epochs on COCO object detection by default. Ablations on this parameter are presented in the Appendix.
Table 1 compares CBN with other feature normalization methods. The key difference among these approaches is the axis along which the statistics are counted and the features are normalized. The previous techniques are all designed to exploit examples from the same iteration. By contrast, CBN explores the aggregation of examples along the temporal dimension. As the data utilized by CBN lies in a direction orthogonal that of previous methods, the proposed CBN could potentially be combined with other feature normalization approaches to further enhance statistics estimation in certain challenging applications.
EXPERIMENTS
4.1 IMAGE CLASSIFICATION ON IMAGENET
Experimental settings. ImageNet (Russakovsky et al., 2015) is a benchmark dataset for image classification, containing 1,28M training images and 50K validation images from 1000 classes. We
5

---

Under review as a conference paper at ICLR 2020
follow the standard setting in (He et al., 2015) to train deep networks the training set and report the single-crop top-1 accuracy on the validation set. Our preprocessing and augmentation strategy strictly follows the GN bascline (Wu & He, 2018). We use weight decay of 0,0001 for all weight layers, including y and B. We train standard ResNet-18 for 100 epochs on 4 GPUs, and decrease the learning rate by the cosine decay strategy (He et al., 2019). We use the average over 5 trials for c.g. group size of GN, are carefully tuned via cross-validation. ResNet-18 with BN is our base model. To compare with other normalization methods, we directly replace BN with IN, LN, GN, BRN, and our proposed CBN.
all results. All
Comparison of feature normalization meth- ods. We compare the performance of each nor- malization method with a normal batch size, 32, in Table 2. With sufficient data for reliable statis- ties, BN easily reaches the highest top-1 accu- racy. Similar as the results in previous papers
IN
LN
GN BN
CBN
Top-1 accuracy 64.4
67.9
68.9
70.2
70.2
Table 2: Top-1 accuracy of feature normalization methods using R8 on ImageNet.
(Wu & He, 2018), IN and LN achieve significantly worse performance than BN. GN works well on image classification, but still has a small degradation of 1.2% compared with BN. Over all the methods, our CBN is the only one that is able to achieve comparable accuracy with BN, as it converges to the procedure of BN as the batch size becomes larger.
Sensitivity to batch size. We compare the be- havior of CBN, original BN (loffe & Szegedy, 2015), GN (Wu & He, 2018), and BRN (Ioffe, 2017) at the same number of images per GPU on ImageNet classification. For CBN, the recent iterations are utilized so as to ensure that the number of effective examples is no fewer than 16. For BRN, the settings strictly follow the original paper. We adopt a learning rate of 0.1 for the batch size of 32, and linearly scale the learning rate by N/32 for batch size of N.
batch size per GPU
32
16
4
2
BN
70.2
70.2
68.4
63.1
55,9
GN
68.9
69.0
68.9
69.0
69.1
BRN
70.1
68.5
68.2
67.9
60.3
CBN
70.2
70.2
70.1
69.8
69.3
Table 3: Top-1 accuracy of feature normalization meth- ods with different batch sizes using R8 as the base model on ImageNet.
The results are shown in Table 3. For the original BN, its accuracy drops noticeably as the number of images per GPU is reduced from 32 to 2. BRN suffers significant performance drop well. GN maintains its accuracy by utilizing the channel dimension but not batch dimension. For CBN, its accuracy holds by exploiting the examples of recent iterations. Also, CBN outperforms GN by 0.9% on average top-1 accuracy with different batch sizes. This is reasonable, because the statistics computation of CBN introduces uncertainty caused by the stochastic batch sampling like in BN, but this uncertainty is missing in GN which results in some loss of regularization ability.
4.2 OBJECT DETECTION AND INSTANCE SEGMENTATION ON COCO
Experimental settings. COCO (Lin et al., 2014) is chosen as the benchmark for object detection and instance segmentation. Models trained the COCO 2017 train split with 118k images, and evaluated on the COCO 2017 validation split with 5k images. Following the standard protocol in (Lin al., 2014), the object detection and instance segmentation accuracies are measured by the mean average precision (mAP) scores at different intersection-over-uiion (IoU) overlaps at the box and the mask levels, respectively.
Following (Wu& & He, 2018), Faster R-CNN (Ren et al., 2015) and Mask R-CNN (He et al., 2017) with FPN (Lin al., 2017) are chosen as the baselines for object detection and instance segmentation, respectively. For both, the 2fc box head is replaced by a 4conv Ifc head for better use of the normalization mechanism (Wu & He, 2018). The backbone networks are ImageNet pretrained ResNet-50 (default) or ResNet- 101, with specific normalization. Finetuning is performed on the COCO train set for 12 epochs on 4 GPUs by SGD, where each GPU processes 4 images (default). Note that the mean and variance statistics in CBN are computed within each GPU. The learning rate is initializeed to be 0.02 x N/16 for batch size per GPU of N, and is decayed by a factor of 10 at the 9-th and the 11-th epochs. The weight decay and momentum parameters are set to 0.0001 and 0.9, respectively. We use the average over 5 trials for all results. All hyper- parameters, e.g. group size of GN, are carefully tuned via cross-valice volosavali....n....
6

---

Under review as a conference paper at ICLR 2020
backbone
box head
Aphbox
50
AP
APp
APM
APp
fixed BN
36,9
58.2
39.9
21.2
40.8
46.9
fixed
BN
36.3
57.3
39.2
39.7
47.3
fixed BN
syneBN
37.7
58.5
41.1
22.0
40.9
49.0
fixed BN
GN
37.8
59,0
40.8
22.3
41.2
48.4
fixed BN
CBN
37.7
59.0
40.7
22.1
40.9
48.8
BN
BN
35.5
56.4
38.7
19.7
38.8
47.3
SyneBN
syneBN
37.9
58.5
41.1
21.7
41.5
49.7
GN
GN
37.8
59.1
40.9
22.4
41.2
49.0
CBN
CBN
37.3
57.7
39.3
21.9
40.8
48.2
Table 4: Results of feature normalization methods Faster R-CNN with FPN and ResNet50 COCO.
Backbone
method
norm
Aphhox
APhbox
APp
APphox
App
App
GN
37.8
59.0
40.8
22.3
48.4
R50+FPN
Faster RCNN
syncBN
37.7
58.5
41.1
22.0
40.9
49.0
CBN
37.7
59.0
40.7
22.1
40,9
48.8
GN
39.3
60.6
42.7
22.5
42.5
51.3
RI01+FPN
Faster RCNN
syneBN
39.2
43.0
22.2
42.9
51.6
CBN
39.2
60.0
42.6
22.3
42.6
51.1
Aphhox
APhbox
AP
Apmask
AP
Apmask
GN
38.6
59.8
41.9
35.0
56.7
37.3
R50+FPN
Mask RCNN
synсBN
38.5
58.9
42.3
34.7
56.3
36.8
CBN
38.5
59.2
42.1
34.6
56.4
36.6
GN
40.3
61.2
44.2
36.6
58.5
39,2
R101+FPN
Mask RCNN
syncBN
40.3
60.8
44.2
36.0
57.7
38.6
CBN
40.1
60.5
44.1
35.8
57.3
38.5
Table 5: Results with stronger backbones COCO object detection and instance segmentation.
As done in (Wu & He, 2018), we experiment with two settings where the normalizers are activated only the task-specific heads with frozen BN at the backbone (default), the normalizers activated at all the layers except for the early conv1 and conv2 stages in ResNet.
Normalizers at backbone and task-specific heads. We further study the effect of different normal- izers on the backbone network and task-specific heads for object detection on COCO. CBN, original BN, syncBN, and GN included in the comparison.
Table 4 presents the results. When BN is frozen in the backbone and no normalizer is applied at the head, the Apbbox score is 36.9%. When the original BN is applied the head only and at both the backhone and the head, the accuracy drops to 36.3% and 35.5%, respectively. For CBN, the accuracy 37.7% and 37.3% these two settings, respectively. Without any synchronization across GPUs, CBN can achieve comparable performance with syncBN and GN, showing the superiority of the proposed approach. Unfortunately, due the accumulation of approximation error, CBN observes 0.4% decrease in Apbbox when replacing frozen BN with CBN in the backbone. Even so, CBN still outperforms the variant with unfrozen BN backbone by 1.8%.
Instance segmentation and stronger backbones. Results of object detection (Faster R-CNN (Ren al., 2015)) and instance segmentation (Mask R-CNN (He et al., 2017)) with ResNet-50 and ResNet- 101 are presented in Table 5. We can observe that our proposed CBN achieves performance comparable to syncBN and GN with R50 and R101 as the backbone on both Faster R-CNN and Mask R-CNN, which demonstrates that CBN is robust and versatile to various deep models and tasks.
4.3 ABLATION STUDY
Effeet of temporal window size We conduct this ablation on ImageNet image classification and COCO object detection, with each GPU processing 4 images. Figure 3 presents the results. When k 1, only the batch from the current iteration utilized; therefore, CBN degenerates to the original BN. The accuracy suffers due the noisy statistics on small batch sizes. As the window size k gradually increases, more examples from recent iterations are utilized for statistics estimation, leading to greater accuracy. Accuracy saturates at = 8 and even drops slightly. For more distant iterations, the network weights differ more substantially and the Taylor polynomial approximation becomes less accurate.
7

---

Under review as a conference paper at ICLR 2020
On the other hand, it is empirically observed that the original BN satu- rates at a batch size of 16 or 32 for
Temporal window size (k)
Temporal window CBN
numerous applications (Peng et al.,
Top- accuracy 8 8
CBN SymcEN
37.5
2018; Wu & He, 2018), indicating that the computed statistics become accu- rate. Thus, temporal window size of
Navee CEN
CBN
38.5
k = min( 6s per GPU 1.8) is suggested.
8 16 #examppes for statistics
8
16
32
dexamples for statistics
Effect of compensation. To study this, we compare CBN with I) naive baseline where statistics from recent iterations are directly aggregated with- out compensation via Taylor polyno- mial, referred to as Naive CBN; and 2) the original BN applied with the same effective example number as CBN (i.e., its batch size per GPU is set to the product of the batch size per GPU and the temporal window size of CBN), which does not require any compensation and serves an upper performance bound.
Figure 3: The effect of temporal window size (k) on ImageNet (ResNet-18) and COCO (Faster R-CNN with ResNet-50 and FPN) with #bs/GPU = 4 for CBN and Naive CBN. Naive CBN directly uti- lizes statistics from recent iterations, while BN uses the equivalent #examples as CBN for statistics computation.
The experimental results are also presented in Figure 3. CBN clearly surpasses Naive CBN when the previous iterations are included. Actually, Naive CBN fails when the temporal window size grows to k = 8 as shown in Figure 3(a), demonstrating the necessity of compensating for changing network weights over iterations. Compared with the original BN upper bound, CBN achieves similar accuracy at the same effective example number. This result indicates that the compensation using low-order Taylor polynomial by CBN effective.
Figure 4 presents the train and test curves of CBN, Naive CBN, BN- and BN- on ImageNet, with 4 im- ages per GPU and a temporal window
Train Error (%)
80-
Test (16)
50
size of 4 for CBN, Naive CBN, and BN-bs4, and 16 images per GPU for BN-bs 16. The train curve of CBN is close to BN- at the beginning. and approaches BN-bs16 666 at the end. The reason is that we adopt a burn-in pe- riod avoid the disadvantage of rapid statistics change at beginning of train- ing. The gap between the train curve of Naive CBN and CBN shows that Naive CBN cannot even reach a good convergence on the training set. The test curve of CBN is close to BN-bs 16 at the end, while Naive CBN exhibits considerable jitter. All these phenomena indicate the effectiveness of our proposed Taylor compensation.
40-
Epoch 100
20
Epoch
Figure 4: Training and test curves for CBN, Naive CBN, and BN on ImageNet, with batch size per GPU of 4 and temporal window size k = 4 for CBN. Naive CBN, and BN-bs4, and batch size per GPU of 16 for 111. Thus, the plot of 166 the ideal bound.
Additional computational overhead and memory footprint. As the inference stage of CBN is the same as BN, we only need to com- pare the computational overhead and memory footprint at the training time, shown in Table 6. The extra computational overhead mainly in- cludes calculations of the statistics' respective gradients, Taylor compensations, and averaging operations. For the extra memory, the statistics (u and v). their respective gradients, and the network parameters (0, ... ((-(k-)) of previ- ous iterations are all stored when applying CBN. From these results, the additional overhead of CBN seen to be minor.
ImageNet
COCO
BN
CBN
BN
CBN
Base model
5.96 5.96
5155.1
5809.7
GFLOPs Taylor expansion
0.21
654.6
Total
5.96 6.17
5155.1
5809.7
Feature map
0.15 0.45
14.1
15.1
Memory (GB)
Net params
0.09 0.21
0.3
0.6
Total
0.24 0.66
14.4
15.7
Table 6: Comparison of theoretical FLOPS and mem- ory footprint between CBN and original BN in both forward and backward passes at training time.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint vrXiv:1607.000 06750, 2016.
Liang-Chieh Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4):834-848, 2017.
Jifeng Dai, Haozhi Qi, Yuwen Xiong, Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In Proceedings of the IEEE international conference on computer vision, pp. 764-773, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. arXiv preprint arXiv: 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference computer vision, pp. 2961-2969, 2017.
Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of tricks for image classification with convolutional neural networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Sergey loffe. Batch renormalization: Towards reducing minibatch dependence in batch- normalized models. In Advances in Neural Information Processing Systems, pp. 1945-1953, 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference Machine Learning, pp. 448-456, 2015.
Yann LeCun, Léon Bottou, Genevieve B OIT, and Klaus- Müller. Efficient backprop. In Neural Networks: Tricks of the Trade, this book an outgrowth of 1996 NIPS workshop, pp. 9-50. Springer-Veriiee 1998.
Tsung-Yo Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Tsung-Yr Yi Lin, Piotr Dollar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431-3440, 2015.
Ping Luo, Jiamin Ren, and Zhanglin Peng. Differentiable Sarming-to-normalle via switchable normalization. arXiv preprint 1rXiv:1806.1077,, 2018
Hyconseob Nam and Hyo-Eun Kim. Batch- Batch-instance normalization for adaptively style-invariant neural networks. In Advances in Neural Information Processing Systems, pp. 2563-2572, 2018.
Chao Peng, Tete Xiao, Zeming Li, Yuning Jiang, Xiangyu Zhang, Kai Jia, Gang Yu, and Jian Sun. Megdet: A large mini-batch object detector, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6181-6189, 2018
Siyuan Qiao, Huiyu Wang, Chenxi Liu, Wei Shen, and Alan Yuille. Weight standardization. arXiv preprint arXiv: 1903. 10520, 2019.

---

Under review as a conference paper at ICLR 2020
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pp. 2015.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-252 2015.
Tim Salimans and Diederik Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In Advances in Neural Information Processing Systems, pp. 901-909, 2016.
Wenqi Shao, Tianjian Meng, Jingyu Li, Ruimao Zhang, Yudian Li, Xiaogang Wang, and Ping Luo. Ssn: Learning sparse switchable normalization via sparsestmax. arXiv preprint UIXiv:103.03399. 2019.
J Sola and Joaquin Sevilla. Importance of input data normalization for the application of neural networks to complex industrial problems. IEEE Transactions on nuclear science, 44(3):1464-1465 1468, 1997.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint orXiv:1607.08022 2016.
Guangrun Wang, Ping Luo, Xinjiang Wang, Liang Lin, et al. Kalman normalization: Normalizing
internal representations across network layers. In Advances in Neural Information Processing Systems, pp. 21-31, 2018a.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7794-7803, 201 8b.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European Conference Computer Vision (ECCV), pp. 3-19, 2018.
10

---

//statistics on the current iteration Under review as a conference paper at ICLR 2020
A
ALGORITHM OUTLINE
Algorithm presents an outline of our proposed Cross-Iteration Batch Normalization (CBN).
Algorithm I: Cross-Iteration Batch Normalization(CBNN)
Input: Feature responses of network node of the /-th layer at the 1-th iteration (0) network weights {6/-r)=-0- 14-1
statistics (0)-)() and (v/-r(0:==)) ))> and gradients (d -=
14-1 and {dvi-(00-=) 1-1 from most recent & iterations
Output: (,,, (x,,
(6,)) t 1 ( (,, - D x7,
for TE [I,...,k] k) do
dul 18 (0/ -))))
(1)((0)) + 14- 6 +)+
//approximation from recent iterations
00
+)+ (0: + (0 (0-
dv
(8 (0) e'_+)
//approximation from recent iterations
20
end
(0)) - 1
/avveraging recent iterations //validation and averaging over recent iterations
(1)))) t [ max
(0))) + (0.) - (0))
(0) = (,00)BB'(0))
//normalize
V (8,
10 (0)) + ,0, +B
l/scale and shift
B EFFICIENT IMPLEMENTATION OF (p/_r(0/- AND (v==
Let C¹ and Ct-1 denote the channel dimension of the /-th layer and the (1- 1)-th layer, respectively, and K denotes the kernel size of 0'-,- 1²_ and v are thus of C' dimensions in channels, and e'_ is C x C'-1 K dimensional tensor. A naive implementation of ( and (0==((o)) involves computational overhead of O(C' C' x C'-1 K). Here we find that the operations of μ and can be implemented efficiently in O(C'-1 K) and O(C' x Cl-l K). respectively, thanks to the averaging of feature responses in JI and v.
Here we derive the efficient implementation of ( That of (0-=)(0) about the same. Let us first simplify the notations a bit. Let μr' and 0¹ denote (0,,(0,=z) and 0 respectively, by removing the irrelevant notations for iterations. The element-wise computation in the forward pass can be computed as
(13)
where u/ denotes the j-th channel in μ², and xf,, denotes the j-th channel in the i-th example. xf, is computed as
K - fnn Yitoffset(k).'
X
- R=1
(14)
where and enumerate the input feature dimension and the convolution kernel index, respectively, offset(k) denotes the spatial offset in applying the k-th kernel, and y/-1 is the output the ( 1)-th layer.
11

---

Under review as a conference paper at ICLR 2020
The element-wise calculation of àu' 2p//00l /aol RR''xx'x into consideration:
is follows, taking Eq. (13) and Eq. (14)
p
a 20 j...
m E 1
20
(15)
Q 1 D. m
ΣK 0!
jank y (it)ttset()ju
2
offssett)))
rdd
Thus, laol 1iggp. takes non-zero values only when j j This operation can be implemented
efficiently in O(C'-) K). Similarly, the calculation of /ve/ can be obtained in O x C'-1 x
C ADDITIONAL EXPERIMENTS
CIFAR-10 is selected for the experiments in this section. It consists of 50k training images and 10k test images from 10 classes. We train the standard ResNet-18 18 for 160 epochs on one GPU by SGD. The momentum and weight decay pa- rameters are set to 0.9 and 0.0001, respectively. We experiment with batch sizes of 32, 16, 8, 4, and 2 images per GPU. The learning rate is scaled linearly different batch sizes, following the practice in (Peng et al., 2018). The initial learning rate is 0.025 NN 32 for batch size per iteration of N. The learning rate is divided by 10 at epochs 80 and 120. The images are of 32 32 pixels with per-image standardization in both training and inference. Random flipping is applied in training.
Trials
2
4
5
overall
BN-bs16
95.3
95.0
95.4
95.3
95.26:10.14
BN-bs4
93.6
93.5
93.6
93.6
93.8
93.62.10.11
BRN
94.4
94.7
94.5
94.3
94.4
94.46++.15
GN
94.3
94.0
94.4
94.2
94.4
94.26+00117
CBN
95.0 95.0 94.6 95.0
95.2 94.96+0.22
Table 7: Top-1 accuracy of ResNet-18 with different trials on CIFAR-10. The batch size per GPU 16 and 4 for BN-bs16 and the other methods, respectively.
We report the results BN, BRN, GN, CBN with five trials on CIFAR-IO, shown in Table 7. CBN has the smallest gap with BN-bs 6 compared to BN-bs4, BRN, and GN. This result is consistent with previous experiments on ImageNet and COCO. Also the std is tiny, indicating that performance on CIFAR-10 10 is stable enough for some empirical studies.
96.5
37.5
Top-1 accuracy
94.5
Apbbox
36.5
83.5
CBN
CBN BNN
93
10
30
50
70
90
110
130
150
3
5
7
9
11
Burn-in period (epoch)
Burn-in period (epoch)
(a) CIFAR-10
(b) COCO
Figure 5: Results of different burn-in periods (in epochs) on CBN, with batch size per iteration of 4, on CIFAR-10 and COCO.
On the burn-in period length Tourn-in- We further study the influence of varying the burn-in period length Thum-ins at images per GPU on both CIFAR-10 image classification (ResNet- 18) and COCO object detection (Faster R-CNN with FPN and ResNet-50).
Figure 5(a) and 5(b) present the results. When the burn-in period is too short, the accuracy suffers. This is because at the beginning of training, the network weights change rapidly, causing the
12

---

Under review as a conference paper at ICLR 2020
compensation across iterations to be less effective. On the other hand, the accuracy is stable for wide range of burn-in periods Tburn-in that are not too short.
On the effect of using more than one layer. The efficient implementation is no longer applicable when more than one layer of compensation adopted. Therefore, we only conduct a two-layer experiment of ResNet-18 overhead. CBN using two layers for compensation achieves 95.0 on CIFAR-10 (batch size=4, k=4), which is comparable CBN using only one layer. As using more layers does not further improve performance but consumes more FLOPs, we adopt one-layer compensation on CBN in practice.
CIFAR-10 in consideration of the heavy extra memory and computational
0x
0x
00
Oul
1.
60
861
oul
00
Ovl
0.75
Ratlo
0.5
Raiio
0.5
0.25
0.25
0
0
40
60
80
100
120
140
160
20
40
60
80
100
120
140
160
Epoch
(a) The gradients of μ w.r.t. network weights
(b) The gradients of v w.r.t network weights
Figure 6: Comparison gradients of statistics w.r.t current layer vs. that w.r.t previous layers on CIFAR-
On the gradients from different layers. The key assumption in Eq. (7) and Eq. (8) is that for a node at the /-th layer, the gradient of its statistics with respect to the network weights at the
I-th layer is larger than that weights from the prior layers, i..., || ( ||F
du (0 ||F 00_
ae/
and 2v ||F 20! examine this assumption empirically for networks trained on CIFAR-10 image recognition.
2u dv (0-=))
|F for < where ||- ||e denotes the Frobenius norm. Here
computed ratio of duf (44"((0-)) 06
(0--)
Figure
presents
the
||r and
20
ov (0-r) that
dv ((h-))
|F for S 1, at different training epochs. duf (8-z)
The results suggest
20
dyf 0
dv (0-s)
d6 -
dvl _v 00
|| and ||
||5
|p hold for < I, thus
08
00
validating the approximation Eq. (7) and Eq. (8).
duh 10
Tul (a
We also study the gradients non-ResNet models.
1F
|| and
28
0e
àv ( 08 ?
|| dv (0__1 |r on VGG-16 and Inception V3 are (0.22 and 0.46) and (0.17 and 0.38), 2
respectively, which similar to ResNet-18 18 (0.13 and 0.40), indicating that the assumption should also hold for VGG and the Inception series.
13


Reviews:
[{"review_id": "rygzIvETtS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a novel Cross-Iteration Batch Normalization (CBN) to address the limitation of BN in the case of small mini-batch sizes. Different from existing methods, CBN exploits the statistics cross different iterations to obtain more accurate estimates of the data statistics. Specifically, the proposed CBN uses Taylor polynomials to approximate the statistics using the information of multiple recent iterations. The experiments on both image classification and object detection tasks demonstrate the effectiveness of the proposed method.\n\nPlease see my detailed comments below.\n\nPositive points:\n\n1. The authors propose a novel method that exploits the information from different iterations to estimate the normalization statistics more accurately. \n\n2. Unlike existing methods, the authors use Taylor polynomials to cast the information of previous iterations into the current iteration. The proposed method is theoretically sound.\n\n3. The experiments on both image classification and object detection tasks show the superiority of the proposed CBN over the considered baseline methods.\n\nNegative points:\n\n1. The authors only conduct experiments on a single image classification model ResNet-18. More deep architectures should be considered in the experiments, e.g., DenseNet, MobileNet, etc.\n\n2. Several state-of-the-art normalization methods should be compared in the experiment of \" sensitivity to batch size\", including SN [1] and DN [2].\n\n3. One limitation of the proposed CBN method is that it takes much more computational complexity and memory consumption than the baseline BN method (See Table 6). It is not clear whether the performance improvement comes from the increased computational cost.\n\n4. It would be stronger to compare the inference time of different normalization methods.\n\n5. This paper considers the case of small mini-batch sizes. However, the authors only investigate 5 different batch sizes, such as (32, 16, 8, 4, 2). What would happen if the authors set the batch size to 1?\n\n6. The differences from a closely related work [3] should be discussed. This paper exploits the similar idea of computing the statistics of the current iteration by exploiting the statistics of multiple recent iterations.\n\n7. Some typos\n\n(1) In the experiment of Comparison of feature normalization methods, “Similar as the results ...” should be “Similar to the results ...”\n\n(2) In the section of Effect of compensation, “The train curve ... “ should be “The training curve ...”\n\nSome References\n\n[1] \"Differentiable Learning-to-normalize via Switchable Normalization.\" ICLR, 2019.\n[2] \"Differentiable Dynamic Normalization for Learning Deep Representation.\" ICML, 2019.\n[3] \"Double Forward Propagation for Memorized Batch Normalization.\" AAAI, 2018.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "SJx3HnoiFr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper summary: This paper proposes a new normalization technique specially designed for settings with small mini-batch sizes (where previous methods like BatchNorm are known to suffer). The approach aggregates mean/variance statistics from previous iterations, weighted based on the Taylor expansion, to get a better estimate of population statistics. The authors evaluate their approach on ImageNet classification, and object detection and instance segmentation on the COCO dataset.\n\n\n\nThe paper is clearly written and explores an interesting idea---aggregating mini-batch statistics across iterations. That being said, I am not convinced by the utility of the proposed approach since it doesn’t offer over significant benefits over prior approaches designed for the small mini-batch setting (e.g., Group Normalization)---either in terms of empirical performance, implementation complexity, or lower computational/memory requirements.\n\nSpecific comments/questions:\n\n1. Why do the authors not include the Kalman Normalization baseline in the paper? Based on my understanding, it is also designed for the low-sample regime (and the original paper also conducts experiments on ImageNet/COCO). Also the BRN baseline is included in Table 3 for ImageNet but is missing from the COCO experiments. It is important to thoroughly compare to other normalization techniques specifically designed for this regime to clearly highlight relative benefits of the proposed approach.\n\n2. The authors mention that they average performance across 5 trials but omit confidence intervals in their Figures/Tables. Since the difference in performance between the different approaches compared is small, I think confidence intervals should be reported to see whether improvements are within statistical error margins. In fact, for CIFAR-10 based on Table 7 in the Appendix, performance of batch renormalization, group normalization and CBN is essentially the same.\n\n3. In Table 3, how many iterations was the BRN aggregation performed over? Was it also chosen to ensure effective number of samples was not less than 16 (like CBN). What do Figures 3 and 4 look like for BRN?\n\n4. One thing I find interesting (the authors do not discuss this specifically) about Figure 5 in the Appendix is that both for CIFAR and COCO (the two datasets for which plots are provided), CBN helps as long as it is used before the final learning rate drop. Specifically, choosing a burn-in period as large as 120 for CIFAR and 8 for COCO is fine (in fact, probably the best) as long as you turn on CBN before the final learning rate drop. This makes me curious whether BN in the low-sample regime only suffers in terms of generalization performance in the final stages of training (compared to low-sample alternatives like GN/CBN, etc).\n\n\n\nThe authors should include other recent benchmarks (Kalman normalization and BRN in the omitted Tables) and error bars to make the change in performance clearer. It would also be interesting to see whether (and probably make the paper stronger if) alternatives like GN/KN benefit from being combined with the proposed scheme to aggregate statistics over time.\n\nOverall, while the proposed approach is novel, its performance is comparable to prior approaches, with the disadvantage of an additional computational/memory footprint. Thus, I am not yet convinced about how useful/interesting this approach would be to the community.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BJg9S4Y3YH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper tackles the problem of batch normalization (BN) instability when using small batch sizes. As the fix, authors propose the following modifications to BN:\n 1) Use previous iterations statistics to virtually increase batch size\n 2) Correction via Taylor series linearization of of the previous iterations to compensate weight changes\n\n \nPaper shows in experiments on classification (ImageNet), detection (COCO) and segmentation (COCO), that the proposed method works on par with other normalization methods. Paper also shows that the proposed method does not work well for the beginning of the training and use \"burn-in\" period when standard BN is used instead of proposed CBN.\n\n\nQuestions: \n\n 1) Is it possible to use proposed method with batch size = 1? This could be a killer feature.\n 2) Why the CBN/BN memory footprints ratios for ImageNet and COCO (Table 6) are so different? (2.75 x for ImageNet vs 1.09 for COCO).\n It seems that for ImageNet there is no benefit of using CBN, because it takes 3x more memory than BN (0.66 vs 0.24 Gb), so one can just increase batch size 3 times for standard BN. \n 3) Paper reports results for \"validation set\", yet \"hyper-parameters were set by cross-validation\". Could you please specify, how was cross-validation done? \n 4) There is a practice of gradient accumulation:  doing multiple forward-backward passes, then apply optimization step once. Could you please comment, how such practice may possible interfere with proposed weight compensation? \n Wouldn`t it be benefitial to do similar practice for cross-iteration BN, so that no weight compensation be needed for such case?\n5) What are potential use cases of CBN compared to use  BN for big batches and GN for small batches, even bs = 1?\n \n\nOverall I think that paper is OK, but don`'t see practical applications where it is beneficial to use CBN instead of BN (for big batches) or GN (for small batches, even bs = 1).  I may be willing to increase my score, if my questions would be addressed. \n\n\n####\n\nMy concerns were addressed in the rebuttal and I am happy to increase my rating to weak accept. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes cross-iteration batch normalization, which is a strategy for maintaining statistics across iterations to improve the applicability of batch normalization on small batches of data. 

The reviewers pointed out some strong points but also some weak points about the paper. The paper was judged to be novel and theoretically sound, and the paper was judged to be well-written. 

However, there were some doubts regarding the relevance and significance of the work. Reviewers commented on being unconvinced by the utility of the approach, it being unclear when the proposed method is beneficial, and the relative small magnitude of the empirical improvement. 

On the balance, the paper seems decent but not completely convincing. This means that with the current high competitiveness and selectivity of ICLR I unfortunately cannot recommend the manuscript for acceptance. 
Decision: Reject