Towards Stabilizing Batch Statistics in Backward Propagation of Batch Normalization
Authors: Junjie Yan
Ruosi Wan
Xiangyu Zhang
Wei Zhang
Yichen Wei
Jian Sun
Venue:  (2020)
Keywords: batch normalization
small batch size
backward propagation

Abstract: Batch Normalization (BN) is one of the most widely used techniques in Deep Learning field. But its performance can awfully degrade with insufficient batch size. This weakness limits the usage of BN on many computer vision tasks like detection or segmentation, where batch size is usually small due to the constraint of memory consumption. Therefore many modified normalization techniques have been proposed, which either fail to restore the performance of BN completely, or have to introduce additional nonlinear operations in inference procedure and increase huge consumption. In this paper, we reveal that there are two extra batch statistics involved in backward propagation of BN, on which has never been well discussed before. The extra batch statistics associated with gradients also can severely affect the training of deep neural network. Based on our analysis, we propose a novel normalization method, named Moving Average Batch Normalization (MABN). MABN can completely restore the performance of vanilla BN in small batch cases, without introducing any additional nonlinear operations in inference procedure. We prove the benefits of MABN by both theoretical analysis and experiments. Our experiments demonstrate the effectiveness of MABN in multiple computer vision tasks including ImageNet and COCO. The code has been released in https://github.com/megvii-model/MABN.

Published as a conference paper at ICLR 2020
TOWARDS STABILIZING BATCH STATISTICS IN BACK- WARD PROPAGATION OF BATCH NORMALIZATION
Junjie Yan
Y:n 1.2 Ruosi Wan Xiangyu Zhang?! Wei Zhang Yichen Wei Jian Sun
Shanghai Key Laboratory of Intelligent Information Processing School of Computer Science, Fudan University Megvii Technology.
{jiyan17, weizh (eannffuudnneeeiie edu
{wanruosi, zhangxiangyu, weiyichen, sunjian) eeeenieeeeiii.oomm
ABSTRACT
Batch Normalization (BN) is one of the most widely used techniques Deep Learning field. But its performance size. This weakness limits the usage of BN on many computer vision tasks like de- tection or segmentation, where batch size is usually small due the constraint of memory consumption. Therefore many modified normalization techniques have been proposed, which either fail to restore the performance of BN completely, or have to introduce additional nonlinear operations in inference procedure and in- crease huge consumption. this paper, we reveal that there are two extra batch statistics involved in backward propagation of BN, on which has never been well discussed before. The extra batch statistics associated with gradients also can severely affect the training of deep neural network. Based our analysis, we propose a novel normalization method, named Moving Average Batch Normal- ization (MABN). MABN completely restore the performance of vanilla BN in small batch cases, without introducing any additional nonlinear operations inference procedure. We prove the benefits of MABN by both theoretical anal- ysis and experiments. Our experiments demonstrate the effectiveness of MABN in multiple computer vision tasks including ImageNet and COCO. The code has been released in https: httpt//gtttbb/t////// wtwpuu..cooo///o com cww/ce/v///eceeecccc mww.imgiiiin wwwemmdeeella.
awfully degrade with insufficient batch
INTRODUCTION
Batch Normalization (BN) (Lfff & Szegedy 2013) is one of the most popular techniques for train- ing neural networks. It has been widely proven effective in many applications, and become the indispensable part of many state of the deep models.
Despite the success of BN, it's still challenging to utilize BN when batch size is extremely smal] The batch statisties with small batch size are highly unstable, leading to slow convergence during training and bad performance during inference. For example, in detection segmentation tasks, the batch size often limited to 1 or 2 per GPU due to the requirement of high resolution inputs complex structure of the model. Directly computing batch statistics without any modification each GPU will make performance of the model severely degrade.
To address such issues, many modified normalization methods have been proposed. They can be roughly divided into two categories: some of them try to improve vanilla BN by correcting batch statistics (Loffe 2017 [ingh & Shrivastaval 2019), but they all fail to completely restore the perfor- mance of vanilla BN; Other methods get over the instability of BN by using instance-level normal- ization (Ulyanov et al. 2016; Ba et al. 2016 Wu & He 2018), therefore models can avoid the affect
"Equal Contribution. Work was done when Junjie Yan was an intern at Megvii Technology. 'Corresponding author.
'In the context of this paper, we use "batch size/normalization batch size" to refer the number of samples used to compute statistics unless otherwise stated. We use "gradient batch size" to refer the number of samples used update weights.

---

Published as a conference paper at ICLR 2020
of batch statistics. This type of methods restore the performance in small batch cases some extent. However, instance-level normalization hardly meet industrial or commercial needs so far, for this type of methods have to compute instance-leve' statistics both in training and inference, which will introduce additional nonlinear operations in inference procedure and dramatically increase con- sumption (haa al. (2019). While vanilla BN uses the statistics computed over the whole training data instead of batch of samples when training finished. Thus BN is linear operator and can be merged with convolution layer during inference procedure. Figure ((a) shows with ResNet-50 (He et al. 2016), instance-level normalization almost double the inference time compared with vanilla BN. Therefore, it's tough but necessary task to restore the performance of BN in small batch training without introducing any nonlinear operations in inference procedure.
In this paper, we first analysis the formulation of vanilla BN, revealing there are actually not only but 4 batch statistics involved in normalization during forward propagation (FP) as well as backward propagation (BP). The additional batch statisties involved in BP are associated with gradients of the model, and have never been well discussed before. They play important role in regularizing gradients the model during BP. In our experiments (see Figure 2). variance the batch statistics associated with gradients in BP, due to small batch size, is even larger than that of the widely- known batch statistics (mean, variance of feature maps). We believe the instability of batch statistics associated with gradients is one of the key reason why BN performs poorly in small batch cases.
Based on our analysis, we propose a novel normalization method named Moving Average Batch Normalization (MABN). MABN can completely get small batch issues without introducing any nonlinear manipulation in inference procedure. The core idea of MABN is to replace batch statistics with moving average statistics. We substitute batch statistics involved in BP and FP with different type of moving average statistics respectively, and theoretical analysis is given to prove the benefits. However, we observed directly using moving average statistics as substitutes for batch statistics can't 't make make training converge in practice. We think the failure takes place due to the occasional large gradients during training, which has been mentioned in loffe (2017). To avoid training collapse, we modified the vanilla normalization form by reducing the number of batch statistics, centralizing the weights of convolution kernels, and utilizing renormalizing strategy. We also theoretically prove the modified normalization form is more stable than vanilla form.
MABN shows its effectiveness in multiple vision public datasets and tasks, including Ima- geNet (Russakovsky et al. 2015), COCO (Lin al. 2014). All results of experiments show MABN with small batch size (1 or 2) can achieve comparable performance as BN with regular batch size (see Figure ((b). Besides, it has same inference consumption vanilla BN (see Figure [(a)). We
also conducted sufficient ablation experiments to verify the effectiveness of MABN further.
7000
SN
SMW
545ON
5000
32
4000
30
3000
10g-
24
IN
Batch
(a)
(b)
Figure 1: (a) Throughout (iterations per second) in inference procedure using different Normaliza- tion methods. The implementation details can be seen in appendix B.2 (b)lmageNet classification validation error vs. batch sizes.
2 RELATED WORK
Batch normalization (BN) loffe Szegedy 2015) normalizes the internal feature maps of deep neu- ral network using channel-wise statistics (mean, standard deviation) along batch dimension. It has
2

---

Published as a conference paper at ICLR 2020
been widely proven effectively in most of tasks. But the vanilla BN heavily relies sufficient batch size in practice. To restore the performance of BN small batch cases, many normalization tech- niques have been proposed: Batch Renormalization (BRN) lofte 2017) introduces renormalizing parameters in BN to correct the batch statistics during training. where the renormalizing parame- ters are computed using moving average statistics; Unlike BRN, EvalNorm (Singh & Shrivastava 2019) corrects the batch statistics during inference procedure. Both BRN and EvalNorm can restore the performance of BN to some extent, but they all fail to get over small batch issues completely. Instance Normalization (IN) (Ulyanov et al. 2016), Layer Normalization (LN) (Ba et al., 2016), and Group normalization (GN) (Wu Hej 2018) all try avoid the effect of batch size by utiliz- ing instance level statistics. IN uses channel-win statistics per instance instead of per batch, while LN uses instance-level statisties along channel dimension. But IN and LN shows no superiority to vanilla BN in most of cases. GN divides all channels in predefined groups, and uses group-wise statistics per instance. can restore the performance of vanilla BN very well in classification and detection tasks. But it have introduce extra nonlinear manipulations in inference procedure and severely increase inference consumption, as we have pointed out Section 1. SyncBN (Peng et al, 2018) handle the small batch issues by computing the mean and variance across multiple GPUs. This method doesn't 't essentially solve the problem, and requires a lot of resource. Online Normal- ization ization[Chiley al. (2019) modifies BP by using moving average statistics, so they set batch size as I without degradation of performance, but Online Normalization still have to use instance- normalization cooperate with modification BP. so its inference efficiency much lower than original BN
Apart from operating on feature maps, some works exploit normalize the weights of convolution: Weight Standardization (Qiao et al. 2019) centralizes weight at first before divides weights by its standard deviation. It still has combine with GN to handle small batch cases.
STATISTICS IN BATCH NORMALIZATION
3.1 REVIEW OF BATCH NORMALIZATION
First of all, let's review the formulation of batch Normalization (loffe & Szegedy 2015): assume the input of BN layer denoted X E Bxp, where B denotes the batch size, p denotes number of features. In training procedure, the normalized feature maps Y at iteration t is computed as:
118 X-16.
OB,
where batch statistics μss, and of the sample mean and sample variance computed over the batch of samples B, at iteration t:
μB. B Σ oB. = 1-BEEO
1 Σ(x: - -MB.)
Besides, a pair parameters 7, 3 are used to scale and shift normalized value Y:
z=Y + B.
(3)
The scaling and shifting part is added in all normalization form by default, and will be omitted in the following discussion for simplicity.
As loffe& & Szegedy (2015) demonstrated, the batch statistics μs,, o are both involved in backward propagation (BP). We can derive the formulation of BP in BN as follows: let L denote the loss, O, denote the set of the whole learnable parameters of the model at iteration t. Given the partial
gradients
ac the partial gradients
ac is computed as
BY
8x
B.
B.
0C =
ac
IB, Y W..
(4)
ax e, σB, OY
e..s,
where- denotes element- production, I8, and M, are computed
Σ OC
OC WB = B oY, 1
98. = B OY 1..B
M Yo.
(5)
3

---

Published as a conference paper at ICLR 2020
It can be seen from (5) that 9B, and Va, are also batch statistics involved in BN during BP. But they have never been well discussed before.
3.2 INSTABILITY OF BATCH STATISTICS
According to lofte & Szegedy (2013), the ideal normalization is normalize feature maps X using expectation and variance computed over the whole training data set:
X EX
(6)
VVar[X]
But it's impractical when using stochastic optimization. Therefore, loffe & Szegedy (2015) uses
mini-batches in stochastic gradient training, each mini-batch produces estimates the mean and vari- ance of each activation. Such simplification makes it possible to involve mean and variance in BP. From the derivation in section 3.1, we can see batch statistics JAB,, oB, are the Monte Carlo (MC) estimators of population statistics E e.). Var [arl(|()) [] respectively at iteration L. Similarly, batch statistics 98,, Vs are MC estimators of population statistics E ac aY, [0,]. E Y.Y,, aY,, [Or] at it-
eration I [0r], E= aY.
[0,] are computed over the whole data set. They contain the
information how the mean and the variance of population will change as model updates, so they play
an important role make trade off between the change of individual sample and population. There- fore, it's crucial to estimate the population statistics precisely, in order to regularize the gradients of the model properly as weights update.
It's well known the variance of MC estimator is inversely proportional to the number of samples, hence the variance of batch statistics dramatically increases when batch size is small. Shoos the change of batch statistics from a specific normalization layer of ResNet-50 during training on ImageNet. Regular batch statistics (orange line) are regarded as good approximation for population statistics. We can see small batch statistics (blue line) are highly unstable, and contains notable error compared with regular batch statisties during training. In fact, the bias of gs, and Vs, in BP more serious than that of μss and o3, (see Figure 2(c) 2(d) The instability of small batch statistics can worsen the capacity of the models in two aspects: Tirstly the instability small batch statistics will make training unstable, resulting in slow convergence; Secondly the instability of small batch can produce huge difference between batch statistics and population statistics. Since the model is trained using batch statistics while evaluated using population statistics, the difference between batch statistics and population statistics will cause inconsistency between training and inference procedure, leading bad performance of the model evaluation data.
(a) liB
(b) oB
(c) gB
(d) V
Figure 2: Plot of batch statistics from layerl.f bnl in ResNet-50 during training. The formulation of these batch statistics (Hg, Z.. 9s, Vg) have been shown in Section 3.1 Blue line represents the small batch statistic (iB) 2) to compute, while orange line represents the regular batch statistics( [B] 32). The x-axis represents the iterations, while the y-axis represents the (2 norm of these statistics in each figures. Notice the mean of y and Vi is close to zero, hence 1 norm of gs and V essentially represent their standard deviation.
4 MOVING AVERAGE BATCH NORMALIZATION
Based on the discussion in Section 3.2 the key to restore the performance of BN is to solve the instability of small batch statisties. Therefore we considered two ways to handle the instability of
4

---

Published as a conference paper at ICLR 2020
small batch statistics: using moving average statistics to estimate population statistics, and reducing the number of statistics by modifying the formulation normalization.
4.1 SUBSTITUTE BATCH STATISTICS BY MOVING AVERAGE STATISTICS.
Moving average statistics seem to be a suitable substitute for batch statistics to estimate population statistics when batch is small. We consider two types of moving average statisties: simple mov- ing average statistics (SMAS and exponential moving average statistics (EMAS The following theorem shows under mild conditions, SMAS and EMAS are more stable than batch statistics:
Theorem Assume there exists a sequence of random variable (F.v.) {E) which are indepen- dent, uniformly bounded, i.e. Vt, 1&al < C, and have uniformly bounded density, Define:
S.
Σ
Ei
Fg = (1 - a))
o Ei:
(7)
i=t-m+1
i=1
where E IR.. If the sequence
{& h satisfies
35 Ve E R, lim P(s 5) P(s S sc,
(8)
8000
then we have
Var(E,) = (1 a1 a)
B(E,) = B(g) + o()),
Var(E)+ + o1);
(9)
1+a
If the sequence {E satisfies
lim sup P( < -) P(g = 1000
(10)
then we have
E(S)) = (E)) + o(1), Var(S,) Var(E) oo();
(1I)
m
only proves moving aver- age statistics have lower variance compared with batch statistics, but also reveals that with large momentum a, EMAS better than SMAS with lower variance. However, using SMAS and EMAS request different conditions: Condition (8) means the sequence of the given statistics need to weakly converge to a specific random variable. For (us,) (o J they converge to the "final" batch statistics μuB, o (when training finished), hence condition (8) satisfied, EMAS can be applied to replace 1 1 for 1::; Unfortunately { (V joe don't share the same property, EMAS is not suitable to take replace of (98, 1 { 1 However, under the assumption that learning rate is extremely small, the difference between the distribution &--1 and & is tiny, thus condition (10 is satisfied, we can use SMAS to replace {96, J (VV J In word, we can use EMAS fr.. of to replace μdB,, Gi. and use SMAS Ht, W, to replace I8,, WB, in and (4 where
The proof of theorem | can be seen in appendix A. Theorem
f4 art + (1-0)B8
aot
+ 0)
(12)
1 m
9:
GE
V.
B
(13)
m
Notice neither of SMAS and EMAS is the unbiased substitute for batch statistics, but the bias can be extremely small comparing with expectation and variance of batch statistics, which is proven by equation II in theorem I. our experiments also prove the effectiveness of moving average statisties as substitutes for small batch statistics (see Figure 3 Q in appendix B.I.
"The exponential moving average (EMA) for a series (Yr)a) is calculated as: S;= a. a (1-a)= SS "The simple moving average (SMA) for series (Y)) is calculated S, D/
V.
5

---

Published as a conference paper at ICLR 2020
Relation to Batch Renormalization Essentially, Batch Renormalization (BRN) loffe 2017)
places batch statistics μAB,, oB with EMAS fi. 0g both FP ( and BP The formulation of BRN during training is written
X MB.
Y
Y -rr-Y
(14)
OB,
08 d cl,4 88, Based analysis, BRN successfully where c/
eliminates the effect of small batch statistics μuB, and o by EMAS, small batch statistics associated with gradients 98, and V, remains during backward propagation, preventing BRN from completely restoring the performance of vanilla BN.
4.2 STABILIZING NORMALIZATION BY REDUCING THE NUMBER OF STATISTICS
To further stabilize training procedure in small batch cases, we consider normalizing feature maps X using E instead of EXX and V Var( (X). The formulation of normalization is modified as:
X
Y
Z=Y 7
(15)
XB.
where xB, = B Eb x Given ac the backward propagation is:
dc
dc
V..).
(16)
0X 0,,8 OY 15.85
The benefits of the modification seems obvious: there's only two batch statisties left during FP and BP, which will introduce less instability into the normalization layer compared with vanilla normalizing form. In fact theoretically prove the benefits of the modification by following theorem:
Theorem If the following assumptions hold:
Varlā] = o11), Var(x] = o(1);
2. Cov({
0C by w). (98, vB}) o(1):
3. By o(1):
Then we have:
V ar 0C dr modified
JC
Var[gs] 0²
< V Var
(17)
panilla
The proof can be seen in appendix A.2 According to Varloc/ax] coonittel is larger than that of V (rr/0 ftell the gap least Varlgs]/or / which mainly caused by the variance of g8/ô. So the modification essentially reduces the variance of the gradient by eliminating the batch statisties g8 during BP. Since 98, Monte Carlo estimator, the gap is inversely proportional to batch size. This can also explain why the improvement of modification is significant in small batch cases, but modified BN shows no superiority to vanilla BN within sufficient batch size (see ablation study in section 5.1).
Centralizing weights of convolution kernel Notice theorem relles on assumption 3. The vanilla normalization naturally satisfies Ey = O by centralizing feature maps, but the modified normaliza- tion doesn't it necessarily satisfy assumption 3. To deal with that, inspired by Qiao et al. (2019), we find centralizing weights W E x of convolution kernels, named as Weight Centralization (WC) can be compensation for the absence of centralizing feature maps in practice:
W Σ Was Xoutpat = (W W) Ximput:
(18)

---

Published as a conference paper at ICLR 2020
where Xnsput- Xoutput are the input and output of the convolution layer respectively. We conduct further ablation study to clarify the effectiveness of WC (see ia uppendix[B2] It shows that WC has little benefits to vanilla normalization, but it can significantly improve the performance of modified normalization. We emphasize that weight centralization is only a practical remedy for the absence of centralizing feature maps. The theoretical analysis remains future work.
Clipping and renormalizing strategy. In practice, we find directly substituting batch statistics by moving average statistics normalization layer will meet collapse during training. Therefore take use of the clipping and renormalizing strategy from BRN (Toffe 2017).
All all, the formulation of proposed method MABN is:
X
Y
Y
=r.
(19)
Xt
ac
OC e.
0C
JL
r.
Y @ v)
(20)
0Y e,
OY
0X
Xt 0Y 1,,85
0,
where the EMAS X is computed as Xt ax X Ex= XB. mta? SMAS V is defined as
a)xs,, SMAS X is defined as The renormalizing parameter is set as
13
r -=diiippaaaa
clppu/AA X1).
EXPERIMENTS
This section presents main results MABN on ImageNet (Russakovsky et al., 2015), COCO et al. 2014). Further experiment results on ImangeNet, COCO and Cityscapes (Cordts et al. 20
can seen in appendix B.2 B.3 B.4 resepectively. We also evaluate the computational overhead and memory footprint of MABN, the results is shown in appendix B.5
5.1 IMAGE CLASSIFCATION IN IMAGENET
We evaluate the proposed method on ImageNet (Russakovsky et al. 2015) classification datatsets with 1000 classes. All classification experiments conducted with ResNet-50 (He et al. 2016).
More implementation details can be found in the appendix B.2
BN
BN
BRN
MABN
(Regular)
(Small)
(Small)
(Small, m 16)
val error
23.41
35.22
30.29
23.58
A
11.81
6.88
0.17
(vs BN(Regular)
Table 1: Comparison of top-1 error rate (%) of ResNet-50 on ImageNet Classification. The gradient batch size is 32 GPU. Regular means normalization batch size is 32, while Small means normalization batch size is 2.
Comparison with other normalization methods. Our baseline BN using small (B 2) or regular (18) = 32) batch size, and BRN (loffe 2017) with small batch size. We don't ion't present the per- formance of instance-level normalization counterpart on ImageNet, because they are not linear-type method during inference time, and they also failed restore the performance of BN (over +0.5%), according to Wu & He (2018). Table shows vanilla BN with small batch size can severely worsen the performance of the model( + 11--118TT1 but there's 's still remaining far from complete recovery ( 6.88%); While MABN almost completely restore the performance of vanilla BN(+0.17%)
11.8I%)..) 51%); BRN (loffe 2017) alleviates the issue some extent,
We also compared the performance of BN, BRN and MABN when varying the batch size (see Figure T(b). BN and BRN are heavily relies the batch size of training, though BRN performs better than

---

Published as a conference paper at ICLR 2020
vanilla BN. MABN can always retain the best capacity of ResNet-50, regardless of batch size during training.
Experiment Number
Vanilla Normalization
Modified
EMAS in FP
SMAS in BP
Top-1 1 Error (%)
Normalization
!
23.41 (BN, regular)
23.53 (regular)
35.22 (BN)
30.29 (BRN)
29.68
27.03
23.58 (MABN)
Table 2: Ablation study on ImageNet Classification with ResNet-50. The normalization batch size is 2 in all experiments otherwise stated. The memory size is 16 and momentum is 0.98 when using SMAS, otherwise the momentum is 0.9. "L" means the training can't converge.
Ablation study ImageNet. We conduct ablation experiments on ImageNet clarify the con- tribution of each part of MABN (see table 2 With vanilla normalization form, replacing batch statistics in FP with EMAS (as BRN) will restore the performance to some extents(-4.933) com- paring c and 4), but there's 's still huge gap (+6.8%%, comparing ! and 4 from complete restore. Directly using SMAS in BP with BRN will meet collapse during training (5), no matter how we tuned hyperparameters. We think it's due to the instability of vanilla normalization structure in small cases, so we modify the formulation of normalization shown in section 4.2. The modified normalization even slightly outperforms BRN in small batch cases (comparing ④ and ④). How- ever, modified normalization shows no superiority to vanilla form (comparing ! and ()), which can be interpreted by the result of theorem With EMAS in FP, modified normalization significantly reduces the error rate further (comparing c and D). but still fail to restore the performance com- pletely (+3.62%, comparing !I) and @). Applying SMAS in BP finally fills the rest of gap, almost completely restore the performance of vanilla BN in small batch cases (-0.17 .comparing (I) and
5.2 DETECTION AND SEGMENTATION IN COCO FROM SCRATCHI
We conduct experiments Mask R-CNN (He et al. 2017 benchmark using Feature Pyramid NetWork(FPN) (Lin et al. 2017a) following the basic setting He et al. (2017). We train the networks from seratch (He al. 2018) for 2x times. Only the backbone contains normalization layers. More implementation details and experiment results can be seen in the appendix B.3
Apbbor
APtboz
AP
Apmask
APpask
A A.Paaas
BN
30.41
48.47
32.70
27.91
45.79
29.33
BRN
31.93
50.95
34.48
29.16
48.16
30.69
SyncBN
34.81
55.18
37.69
31.69
51.86
33.68
MABN
34.85
54.97
38.00
31.61
51.88
33.64
Table 3: Comparison of Average Precision(AP) of MCN on COCO Detection and Segmen- tation. The gradients batch size is 16. The normalization batch size of SyneBN is 16, while that of BN. BRN and MABN are both 2. The momentum of BRN and MABN are both 0.98, while the momentum of BN and SyncBN are both 0.9. The buffer size (m) is 16).
8

---

Published as a conference paper at ICLR 2020
shows the result of MABN compared with vanilla BN, BRN and SyneBN (Peng et al, 2018 It can be seen that MABN outperforms vanilla BN and BRN by a clear margin and get comparable performance with SyncBN. Quite different from Imagenet experiments, we update the parameters every single batch (with Bnorm = 2). With such a complex pipeline, MABN still achieves comparable performance SyneBN.
Table
CONCLUSION
This paper reveals the existence the batch statistics 9B and Ve involved in backward propagation of BN, and analysis their influence to training process. This discovery provides new perspective to understand why BN always fails in small batch cases. Based on our analysis, we propose MABN to deal with small batch training problem. MABN can completely restore the performance of vanilla BN in small batch cases, and is extraordinarily efficient compared with its counterpart like GN, Our experiments multiple computer vision tasks (classification, detection, segmentation) have shown the remarkable performance of MABN.

---

Published as a conference paper at ICLR 2020
ACKNOWLEDGEMENT
This research was partially supported by National Key RD Program of China (No. 201YYF00008000, Beijing Academy of Artificial Intelligence (BAAI), and NSFC under Grant No. 61473091.
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint MxXiv:667.0064500 2016.
Liang-Chisl Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions pattern analysis and machine intelligence, 40(4):
834-848, 2017.
Vitaliy Chiley, Ilya Sharapov, Atli Kosson, Urs Koster, Ryan Reece, Sofia Samaniego de la Fuente, Vishal Subbiah, and Michael James. Online Normalization for Training Neural Networks. arXiv e-prints, art, 1eXiv:1905.000
May 2019.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Sam Gross and Michael Wilber. Training and investigating residual nets, 2016. URL https: wwwp///aaccoooo/eaaa //ssttabb.cm///aabbbb//.....bb resseet..eet....
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cmn. In Proceedings of the IEEE international conference computer vision, pp. 2961-2969, 2017.
Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking imagenet pre-training. arXiv preprint arXiv: 181.88881.888, 2018.
Sergey loffe. Batch renormalization: Towards reducing minibatch dependence in batch-normalized models. In Advances neural information processing systems, pp. 1945-1953, 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pp. 448-456, 2015.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference com- puter vision and pattern recognition, pp. 2117-2125, 2017a.
Tsung- Yi Lin, Priya Goyal, Ross Girshick. Kaiming He, and Piotr Dollár. Focal loss for dense object detection. Proceedings of the IEEE international conference on computer vision, pp- 2980-2988, 2017b.
10

---

Published as a conference paper at ICLR 2020
Chao Peng, Tete Xiao, Zeming Li, Yuning Jiang, Xiangyu Zhang, Kai Jia, Gang Yu, and Jian Sun. Megdet: large mini-batch object detector. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6181-6189, 2018.
Siyuan Qiao, Huiyu Wang, Chenxi Liu, Wei Shen, and Alan Yuille. Weight standardization. arXiv preprint arXiv:1903. 10520, 2019.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-255 2015.
Wenqi Shao, Tianjian Meng. Jingyu Li, Ruimao Zhang, Yudian Li, Xiaogang Wang, and Ping Luo. Ssn: Learning sparse switchable normalization via sparsestmax. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 443-451, 2019.
Saurabh Singh and Abhinav Shrivastava. Evalnorm: Estimating batch normalization statistics for evaluation. arXiv preprint arXiv: 1eee:190400,,,, 2019.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing in- gredient for fast stylization. arXiv preprint 1607.0020,. 2016.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 3-19, 2018.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2890, 2017.

---

Published as a conference paper at ICLR 2020
A SKETCH OF PROOF
A.1 PROOF OF THEOREM
If the condition 8 is satisfied, i.e. (E weakly converge to Since (6.)2= has uniformly bounded density, we have:
lim EEE,
Eg
(21)
1-000
lim Var[er] Var(c] 17000
(22)
Since {f are independently, hence we have:
Var Varl(1 a)
i=1
(1 a)² 2
Var(El
(23)
(1 a)² 2
1) V (rrl + (1 a1=
Σ a 2(t (Var((s) - Var(s))
1=1
21 (1
V
Varl + 01)
1 1+a
as + co. Hence
O has been proven.
If the condition 10
satisfied. Since ( is uniformly bounded, then 3C RR V. 15l < C.
As + 0o, We have
/
TPt (x)dd - 1 S (xp(x)da
EG B5,l =
xEl-COC
x(p (c(2))
JEEl-C.C
(xFFi1 (x) [o - J
- dx(
JZel-CC
(24)
$ JxEl-C,01 [F-1(1)) Fe(x))drr
2C suu 1 Fx(x)]
20 sup P - <P) < x)]
o(1)
Similarly, we have
/
RCP EC?
(=p)=1(0)-P<())dd) - (x()))dz
J
122
(F(1(()) F((x))1_ Jss-ca 2x 2x(F(-1(x) dz
{ /xe[-c.c)
(25)
2|zl|F(1(x)) =F(x)
< 40 sup 1x) - F((x))
a
o(1)
Therefore combining
and
we have
| |arl
Varl < EEC_1
E ((E,,1)) -(E{))²1
(26)
= o(1)
12

---

Published as a conference paper at ICLR 2020
For fixed memory size m, as t - oc, we have
Var(S:) = Varl
1=0
1
Σ
m
(27)
1
(Var + (())
m
1=0
Vark +
Therefore,
11 has been proven.
A.2 PROOF OF THEOREM
Without loss of generality, given the backward propagation of two normalizing form of a single input x with batch B:
ac
1 0C Oy 9s Vol.
Oc
-
1 0C
y Vsl
(28)
8x
dr modified
8 8y
vanilla
where IB. V are the batch statistics, and o, x are the EMAS, defined before. We omitted the subscript for simplicity. Then the variance of partial gradients w.r.t. inputs is written
Oc
1 1.0c 0 dy
V ar Oz
1
Var
y- -.lll]
(29)
vanilla
Oc
dC
Var
y Vsl + Var [98] + 2Cov
y v 98
(30)
gz
Dy
dc
Var
y VB] + Var [
(31)
Dy
UC
Var [gs
Varl
VBl
(32)
X
Oy
62
DL
Var [gs]
V
(33)
dx
modi fied
where 30 is satisfied due to assumption 1. The variance so small that can be regarded as
fixed number; (3I) satisfied because
dc
DC
Coul Oy
M.. sts] Coul Oy .B]] -Couly Vs [6.9s]
(34)
dc
Coul
[b]
EyyV EgB)] Byy BB
(35)
Dy
Due to assumption 2, the correlation between individual sample and batch statistics is close to 0, hence we have
Coul dc
Oy" gel
(36)
E EygBs(g - EgB)] = EyE EgB)
(37)
EgyBal EyEVs
(38)
Besides, EE is close to 0 according assumption 3, hence
OL
Coul
Dy y Us 98] = 0.
(39)
32
is satisfied due the definition of x and o, have
x² 82 + i!2
(40)
Similar to ô, the variance of X is also too small that X can be regarded as a fixed number due to
assumption 1, (33) is satisfied.
13

---

Published as a conference paper at ICLR 2020
B EXPERIMENTS
B.1 STATISTICS ANALYSIS
We analyze the difference between small batch statistics ( = 2) and regular batch statistics ( 32) with the modified formulation normalization (15) shown in Section 4.2.
1e-4
Small!!
Small/21 Regular33)
Ragg
6.8
c.6
0.4
6.0
620000
100000
600003
(a) X
(b) Vg
Figure 3: Plot of batch statistics from layer1.0..e .0.bn1 1 in ResNet-50 with a modified structure during
training. The formulation of these batch statistics [x W/5) is shown in section 4.2, 3.1 respectively. Blue line represents the small batch statistic (i5) = 2). while orange line represents the regular batch statistics (B3) 32). We use the small batch statistics update the network parameters.
h--6
17.5
Regular
0.5
15.0
0.8
10.0
0.5
0.4
2.5
4MA12430
Regular:321
100000
200000
300000
400000
500000
000000
100000
200000
500000
400000
500000
6500co
heratonne
Buralfort
X
(b) Vg
Figure 4: Plot of batch statistics from layerl.0.bll 1.0. in ResNet-50 with MABN. The formulation of these batch statistics x2- Wg) is shown in section 4.2, 3.1 respectively. Blue line represents the SMA batch statistic(++30), while orange line represents the regular batch statistics(32). We the moving average batch statistics to update the network parameters.
Figure illustrates the change of small batch statistics and regular batch statistics FP and BP respectively. The variance of small batch statistics is much higher than the regular one. However, when we use SMAS as a approximation for regular batch statistics, the gap between SMAS and regular batch statistics is not obvious shown in Figure 4
14

---

Published as a conference paper at ICLR 2020
B.2 EXPERIMENTS ON IMAGENET
Implementation details. All experiments on ImageNet are conducted across 8 GPUs. We train models with gradient batch size of B, 32 images per GPU. To simulate small batch training, we split the samples on each GPU into / / groups where [B] denotes the normalization batch size. The batch statistics are computed within each group individually.
All weights from convolutions are initialized as He et al. (2015). We use 1 to initialize all y and initialize all /3 normalization layers. We use weight decay of 10-4 for all weight layers including and 3 (following Wu & He 2018)) We train 600, 000 iterations (approximately equal to 120 epoch when gradient batch size 256) for all models, and divide the learning rate by at 150, 000, 300, 000 and 450, 000 iterations. The data augmentation follows Gross & Wilber (0016). The models are evaluated by top-1 classification error on center crops of 224 x 224 pixels in the validation set. In vanilla BN or BRN, the momentum 0.9, in MABN, the momentum
0.98.
Additional ablation studies. Table 4 shows the additional ablation results. We test all possible combination of all three kinds of statistics (SMAS, EMAS, BS) in FP and BP. The experiments results strongly prove our theoretical analysis in section 4.3. Besides, we verify the necessity of centralizing weights with modified normalization form.
Experiment number
w/o centralizing feature maps X
Centralizing
FP statistics
BP statistics
Top-1 Error (%)
weights W
EMAS
SMAS
23.58(MABN)
SMAS
SMAS
26.63
EMAS
EMAS
24.83
EMAS
BS
27.03
v
BS
BS
29.68
EMAS
SMAS
25.45
EMAS
BS
29.57
BS
BS
32.95
BS
BS
35.22
BS
BS
34.27
BS
BS
23.35(regular)
Table 4: Further ablation study on ImageNet with ResNet-50. The normalization batch size is
2 in all experiments. The buffer size (m) is 16 and momentum is 0.98 when using SMA statistics, otherwise the momentum is 0.9. BS means vanilla batch statistics.
B.3 EXPERIMENTS ON COCO
Implementation details. We train the Mask-RCNN pipeline from scratch with MABN. We train the model on GPUs, with 2 images per GPU. We train our model using COCO 2014 train and trainval35k dataset. We evaluate the model on COCO 2014 minival dataset. We set the momentum a = 0.98 for all MABN layers. We report W the standard COCO merics ApMoor A AAP for bounding box detection and Apmosk APmask APmask for instance segmentation. Other basic settings follow He et al. (2017)
MABN used on heads. We build mask-renn baseline using a Feature Pyramid (et)orr(FPNN)Lin al. 2017a) backbone. The base model is ResNet-50. We train the models for 2x iterations. We use 4conv Ifc instead of 2fc as the box head. Both backbone and heads contain normalization layers. We replace all normalization layers in each experiments. While training models with MABN, use batch statistics normalization layers on head during first 10,000 iterations. Table T shows the result. The momentum set to 0.98 in BRN and MABN.
15

---

Published as a conference paper at ICLR 2020
A Apbbor
A Apbboz
APbioz
Apmaak
A.mask
A Pmmak
BN
32.38
50.44
35.47
29.07
47.68
30.75
BRN
34.07
52.66
37.12
30.98
50.03
32.93
SyncBN 36.81
56.23
40.08
33.11
53.46
35.28
MABN
36.50
55.79
40.17
32.69
52.78
34.71
Table 5: Comparision of Average Precision(AP) of Mask-RCNN COCO Detection and Segmen-
lation. The gradients batch size is 16. The normalization batch size of SyncBN is 16, while that of BN, BRN, MABN are both the buffer size (m) of MABN 32.
Training from pretrained model. We compare the performance of MABN and SyncBN when training model based ImageNet pretrained weights for 2x iterations. The results shown in Table
Abbbor
A.Pbooz
A.
Apmask
A.maask
A APmask
SyncBN
38.25
57.81
42.01
34.22
54.97
36.34
MABN
38.42
58.19
41.99
34. 12
55.10
36.12
Table Comparision of Average Precision(AP) Mask-RCNN on COCO Detection and Segmen- tation. The gradients batch size is 16. The normalization batch size of SyncBN is 16, while that of BN, BRN, MABN are both 2, the buffer size (m) of MABN is 32.
Training from scratch for one-stage model. We also compare MABN and SyncBN based on one-stage pipeline. We build on retinanet(Lin et al. 2017b) benchmark. We train the model from scratch for 2x iterations. The results are shown in Table
Abbboz
Abbbor
Abbor
SyncBN
29.80
46.21
31.47
MABN
29.52
45.69
31.14
Table 7: Comparison of Average Precision(AP) of retinanet COCO Detection. The gradients
batch size is 16. The normalization batch size SyncBN is 16, while that of MABN 2.
All experiment results shows MABN can get comparable as SyneBN, and significantly outperform BN on COCO.
B.4 SEMANTIC SEGMENTATION CITYSCAPES
We evaluate semantic segmentation in Cityscapes(Cordts et al. 2016). It contains 5,000 high qual- ity pixel-level finely annotated images collected from 50 cities different seasons. We conduct experiments PSPNET baseline and follow the basic settings mentioned in Zhao et al. (2017).
For fair comparison, our backbone network is ResNet-101 as in Chen et al. (001)) Since we cen- tralize weights of convolutional kernel to MABN, have to re-pretrain our backbone model on Imagenet dataset. During fine-tuning process, we linearly increase the learning rate for 3 epoch (558 iterations)
first. Then follow the "poly" learning schedule as iZhao et al. (2017). Table 8 shows the result of MABN compared with vanilla BN, BRN and SyncBN. The buffer size (m) of MABN is 16, the modementum of MABN and BRN is 0.98.
Since the statistics (mean and variance) is more stable in a pre-trained model than a random initial- ized one, the gap between vanilla BN and SyneBN is not significant (+1.41%). However, MABN
16

---

Published as a conference paper at ICLR 2020
pretrain Top-1
mloU
BN
21.74
77.11
BRN
21.74
77.30
SyncBN
21.74
78.52
MABN
21.70
78.20
Table 8: Results Cityscapes testing set.
still outperforms vanilla BN by a clear (argin.(+1..9%% Besides, BRN shows no obvious superior- ity to vanilla BN(+0.19%) Cityscapes dataset.
B.5 COMPUTATIONAL OVERHEAD
We compare the computational overhead and memory footprint of BN, GN and MABN. We use maskrenn with resnet50 and FPN as benchmark. We compute the theoretical FLOPS during infer- ence and measure the inference speed when a single image x3x224> x 224) goes through the backbone (resnet50 + FPN). We assume BN and MABN can absorbed in convolution layer during infer- ence. GN can not be absorbed in convolution layer, so its FLOPS is larger than BN and MABN. Besides GN includes division and sqrt operation during inference, therefore it's much slower than BN and MABN during inference time.
We also monitor the training process of maskrenn on COCO (8 GPUs, images per GPU), and show its memory footprint and training speed. Notice we have not optimized the implementation of MABN, so its training speed is a little slower than BN and GN.
FLOPS (M)
Memory (GB)
Training Speed (iter/s)
Inference Speed (iter/s)
BN
3123.75
58.875
2.35
12.73
GN
3183.28
58.859
2.22
6.34
MABN
3123.75
60.609
1.81
12.73
Table 9: Computational overhead and memory footprint of BN, GN and MABN.
17


Reviews:
[{"review_id": "rJxHNwjTKr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper extends recently proposed BatchRenormalization (BRN) technique which uses exponential moving average (EMA) statistics in forward and backward passes of BatchNorm (BN) instead of vanilla batch statistics. Motivation of the work is to stabilize training neural networks on small batch size setup. Authors propose to replace EMA in backward pass by simple moving average (SMA) and show that under some assumptions such replacement reduces variance. Also they consider slightly different way of normalization without centralizing features X, but centralizing convolutional kernels according to Qiao et al. (2019).\n\nConcerns:\nChanging batch statistics with moving averages in Eq. (4) and Eq. (16) may introduce bias in stochastic gradients. Authors do not reflect this problem in the work.\nAssumption (3) from Theorem 3 does not hold in practice since authors centralize weights not features. Authors do not study the influence of this on the performance of the method.\nAuthors’ main motivation is to study small batch size experimental setup which is important in segmentation and detection, but they don’t include main competitor (BRN) in these experiments.\n\nOverall, the proposed method has a lack novelty and thorough comparison against BRN. Therefore, I would suggest rejecting the current version.\n--------------------------------------------------------\nUpdate after author rebuttal\n\nThank you for your clarification. Below I provide an updated review for the paper.\n\nThe paper proposes the improvement of batch normalization techniques for the case of small batch size. Authors reveal new statistics used in gradient calculation of original BatchNorm and show their instability in case of small batch size. Thereafter they improve the method by reducing the number of statistics in backward pass resulting in more stability and performance increase. The authors provide a good experimental study on the influence of individual parts of the method. However, I still have concerns about the strictnesses of theorems assumptions and interpretability of their results in practice (i.e., o(1) biases in Theorem 1, o(1) assumptions in Theorem 2).\n\nOverall, almost all of my concerns were justified. Therefore I increase my score to 6.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Bke_CWyaFH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new approach for batch-normalization. Standard approaches are sensitive to the batch size, because small batches will lead to unstable statistics. So when the mini-batch is small, the performance can drop significantly. The paper addresses this issue by analyzing extra statistics in the batch normalization and introducing moving average statistics, weights centralization and a slightly modified normalization. The proposed method does not require large batch sizes and nonlinear operations, but still maintain the robustness. The theoretical analysis and guarantees are provided as well. Experiments on typical datasets demonstrate the effectiveness of the proposed trick.\n\nOverall, the idea is interesting to me. The work is solid in both theory and practice. Hopefully the proposed scheme has the potential to fundamentally enhance the training of deep neural networks. ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BkeYyKYnKS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper provides a new method to deal with the small batch size problem of BN, called MABN. Compared to BRN, MABN has two main contributions: 1) find two statistics, g and ψ, in BP to apply moving average operation without introducing too much overhead; 2) reduce the number of statistics of BN via centralizing weight for better stability.\n\nThough, I still have several concerns:\n1.\tYou mentioned that “centralizing weights ... to satisfy the zero mean assumption” in Section 4.2. In other words, given E(W-mean(W))=0, E(X_{output})=E(W-mean(W))E(X_{input})=0. This equation holds only when W-mean(W) and X_{input} are irrelevant. However, model weights are updated based on the input and the loss function during training. Do you have any proof to ensure W-mean(W) and X_{input} are irrelevant?\n\n2.\tYour “Modified Structure” contains two operations, centralizing weights and reducing BN’s statistics. As much as I know, weight standardization can significantly improve BN’s performance, but the effect of weight centralization, i.e. “part” of weight standardization, is unclear yet. Therefore, I think a vanilla BN with weight centralization should also be included in ablation study. This experiment can help us better understand the pure effectiveness of reducing BN’s statistics.\n\n3.\tI think the results on COCO should be discussed in detail since object detection is an important task in computer vision and suffers from the small batch size regime. I wonder if it’s possible to compare MABN with SyncBN/GN on a higher baseline, such as finetuning from ImageNet pretrained model or training from scratch with at least 6x scheduler.\n\n4.\tWhat about the FLOPS, memory footprint and practical speed? These results would affect the application value of MABN. It would also be great if you can release the code, which would upgrade my rating a lot.\n\n5.\tWhat is the value of m for SMA? Is this value changed in different datasets? This would affect the performance of the proposed method. \n\n\nAlthough there are several issues not addressed by the authors (the assumption of this paper  \"W-mean(W) and X_{input} are irrelevant\"; why not using ImageNet pre-trained model on COCO), I keep my initial rating of weak accept. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work introduces Moving Average Batch Normalization (MABN) method to address performance issues of batch normalization in small batch cases. The method is theoretically analyzed and empirically verified on ImageNet and COCO.
Some issues were raised by the reviewers, such as restrictive nature of some of the assumptions in the analysis as well as performance degradation due lack of centralizing feature maps. Nevertheless, all the reviewers found the contributions of this paper interesting and important, and they all recommended accept.

Decision: Accept (Poster)