Year: 2020

Abstract:
Batch Normalization (BN) is one of the most widely used techniques in Deep Learning field. But its performance can awfully degrade with insufficient batch size. This weakness limits the usage of BN on many computer vision tasks like detection or segmentation, where batch size is usually small due to the constraint of memory consumption. Therefore many modified normalization techniques have been proposed, which either fail to restore the performance of BN completely, or have to introduce additional nonlinear operations in inference procedure and increase huge consumption. In this paper, we reveal that there are two extra batch statistics involved in backward propagation of BN, on which has never been well discussed before. The extra batch statistics associated with gradients also can severely affect the training of deep neural network. Based on our analysis, we propose a novel normalization method, named Moving Average Batch Normalization (MABN). MABN can completely restore the performance of vanilla BN in small batch cases, without introducing any additional nonlinear operations in inference procedure. We prove the benefits of MABN by both theoretical analysis and experiments. Our experiments demonstrate the effectiveness of MABN in multiple computer vision tasks including ImageNet and COCO. The code has been released in https://github.com/megvii-model/MABN.

heading: 1 INTRODUCTION
text: Batch Normalization (BN) (Ioffe & Szegedy, 2015) is one of the most popular techniques for training neural networks. It has been widely proven effective in many applications, and become the indispensable part of many state of the art deep models.
Despite the success of BN, it’s still challenging to utilize BN when batch size is extremely small1. The batch statistics with small batch size are highly unstable, leading to slow convergence during training and bad performance during inference. For example, in detection or segmentation tasks, the batch size is often limited to 1 or 2 per GPU due to the requirement of high resolution inputs or complex structure of the model. Directly computing batch statistics without any modification on each GPU will make performance of the model severely degrade.
To address such issues, many modified normalization methods have been proposed. They can be roughly divided into two categories: some of them try to improve vanilla BN by correcting batch statistics (Ioffe, 2017; Singh & Shrivastava, 2019), but they all fail to completely restore the performance of vanilla BN; Other methods get over the instability of BN by using instance-level normalization (Ulyanov et al., 2016; Ba et al., 2016; Wu & He, 2018), therefore models can avoid the affect
∗Equal Contribution. Work was done when Junjie Yan was an intern at Megvii Technology. †Corresponding author. 1In the context of this paper, we use ”batch size/normalization batch size” to refer the number of samples used to compute statistics unless otherwise stated. We use ”gradient batch size” to refer the number of samples used to update weights.
of batch statistics. This type of methods can restore the performance in small batch cases to some extent. However, instance-level normalization hardly meet industrial or commercial needs so far, for this type of methods have to compute instance-level statistics both in training and inference, which will introduce additional nonlinear operations in inference procedure and dramatically increase consumption Shao et al. (2019). While vanilla BN uses the statistics computed over the whole training data instead of batch of samples when training finished. Thus BN is a linear operator and can be merged with convolution layer during inference procedure. Figure 1(a) shows with ResNet-50 (He et al., 2016), instance-level normalization almost double the inference time compared with vanilla BN. Therefore, it’s a tough but necessary task to restore the performance of BN in small batch training without introducing any nonlinear operations in inference procedure.
In this paper, we first analysis the formulation of vanilla BN, revealing there are actually not only 2 but 4 batch statistics involved in normalization during forward propagation (FP) as well as backward propagation (BP). The additional 2 batch statistics involved in BP are associated with gradients of the model, and have never been well discussed before. They play an important role in regularizing gradients of the model during BP. In our experiments (see Figure 2), variance of the batch statistics associated with gradients in BP, due to small batch size, is even larger than that of the widelyknown batch statistics (mean, variance of feature maps). We believe the instability of batch statistics associated with gradients is one of the key reason why BN performs poorly in small batch cases.
Based on our analysis, we propose a novel normalization method named Moving Average Batch Normalization (MABN). MABN can completely get over small batch issues without introducing any nonlinear manipulation in inference procedure. The core idea of MABN is to replace batch statistics with moving average statistics. We substitute batch statistics involved in BP and FP with different type of moving average statistics respectively, and theoretical analysis is given to prove the benefits. However, we observed directly using moving average statistics as substitutes for batch statistics can’t make training converge in practice. We think the failure takes place due to the occasional large gradients during training, which has been mentioned in Ioffe (2017). To avoid training collapse, we modified the vanilla normalization form by reducing the number of batch statistics, centralizing the weights of convolution kernels, and utilizing renormalizing strategy. We also theoretically prove the modified normalization form is more stable than vanilla form.
MABN shows its effectiveness in multiple vision public datasets and tasks, including ImageNet (Russakovsky et al., 2015), COCO (Lin et al., 2014). All results of experiments show MABN with small batch size (1 or 2) can achieve comparable performance as BN with regular batch size (see Figure 1(b)). Besides, it has same inference consumption as vanilla BN (see Figure 1(a)). We also conducted sufficient ablation experiments to verify the effectiveness of MABN further.

heading: 2 RELATED WORK
text: Batch normalization (BN) (Ioffe & Szegedy, 2015) normalizes the internal feature maps of deep neural network using channel-wise statistics (mean, standard deviation) along batch dimension. It has
been widely proven effectively in most of tasks. But the vanilla BN heavily relies on sufficient batch size in practice. To restore the performance of BN in small batch cases, many normalization techniques have been proposed: Batch Renormalization (BRN) (Ioffe, 2017) introduces renormalizing parameters in BN to correct the batch statistics during training, where the renormalizing parameters are computed using moving average statistics; Unlike BRN, EvalNorm (Singh & Shrivastava, 2019) corrects the batch statistics during inference procedure. Both BRN and EvalNorm can restore the performance of BN to some extent, but they all fail to get over small batch issues completely. Instance Normalization (IN) (Ulyanov et al., 2016), Layer Normalization (LN) (Ba et al., 2016), and Group normalization (GN) (Wu & He, 2018) all try to avoid the effect of batch size by utilizing instance level statistics. IN uses channel-wise statistics per instance instead of per batch, while LN uses instance-level statistics along channel dimension. But IN and LN shows no superiority to vanilla BN in most of cases. GN divides all channels in predefined groups, and uses group-wise statistics per instance. It can restore the performance of vanilla BN very well in classification and detection tasks. But it have to introduce extra nonlinear manipulations in inference procedure and severely increase inference consumption, as we have pointed out in Section 1. SyncBN (Peng et al., 2018) handle the small batch issues by computing the mean and variance across multiple GPUs. This method doesn’t essentially solve the problem, and requires a lot of resource. Online Normalization Chiley et al. (2019) modifies BP by using moving average statistics, so they can set batch size as 1 without degradation of performance, but Online Normalization still have to use instance-level normalization to cooperate with modification in BP, so its inference efficiency is much lower than original BN.
Apart from operating on feature maps, some works exploit to normalize the weights of convolution: Weight Standardization (Qiao et al., 2019) centralizes weight at first before divides weights by its standard deviation. It still has to combine with GN to handle small batch cases.

heading: 3 STATISTICS IN BATCH NORMALIZATION

heading: 3.1 REVIEW OF BATCH NORMALIZATION
text: First of all, let’s review the formulation of batch Normalization (Ioffe & Szegedy, 2015): assume the input of a BN layer is denoted as X ∈ RB×p, where B denotes the batch size, p denotes number of features. In training procedure, the normalized feature maps Y at iteration t is computed as:
Y = X − µBt σBt , (1)
where batch statistics µBt and σ 2 Bt are the sample mean and sample variance computed over the batch of samples Bt at iteration t:
µBt = 1
B ∑ b Xb,:, σ 2 Bt = 1 B ∑ b (Xb,: − µBt)2. (2)
Besides, a pair of parameters γ, β are used to scale and shift normalized value Y :
Z = Y γ + β. (3)
The scaling and shifting part is added in all normalization form by default, and will be omitted in the following discussion for simplicity.
As Ioffe & Szegedy (2015) demonstrated, the batch statistics µBt , σ 2 Bt are both involved in backward propagation (BP). We can derive the formulation of BP in BN as follows: let L denote the loss, Θt denote the set of the whole learnable parameters of the model at iteration t. Given the partial
gradients ∂L∂Y ∣∣∣∣ Θt,Bt , the partial gradients ∂L∂X ∣∣∣∣ Θt,Bt is computed as
∂L ∂X ∣∣∣∣ Θt,Bt = 1 σBt ( ∂L ∂Y ∣∣∣∣ Θt,Bt − gBt − Y ·ΨBt) (4)
where · denotes element-wise production, gBt and ΨBt are computed as
gBt = 1
B ∑ b ∂L ∂Yb,: ∣∣∣∣ Θt,Bt , ΨBt = 1 B ∑ b Yb,: · ∂L ∂Yb,: ∣∣∣∣ Θt,Bt , (5)
It can be seen from (5) that gBt and ΨBt are also batch statistics involved in BN during BP. But they have never been well discussed before.

heading: 3.2 INSTABILITY OF BATCH STATISTICS
text: According to Ioffe & Szegedy (2015), the ideal normalization is to normalize feature maps X using expectation and variance computed over the whole training data set:
Y = X − EX√ V ar[X] . (6)
But it’s impractical when using stochastic optimization. Therefore, Ioffe & Szegedy (2015) uses mini-batches in stochastic gradient training, each mini-batch produces estimates the mean and variance of each activation. Such simplification makes it possible to involve mean and variance in BP. From the derivation in section 3.1, we can see batch statistics µBt , σ 2 Bt are the Monte Carlo (MC) estimators of population statistics E[X|Θt], V ar[X|Θt] respectively at iteration t. Similarly, batch statistics gBt , ΨBt are MC estimators of population statistics E[ ∂L∂Yb,: |Θt], E[Yb,: · ∂L ∂Yb,: |Θt] at iteration t. E[ ∂L∂Yb,: |Θt], E[Yb,: · ∂L ∂Yb,: |Θt] are computed over the whole data set. They contain the information how the mean and the variance of population will change as model updates, so they play an important role to make trade off between the change of individual sample and population. Therefore, it’s crucial to estimate the population statistics precisely, in order to regularize the gradients of the model properly as weights update.
It’s well known the variance of MC estimator is inversely proportional to the number of samples, hence the variance of batch statistics dramatically increases when batch size is small. Figure 2 shows the change of batch statistics from a specific normalization layer of ResNet-50 during training on ImageNet. Regular batch statistics (orange line) are regarded as a good approximation for population statistics. We can see small batch statistics (blue line) are highly unstable, and contains notable error compared with regular batch statistics during training. In fact, the bias of gBt and ΨBt in BP is more serious than that of µBt and σ 2 Bt (see Figure 2(c), 2(d)). The instability of small batch statistics can worsen the capacity of the models in two aspects: firstly the instability of small batch statistics will make training unstable, resulting in slow convergence; Secondly the instability of small batch can produce huge difference between batch statistics and population statistics. Since the model is trained using batch statistics while evaluated using population statistics, the difference between batch statistics and population statistics will cause inconsistency between training and inference procedure, leading to bad performance of the model on evaluation data.

heading: 4 MOVING AVERAGE BATCH NORMALIZATION
text: Based on the discussion in Section 3.2, the key to restore the performance of BN is to solve the instability of small batch statistics. Therefore we considered two ways to handle the instability of
small batch statistics: using moving average statistics to estimate population statistics, and reducing the number of statistics by modifying the formulation of normalization.

heading: 4.1 SUBSTITUTE BATCH STATISTICS BY MOVING AVERAGE STATISTICS.
text: Moving average statistics seem to be a suitable substitute for batch statistics to estimate population statistics when batch is small. We consider two types of moving average statistics: simple moving average statistics (SMAS)2 and exponential moving average statistics (EMAS)3. The following theorem shows under mild conditions, SMAS and EMAS are more stable than batch statistics:
Theorem 1 Assume there exists a sequence of random variable (r.v.) {ξt}∞t=1, which are independent, uniformly bounded, i.e. ∀t, |ξt| < C, and have uniformly bounded density. Define:
St = 1
m t∑ i=t−m+1 ξi, Et = (1− α) t∑ i=1 αt−iξi, (7)
where m ∈ R+. If the sequence {ξt}∞t=1 satisfies
∃ξ,∀ ∈ R, lim t→∞ P (ξt ≤ ) = P (ξ ≤ ), (8)
then we have
E(Et) = E(ξ) + o(1), V ar(Et) = (1− α2t)(1− α)
1 + α V ar(ξ) + o(1); (9)

heading: If the sequence {ξt}∞t=1 satisfies
text: lim t→∞ sup λ |P (ξt−1 < λ)− P (ξt < λ)| = 0, (10)
then we have
E(St) = E(ξt) + o(1), V ar(St) = V ar(ξt)
m + o(1); (11)
The proof of theorem 1 can be seen in appendix A.1. Theorem 1 not only proves moving average statistics have lower variance compared with batch statistics, but also reveals that with large momentum α, EMAS is better than SMAS with lower variance. However, using SMAS and EMAS request different conditions: Condition (8) means the sequence of the given statistics need to weakly converge to a specific random variable. For {µBt}∞t=1, {σ2Bt} ∞ t=1, they converge to the ”final” batch statistics µB, σ2B (when training finished), hence condition (8) is satisfied, EMAS can be applied to replace {µBt}∞t=1, {σ2Bt} ∞ t=1; Unfortunately {gBt}∞t=1, {ΨBt}∞t=1 don’t share the same property, EMAS is not suitable to take replace of {gBt}∞t=1, {ΨBt}∞t=1. However, under the assumption that learning rate is extremely small, the difference between the distribution of ξt−1 and ξt is tiny, thus condition (10) is satisfied, we can use SMAS to replace {gBt}∞t=1, {ΨBt}∞t=1. In a word, we can use EMAS µ̂t, σ̂2t to replace µBt , σ 2 Bt , and use SMAS ḡt, Ψ̄t to replace gBt , ΨBt in (1) and (4), where
µ̂t = αµ̂t−1 + (1− α)µBt , σ̂2t = ασ̂2t−1 + (1− α)σ2Bt , (12)
ḡt = 1
m m∑ s=1 gBt−m+s , Ψ̄t = 1 m m∑ s=1 ΨBt−m+s . (13)
Notice neither of SMAS and EMAS is the unbiased substitute for batch statistics, but the bias can be extremely small comparing with expectation and variance of batch statistics, which is proven by equation 11 in theorem 1, our experiments also prove the effectiveness of moving average statistics as substitutes for small batch statistics (see Figure 3, 4 in appendix B.1).
2The exponential moving average (EMA) for a series {Yt}∞t=1 is calculated as: St = α ·Yt+(1−α) ·St−1. 3The simple moving average (SMA) for a series {Yt}∞t=1 is calculated as: St = ∑t s=t−M+1 Ys M .
Relation to Batch Renormalization Essentially, Batch Renormalization (BRN) (Ioffe, 2017) replaces batch statistics µBt , σ 2 Bt with EMAS µ̂t, σ̂ 2 t both in FP (1) and BP (4). The formulation of BRN during training is written as:
Y = X − µBt σBt , Ŷ = r · Y + d (14)
where r = clip[1/λ,λ]( σBt σ̂t ), d = clip[−d,d]( µBt−µ̂t σ̂t
). Based on our analysis, BRN successfully eliminates the effect of small batch statistics µBt and σ 2 Bt by EMAS, but the small batch statistics associated with gradients gBt and ΨBt remains during backward propagation, preventing BRN from completely restoring the performance of vanilla BN.

heading: 4.2 STABILIZING NORMALIZATION BY REDUCING THE NUMBER OF STATISTICS
text: To further stabilize training procedure in small batch cases, we consider normalizing feature maps X using EX2 instead of EX and V ar(X). The formulation of normalization is modified as:
Y = X
χBt , Z = Y · γ + β, (15)
where χ2Bt = 1 B ∑ bX 2 b,:. Given ∂L ∂Y , the backward propagation is:
∂L ∂X ∣∣∣∣ Θt,Bt = 1 χBt ( ∂L ∂Y ∣∣∣∣ Θt,Bt − Y ·ΨBt). (16)
The benefits of the modification seems obvious: there’s only two batch statistics left during FP and BP, which will introduce less instability into the normalization layer compared with vanilla normalizing form. In fact we can theoretically prove the benefits of the modification by following theorem:
Theorem 2 If the following assumptions hold:
1. V ar[σ̂] = o(1), V ar[χ̂] = o(1);
2. Cov({∂L∂y , y}, {gB,ΨB}) = o(1);
3. Ey = o(1);
Then we have:
V ar [∂L ∂x ∣∣∣∣ modified ] ≤ V ar [∂L ∂x ∣∣∣∣ vanilla ] − V ar[gB] σ̂2 (17)
The proof can be seen in appendix A.2. According to (17), V ar[∂L/∂X ∣∣ vanilla
] is larger than that of V ar[∂L/∂X ∣∣ modified
], the gap is at least V ar[gB]/σ̂2, which mainly caused by the variance of gB/σ̂. So the modification essentially reduces the variance of the gradient by eliminating the batch statistics gB during BP. Since gBt is a Monte Carlo estimator, the gap is inversely proportional to batch size. This can also explain why the improvement of modification is significant in small batch cases, but modified BN shows no superiority to vanilla BN within sufficient batch size (see ablation study in section 5.1).
Centralizing weights of convolution kernel Notice theorem 2 relies on assumption 3. The vanilla normalization naturally satisfies Ey = 0 by centralizing feature maps, but the modified normalization doesn’t necessarily satisfy assumption 3. To deal with that, inspired by Qiao et al. (2019), we find centralizing weights W ∈ Rq×p of convolution kernels, named as Weight Centralization (WC) can be a compensation for the absence of centralizing feature maps in practice:
W̄ = 1
p ∑ i W:i, Xoutput = (W − W̄ )Xinput, (18)
where Xinput, Xoutput are the input and output of the convolution layer respectively. We conduct further ablation study to clarify the effectiveness of WC (see Table 4 in appendix B.2). It shows that WC has little benefits to vanilla normalization, but it can significantly improve the performance of modified normalization. We emphasize that weight centralization is only a practical remedy for the absence of centralizing feature maps. The theoretical analysis remains as a future work.
Clipping and renormalizing strategy. In practice, we find directly substituting batch statistics by moving average statistics in normalization layer will meet collapse during training. Therefore we take use of the clipping and renormalizing strategy from BRN (Ioffe, 2017).
All in all, the formulation of proposed method MABN is:
Y = X
χ̄t , Ŷ = r · Y (19)
∂L ∂Y ∣∣∣∣ Θt,Bt = r · ∂L ∂Ŷ ∣∣∣∣ Θt,Bt , ∂L ∂X ∣∣∣∣ Θt,Bt = 1 χ̄t ( ∂L ∂Y ∣∣∣∣ Θt,Bt − Y Ψ̄t) (20)
where the EMAS χ̂t is computed as χ̂t = αχ̂t−1 + (1 − α)χBt , SMAS χ̄t is defined as χ̄2t = 1 m ∑m s=1 χ 2 Bt−m+s , SMAS Ψ̄t is defined as (13). The renormalizing parameter is set as r = clip[1/λ,λ]( χ̄t χ̂t ).

heading: 5 EXPERIMENTS
text: This section presents main results of MABN on ImageNet (Russakovsky et al., 2015), COCO (Lin et al., 2014). Further experiment results on ImangeNet, COCO and Cityscapes (Cordts et al., 2016) can be seen in appendix B.2, B.3, B.4 resepectively. We also evaluate the computational overhead and memory footprint of MABN, the results is shown in appendix B.5.

heading: 5.1 IMAGE CLASSIFICATION IN IMAGENET
text: We evaluate the proposed method on ImageNet (Russakovsky et al., 2015) classification datatsets with 1000 classes. All classification experiments are conducted with ResNet-50 (He et al., 2016). More implementation details can be found in the appendix B.2.
gradient batch size is 32 per GPU. Regular means normalization batch size is 32, while Small means normalization batch size is 2.
Comparison with other normalization methods. Our baseline is BN using small (|B| = 2) or regular (|B| = 32) batch size, and BRN (Ioffe, 2017) with small batch size. We don’t present the performance of instance-level normalization counterpart on ImageNet, because they are not linear-type method during inference time, and they also failed to restore the performance of BN (over +0.5%), according to Wu & He (2018). Table 1 shows vanilla BN with small batch size can severely worsen the performance of the model(+11.81%); BRN (Ioffe, 2017) alleviates the issue to some extent, but there’s still remaining far from complete recovery(+6.88%); While MABN almost completely restore the performance of vanilla BN(+0.17%).
We also compared the performance of BN, BRN and MABN when varying the batch size (see Figure 1(b)). BN and BRN are heavily relies on the batch size of training, though BRN performs better than
vanilla BN. MABN can always retain the best capacity of ResNet-50, regardless of batch size during training.
size is 2 in all experiments otherwise stated. The memory size is 16 and momentum is 0.98 when using SMAS, otherwise the momentum is 0.9. ”-” means the training can’t converge.
Ablation study on ImageNet. We conduct ablation experiments on ImageNet to clarify the contribution of each part of MABN (see table 2). With vanilla normalization form, replacing batch statistics in FP with EMAS (as BRN) will restore the performance to some extents(−4.93%, comparing 3© and 4©), but there’s still a huge gap (+6.88%, comparing 1© and 4©) from complete restore. Directly using SMAS in BP with BRN will meet collapse during training ( 5©), no matter how we tuned hyperparameters. We think it’s due to the instability of vanilla normalization structure in small cases, so we modify the formulation of normalization shown in section 4.2. The modified normalization even slightly outperforms BRN in small batch cases (comparing 4© and 6©). However, modified normalization shows no superiority to vanilla form (comparing 1© and 2©), which can be interpreted by the result of theorem 2. With EMAS in FP, modified normalization significantly reduces the error rate further (comparing 6© and 7©), but still fail to restore the performance completely (+3.62%, comparing 1© and 7©). Applying SMAS in BP finally fills the rest of gap, almost completely restore the performance of vanilla BN in small batch cases (+0.17 ,comparing 1© and 8©).

heading: 5.2 DETECTION AND SEGMENTATION IN COCO FROM SCRATCH
text: We conduct experiments on Mask R-CNN (He et al., 2017) benchmark using a Feature Pyramid Network(FPN) (Lin et al., 2017a) following the basic setting in He et al. (2017). We train the networks from scratch (He et al., 2018) for 2× times. Only the backbone contains normalization layers. More implementation details and experiment results can be seen in the appendix B.3.
tation. The gradients batch size is 16. The normalization batch size of SyncBN is 16, while that of BN, BRN and MABN are both 2. The momentum of BRN and MABN are both 0.98, while the momentum of BN and SyncBN are both 0.9. The buffer size (m) is 16).
Table 3 shows the result of MABN compared with vanilla BN, BRN and SyncBN (Peng et al., 2018). It can be seen that MABN outperforms vanilla BN and BRN by a clear margin and get comparable performance with SyncBN. Quite different from Imagenet experiments, we update the parameters every single batch (with Bnorm = 2). With such a complex pipeline, MABN still achieves a comparable performance as SyncBN.

heading: 6 CONCLUSION
text: This paper reveals the existence of the batch statistics gB and ΨB involved in backward propagation of BN, and analysis their influence to training process. This discovery provides a new perspective to understand why BN always fails in small batch cases. Based on our analysis, we propose MABN to deal with small batch training problem. MABN can completely restore the performance of vanilla BN in small batch cases, and is extraordinarily efficient compared with its counterpart like GN. Our experiments on multiple computer vision tasks (classification, detection, segmentation) have shown the remarkable performance of MABN.

heading: ACKNOWLEDGEMENT
text: This research was partially supported by National Key RD Program of China (No. 2017YFA0700800), Beijing Academy of Artificial Intelligence (BAAI), and NSFC under Grant No. 61473091.

heading: A SKETCH OF PROOF
text: A.1 PROOF OF THEOREM 1
If the condition (8) is satisfied, i.e. {ξt}∞t=1 weakly converge to ξ. Since {ξt}∞t=1 has uniformly bounded density, we have:
lim t→∞
Eξt = Eξ (21)
lim t→∞ V ar[ξt] = V ar[ξ] (22)
Since {ξt}∞t=1 are independently, hence we have:
V ar[Et] = V ar[(1− α) ∑ i=1 t∑ i=1 αt−iξi]
= (1− α)2 t∑ i=1 α2(t−i)V ar[ξi]
= (1− α)2 t∑ i=1 α2(t−i)V ar[ξ] + (1− α)2 t∑ i=1 α2(t−i)(V ar[ξi]− V ar[ξ])
= (1− α2t)(1− α)
1 + α V ar[ξ] + o(1)
(23)
as t→∞. Hence (9) has been proven. If the condition (10) is satisfied. Since {ξt}∞t=1 is uniformly bounded, then ∃C ∈ R+, ∀, |ξt| < C. As t→∞, We have∣∣Eξt−1 − Eξt| = |∫
x∈[−C,C] xpt−1(x)dx− ∫ x∈[−C,C] xpt(x)dx ∣∣
= ∣∣ ∫ x∈[−C,C] x(pt−1(x)− pt(x))dx ∣∣
= ∣∣x(Ft−1(x)− Ft(x))∣∣C−C − ∫
x∈[−C,C] (Ft−1(x)− Ft(x))dx ∣∣ ≤ ∫ x∈[−C,C] |Ft−1(x)− Ft(x)|dx
≤ 2C · sup x |Ft−1(x)− Ft(x)|
= 2C · sup x |P (ξt−1 < x)− P (ξt < x)|
= o(1)
(24)
Similarly, we have |Eξ2t−1 − Eξ2t | = ∣∣ ∫ x∈[−C,C] x2(pt−1(x)− pt(x))dx ∣∣
= ∣∣x2(Ft−1(x)− Ft(x))∣∣C−C − ∫
x∈[−C,C] 2x(Ft−1(x)− Ft(x))dx ∣∣ ≤ ∫ x∈[−C,C] 2|x||Ft−1(x)− Ft(x)|dx
≤ 4C2 · sup x |Ft−1(x)− Ft(x)|
= o(1)
(25)
Therefore combining (24) and (25), we have |V ar[ξt−1]− V ar[ξt]| ≤ |Eξ2t−1 − Eξ2t |+ |(Eξt−1)2 − (Eξt)2|
= o(1) (26)
For a fixed memory size m, as t→∞, we have
V ar(St) = V ar[ 1
m m−1∑ i=0 ξt−i]
= 1
m m−1∑ i=0 V ar[ξt−i]
= 1
m m−1∑ i=0 (V ar[ξt] + o(1))
= V ar[ξt] + o(1)
(27)
Therefore, (11) has been proven.
A.2 PROOF OF THEOREM 2
Without loss of generality, given the backward propagation of two normalizing form of a single input x with batch B:
∂L ∂x ∣∣∣∣ vanilla = 1 σ̂ [ ∂L ∂y − gB − y ·ΨB], ∂L ∂x ∣∣∣∣ modified = 1 χ̂ [ ∂L ∂y − y ·ΨB], (28)
where gB, ΨB are the batch statistics, and σ̂, χ̂ are the EMAS, defined as before. We omitted the subscript t for simplicity. Then the variance of partial gradients w.r.t. inputs x is written as
V ar[ ∂L ∂x ∣∣∣∣ vanilla ] = V ar [ 1 σ̂ [ ∂L ∂y − gB − y ·ΨB] ] (29)
= 1
σ̂2
[ V ar [∂L ∂y − y ·ΨB ] + V ar [ gB ] + 2Cov [∂L ∂y − y ·ΨB, gB ]] (30)
= 1
σ̂2
[ V ar [∂L ∂y − y ·ΨB ] + V ar [ gB ]]]
(31)
≥ 1 χ̂2 V ar [∂L ∂y − y ·ΨB ] + V ar
[ gB
σ̂2 (32)
= V ar [∂L ∂x ∣∣∣∣ modified ] + V ar[gB] σ̂2 (33)
where (30) is satisfied due to assumption 1. The variance of σ̂ is so small that σ̂ can be regarded as a fixed number; (31) is satisfied because
Cov [∂L ∂y − y ·ΨB, gB ] = Cov[ ∂L ∂y , gB]− Cov [ y ·ΨB, gB ] (34)
= Cov[ ∂L ∂y , gB]− E[yΨB(gB − EgB)] + E[yΨB]E[gB − EgB](35)
Due to assumption 2, the correlation between individual sample and batch statistics is close to 0, hence we have
Cov[ ∂L ∂y , gB] = 0 (36)
E[yΨB(gB − EgB)] = EyE[ΨB(gB − EgB)] (37) E[yΨB] = EyEΨB (38)
Besides, Ey is close to 0 according to assumption 3, hence Cov [∂L ∂y − y ·ΨB, gB ] = 0. (39)
(32) is satisfied due to the definition of χ̂ and σ̂, we have χ̂2 = σ̂2 + µ̂2. (40)
Similar to σ̂, the variance of χ̂ is also too small that χ̂ can be regarded as a fixed number due to assumption 1, so (33) is satisfied.

heading: B EXPERIMENTS
text: B.1 STATISTICS ANALYSIS
We analyze the difference between small batch statistics (|B| = 2) and regular batch statistics (|B| = 32) with the modified formulation of normalization (15) shown in Section 4.2.
statistics (|B| = 32). We use the small batch statistics to update the network parameters.
moving average batch statistics to update the network parameters.
Figure 3 illustrates the change of small batch statistics and regular batch statistics in FP and BP respectively. The variance of small batch statistics is much higher than the regular one. However, when we use SMAS as a approximation for regular batch statistics, the gap between SMAS and regular batch statistics is not obvious as shown in Figure 4.
B.2 EXPERIMENTS ON IMAGENET
Implementation details. All experiments on ImageNet are conducted across 8 GPUs. We train models with a gradient batch size ofBg = 32 images per GPU. To simulate small batch training, we split the samples on each GPU into Bg/|B| groups where |B| denotes the normalization batch size. The batch statistics are computed within each group individually.
All weights from convolutions are initialized as He et al. (2015). We use 1 to initialize all γ and 0 to initialize all β in normalization layers. We use a weight decay of 10−4 for all weight layers including γ and β (following Wu & He (2018)). We train 600, 000 iterations (approximately equal to 120 epoch when gradient batch size is 256) for all models, and divide the learning rate by 10 at 150, 000, 300, 000 and 450, 000 iterations. The data augmentation follows Gross & Wilber (2016). The models are evaluated by top-1 classification error on center crops of 224 × 224 pixels in the validation set. In vanilla BN or BRN, the momentum α = 0.9, in MABN, the momentum α = 0.98.
Additional ablation studies. Table 4 shows the additional ablation results. We test all possible combination of all three kinds of statistics (SMAS, EMAS, BS) in FP and BP. The experiments results strongly prove our theoretical analysis in section 4.3. Besides, we verify the necessity of centralizing weights with modified normalization form.
2 in all experiments. The buffer size (m) is 16 and momentum is 0.98 when using SMA statistics, otherwise the momentum is 0.9. BS means vanilla batch statistics.
B.3 EXPERIMENTS ON COCO
Implementation details. We train the Mask-RCNN pipeline from scratch with MABN. We train the model on 8 GPUs, with 2 images per GPU. We train our model using COCO 2014 train and trainval35k dataset. We evaluate the model on COCO 2014 minival dataset.We set the momentum α = 0.98 for all MABN layers. We report the standard COCO merics AP bbox, AP bbox75 , AP bbox 50 for bounding box detection and APmask, APmask50 , AP mask 75 for instance segmentation. Other basic settings follow He et al. (2017).
MABN used on heads. We build mask-rcnn baseline using a Feature Pyramid Network(FPN)(Lin et al., 2017a) backbone. The base model is ResNet-50. We train the models for 2× iterations. We use 4conv1fc instead of 2fc as the box head. Both backbone and heads contain normalization layers. We replace all normalization layers in each experiments. While training models with MABN, we use batch statistics in normalization layers on head during first 10,000 iterations. Table 5 shows the result. The momentum are set to 0.98 in BRN and MABN.
tation. The gradients batch size is 16. The normalization batch size of SyncBN is 16, while that of BN, BRN, MABN are both 2, the buffer size (m) of MABN is 32.
Training from pretrained model. We compare the performance of MABN and SyncBN when training model based on ImageNet pretrained weights for 2x iterations. The results are shown in Table
tation. The gradients batch size is 16. The normalization batch size of SyncBN is 16, while that of BN, BRN, MABN are both 2, the buffer size (m) of MABN is 32.
Training from scratch for one-stage model. We also compare MABN and SyncBN based on one-stage pipeline. We build on retinanet(Lin et al., 2017b) benchmark. We train the model from scratch for 2× iterations. The results are shown in Table 7.
batch size is 16. The normalization batch size of SyncBN is 16, while that of MABN is 2.
All experiment results shows MABN can get comparable as SyncBN, and significantly outperform BN on COCO.
B.4 SEMANTIC SEGMENTATION IN CITYSCAPES
We evaluate semantic segmentation in Cityscapes(Cordts et al., 2016). It contains 5,000 high quality pixel-level finely annotated images collected from 50 cities in different seasons. We conduct experiments on PSPNET baseline and follow the basic settings mentioned in Zhao et al. (2017).
For fair comparison, our backbone network is ResNet-101 as in Chen et al. (2017). Since we centralize weights of convolutional kernel to use MABN, we have to re-pretrain our backbone model on Imagenet dataset. During fine-tuning process, we linearly increase the learning rate for 3 epoch (558 iterations) at first. Then we follow the ”poly” learning schedule as Zhao et al. (2017). Table 8 shows the result of MABN compared with vanilla BN, BRN and SyncBN. The buffer size (m) of MABN is 16, the modementum of MABN and BRN is 0.98.
Since the statistics (mean and variance) is more stable in a pre-trained model than a random initialized one, the gap between vanilla BN and SyncBN is not significant (+1.41%). However, MABN
still outperforms vanilla BN by a clear margin.(+1.09%). Besides, BRN shows no obvious superiority to vanilla BN(+0.19%) on Cityscapes dataset.
B.5 COMPUTATIONAL OVERHEAD
We compare the computational overhead and memory footprint of BN, GN and MABN. We use maskrcnn with resnet50 and FPN as benchmark. We compute the theoretical FLOPS during inference and measure the inference speed when a single image (3×224×224) goes through the backbone (resnet50 + FPN). We assume BN and MABN can be absorbed in convolution layer during inference. GN can not be absorbed in convolution layer, so its FLOPS is larger than BN and MABN. Besides GN includes division and sqrt operation during inference, therefore it’s much slower than BN and MABN during inference time.
We also monitor the training process of maskrcnn on COCO (8 GPUs, 2 images per GPU), and show its memory footprint and training speed. Notice we have not optimized the implementation of MABN, so its training speed is a little slower than BN and GN.

Review summaries:
The paper extends recently proposed BatchRenormalization (BRN) technique which uses exponential moving average (EMA) statistics in forward and backward passes of BatchNorm (BN) instead of vanilla batch statistics. Motivation of the work is to stabilize training neural networks on small batch size setup. Authors propose to replace EMA in backward pass by simple moving average (SMA) and show that under some assumptions such replacement reduces variance. Also they consider slightly different way of normalization without centralizing features X, but centralizing convolutional kernels according to Qiao et al. (2019).

The paper proposes a new approach for batch-normalization. Standard approaches are sensitive to the batch size, because small batches will lead to unstable statistics. So when the mini-batch is small, the performance can drop significantly. The paper addresses this issue by analyzing extra statistics in the batch normalization and introducing moving average statistics, weights centralization and a slightly modified normalization. The proposed method does not require large batch sizes and nonlinear operations, but still maintain the robustness. The theoretical analysis and guarantees are provided as well. Experiments on typical datasets demonstrate the effectiveness of the proposed trick.