Heterologous  Normalization
Authors: Chunjie Luo
Jianfeng Zhan
Lei Wang
Wanling Gao
Venue: ICLR 2022 Submitted (2022)

Abstract: Batch Normalization has become a standard technique for training modern deep networks. However, its effectiveness diminishes when the batch size becomes smaller since the batch statistics estimation becomes inaccurate. This paper proposes Heterologous Normalization, which computes normalization's mean and standard deviation from different pixel sets to take advantage of different normalization methods. Specifically, it calculates the mean like Batch Normalization to maintain the advantage of Batch Normalization. Meanwhile, it enlarges the number of pixels from which the standard deviation is calculated, thus alleviating the problem caused by the small batch size. Experiments show that Heterologous Normalization surpasses or achieves comparable performance to existing homologous methods, with large or small batch sizes on various datasets.

Under review as a conference paper at ICLR 2022
HETEROLOGOUS NORMALIZATION
Anonymous authors
Paper under double-blind review
ABSTRACT
Batch Normalization has become a standard technique for training modern deep networks. However, its effectiveness diminishes when the batch size becomes smaller since the batch statistics estimation becomes inaccurate. This paper pro- poses Heterologous Normalization, which computes normalization's 's mean and standard deviation from different pixel sets take advantage of different normal- ization methods. Specifically, it calculates the mean like Batch Normalization to maintain the advantage of Batch Normalization. Meanwhile, it enlarges the num- ber of pixels from which the standard deviation is calculated, thus alleviating the problem caused by the small batch size. Experiments show that Heterologous Nor- malization surpasses or achieves comparable performance to existing homologous methods, with large or small batch sizes on various datasets.
INTRODUCTION
Deep neural networks have received great success in many areas. Batch Normalization (BN) has become a standard technique for training modern deep networks. BN normalizes the features by the mean and the standard deviation (std) computed from a batch of samples. The coordination between examples helps the learning process. The random selection of examples in the minibatch brings the sampling noises, providing regularization effect. However, its effectiveness diminishes when the batch size becomes smaller since the noises are too much and make inaccurate batch statistics esti- mation. That hinders BN's usage for scenes lacking samples memories. For example, federated learning, hot topic in machine learning, aims to train a model across multiple decentralized edge devices or servers address privacy and security issues. The heterogeneous environments require a robust training algorithm with large small batch size.
To address the small batch size problem, some methods try to avoid normalizing along the batch dimension. In the case of NCHW format feature map, let N refer to the batch dimension, C refer to the channel dimension, H and W refer to the spatial height and width dimensions. Layer Normal- ization (LN) (Ba et al.] (0166) computes the mean and the std along (C dimensions. Instance Normalization (IN) (Ulyanov et al. 2016) computes the mean and the std along (H, W) dimensions. Group Normalization (GN) (Wu& & He 2018) is an intermediate state between Layer Normalization and Instance Normalization. mean and the std. Avoiding normalizing along the batch dimension gives up the advantages of BN, resulting in poor performances in many cases. To take advantage of different approaches, Switch- able Normalization (SN) (Luo al., 2018a) combines different normalizations. It uses learnable parameters to combine BN, IN, and LN linearly. SN brings more statistics and computational com- plexity. Besides, the mean and the std of those methods need be calculated inference time. They slow down the inference comparing to BN since BN's mean and std are pre-computed from the training data by the moving average.
uses a group of channels within the sample itself to compute the
Although different normalizations compute the statistics from different pixel sets (pixel refers to a element in the feature map), specific normalization computes the mean and the std from the same pixel set. Thus existing methods can be viewed as homologous normalization. In this paper, we propose Heterologous Normalization (HN), which computes the mean and the std from different pixel sets to take advantage of different normalization methods. Although HN is general method that can use different strategies to compute the mean and the std, we find this combination strategy works well in most situations: calculating the mean along the (N, H. W) dimensions the same as BN, while calculating the std along the (N,C.H. W) dimensions. On the one hand, it maintains the advantage of batch normalization when the batch size is large. On the other hand, it enlarges the

---

Under review as a conference paper at ICLR 2022
number of pixels from which the std is calculated, thus alleviating the problem caused by the small batch size. At inference time, HN's ' mean and std are pre-computed from the training set by moving average the same as BN, keeping the inference efficiency.
We evaluate HN on various datasets: CIFAR-10, 10, C10,R-100, CalTech256, ImageNet. Experiments show that heterologous combination is valid for most situations. HN surpasses achieves compara- ble performance than BN, IN, LN, GN, and SN, with large or small batch sizes on various datasets. Moreover, HN can be combined with SN together to improve the performance further. By analyzing the statistics' evolution over the course of training, we find that the noise of the small batch size mainly caused by the std fluctuation rather than the mean. Enlarging the number of pixels from which the std is calculated can alleviate fluctuation successfully. That explains why HN works well with the small batch size.
We conclude our key contributions follows: 1) We show that it is unnecessary to estimate nor- malization statistics from the same pixel set and propose general Heterologous Normalization that calculates normalization statistics from different pixel sets. 2) We find special heterologous method that surpasses or achieves comparable performance to existing homologous methods, with large or small batch sizes on various datasets. 3) We find the noise of the small batch size is mainly caused by the std fluctuation rather than the mean. We should make an equilibrium between generalization and stability by controlling the number of pixels to calculate the statistics.
2 RELATED WORK
Batch Normalization (BN) (Joffe & Szegedy 2015) is effective at accelerating and improving the training of deep neural networks by reducing internal covariate shift. performs the normaliza- tion for each training minibatch along (N,H,W) dimensions in the case of NCHW format feature. Since BN uses the statistics on minibatch examples, its effect is dependent on the minibatch size. To mitigate this problem, Normalization Propagation (Arpit et al. 2016) uses data-independent parametric estimate of the mean and standard deviation instead of explicitly calculating from data. Batch Renormalization (loffe, 2017) introduces two extra parameters correct the fact that the minibatch statistics differ from the population ones. It needs to train the model for certain number of iterations with Batch Normalization alone, without the correction, then ramps up the amount of allowed correction.
There is a family of methods that avoid normalizing along the batch dimension. Layer Normaliza- tion (LN) (Ba et al. (2166) computes the mean and standard deviation along (C dimensions. Instance Normalization (IN) (Ulyanov al.) 2016) computes the mean and standard deviation along (H,W) dimensions. When the batch size is 1, Batch Normalization is equivalent to Instance Nor- malization. Group Normalization (GN) (Wu & He 2018) is an intermediate state between Layer Normalization and Instance Normalization. It uses a group of channels to compute the mean and the std, while Layer Normalization uses all channels, and Instance Normalization uses one channel. To take advantage of different approaches, Switchable Normalization (SN) (Luo et al. 2018a), Ex- emplar Normalization Zhang et al. (2020) and Batch-Instance Normalization (BIN) (Nam & Kim 2018) try combine different normalization together. Switchable and Exemplar Normalization use Tearnable parameters to combine Batch, Instance, and Layer Normalization. BIN uses a learnable gate parameter combine Batch and Instance normalization.
Weight normalization (Salimans & Kingmal 2016) normalizes the filter weights instead of the acti- vations by re-parameterizing the incoming weight vector. Cosine normalization (Luo et al., 2017) normalizes both the filter weights and the activations by using cosine similarity instead of dot prod- uet in neural networks. Some rescarchers try to use other statistics instead of the mean and the standard deviation in normalization. Instead of the standard L batch normalization, (Hoffer et al.) 2018) uses normalization in I,- and 1 spaces and shows that it can improve numerical stability in low-precision implementations well as provide computational and memory benefits. Generalized batch normalization (Yuan et al.) 2019) investigates a variety of alternative deviation measures for scaling and alternative mean measures for centering. Virtual batch normalization (Salimans et al.) 2016) and spectral normalization MMyyaoo et al.] 2018) focus on the normalization in generative adversarial networks. Self- Normalizing (Klambauer et al., 2017) focuses standard feed-forward neural networks (fully-connected networks). Recurrent batch normalization (Cooijmans et al. 2016)) modifies batch normalization to use in recurrent networks. Kalman normalization (Wang et al.
2

---

Under review as a conference paper at ICLR 2022
2018
estimates the mean and standard deviation of a certain layer by considering the distributions of all its preceding layers, mimicking the merits of Kalman Filtering Process. EvalNorm (Singh & Shrivastava 2019) estimates corrected normalization statistics to use for batch normalization during evaluation. [Ren et al. 2016) provides a unifying view of the different normalization approaches. (Santurkar al. 2018), (Luo al. 2018b) and (Bjorck et al., (018) try explain how batch nor- malization works. Summers & Dinneen (2020) propose several useful techniques to improve batch normalization.
HETEROLOGGOUS NORMALIZATION
We first describe some notations that will be used next. For NCHW format feature map, let U denote the universal pixel set the same feature layer, Un denote the set of pixels that belong to the same example, Uc denote the set of pixels that belong to the same channel, Ua denote the set of pixels that belong to the same group of channels. A family of normalization can be formalized as:
1
(r. PS.)
05
y1 7., B
(2)
where ay is the input, and yi is the output of the normalization. f is the mean, and is the standard deviation (std). S; is the pixel set from which the mean is computed, and S is the pixel set from which the std is computed. Ty and 3 are learned parameters for affine transformation.
As shown in Table [ different normalization methods estimate statistics from different pixel sets. BN computes the mean and the std along the (N, H. W) dimensions. The random noise brought by batch statistics estimation is beneficial the generalization. However, the batch statistics estimation brings too much noise when the batch size is small. IN, LN, and GN try to avoid estimate statistics along the batch dimension. Each example estimates statistics within its own pixel sets, ignoring the information of other examples. Since there are not enough pixels to estimate statistics for BN when batch size is small, straightforward method is to extend the pixel set for statistics calculation. For example, we can use the universal pixels U or the pixels from group of channels UG to calculate statistics. We use Extended Normalization (EN) to refer to that method. In the following, the default number of groups in EN is 1 (the universal set U) when there is mention of it, and we use EN_Gr N_Gn to represent that the groups of channels are n. in EN. Figure shows the visualization of different normalizations.
Table 1: Different normalization
S S,=S S Uc
Description
BN
Estimate statistics from pixels that belong to the same channel.
IN
CcnUN
Estimate statistics from pixels that belong to both the same channel and the same example. Estimate statistics from pixels that belong to the same ex- ample.
IN
UN
GN
UnNUN
Estimate statistics from pixels that belong to both the same group channels and the same example. Estimate statistics from pixels that belong to the same group of channels.
EN
Ua
Although different normalizations compute the statistics from different pixel sets, specific normal- ization computes the mean and the std from the same pixel set. Thus existing methods can be viewed as homologous normalization, which has
S S
In this paper, we propose Heterologous Normalization (HN), which computes the mean and the std from different of pixel sets to take advantage of different normalization methods. That is to say, in HN, have
S # S
3

---

Under review as a conference paper at ICLR 2022
(a) BN
(b) IN
(c) LN
(d) GN
(c) EN
(1) EN_G2
Figure 1: Visualization of different normalization methods. Horizontal is channel dimension, and vertical is batch dimension. The small square grid is the pixel set along the spatial height and width dimensions.
Although HN is general method that can use different strategies to compute the mean and the std, we find this configure works well in most situations:
(5) Specifically, the mean is calculated along the (N, H. W) dimensions the same BN, while the std is computed along the (N,C,H.W, dimensions as the same as EN. On the one hand, it maintains BN's advantage when the batch size is large. On the other hand, it enlarges the number of pixels from which the std is calculated, thus alleviating the problem caused by the small batch size.
S = Uc, S 5.=UB UG
Switchable Normalization (SN) (Luo et al. 2018a) also try take advantage of different normaliza- tions by mixing them together. Its mean and std are both linear combinations of the batch, instance, and layer ones. The pixel sets, from which the mean and the std calculated, are the same. Differ- ent from SN, HN calculates the mean and the std from heterogeneous pixel sets. HN has fewer statis- ties and computational complexity than SN. Besides, the mean and the std of HN are pre-computed from the training data by the moving average as the same as BN. There is need to compute the mean and the std at inference time. Moreover, since the mean and the std are pre-computed and fixed at inference time, the normalization very helpful to speed up the inference, especially on mobile or embedded devices. As for IN, LN, GN, SN, the mean and the std need to be calculated at inference time. Finally, HN and SN not mutually exclusive. We can combine HN and SN together. For example, we can use the mean of BN, and the linear combinations of the std from different normalizations.
further be fused into convolution operation. That is
4 EXPERIMENT
4.1 CIFAR-10
CIFAR-I0 10 (Krizhessyy & Hinton| 2009) is data set of natural 32x32 RGB images in 10 classes with 50, 000 images for training and 10, 000 for testing. The training images padded with 0 to 36x36 and then randomly cropped to 32x32 pixels. Then randomly horizontal flipping is made. Each channel of the input is normalized globally. Weight decay of 0.0005 and SGD with 0.9 momentum are used. For the batch size of 128, the initial learning rate is set to 0. For other batch sizes n, the initial learning rate is to 0.1 n/ 1288, following the linear scaling rule. We train the networks for 200 epochs and decrease the learning rate by 10x at 100 and 150 epochs. One GPU is used to train all networks.
Different combination strategies. Figure show the test accuracy of ResNet18 (He et al., 2016) using different normalization combination. For simplicity, we use (A.B) to represent the mean is calculated as the same as normalization A (column in Figure and the std is calculated as the same as normalization B (row in Figure D And we use * to represent any normalization. The diagonal cell is the traditional homologous normalization whose mean and std are calculated from the same pixel set.
When the batch size is 128, as shown in Figure 2(a) (BN,BN) gets better performance than other homologous normalization, e.g. (IN,IN), (LN,LN), GNN (EN,EN). Heterologous combinations (BN,*) all achieve decent performances except (BN.IN). Among them. (BN,EN) achieves the best accuracy 95.18%, which is higher than (BN,BB (BN.NN) 95.05%. Many other combinations also show heterologous normalization is effective. For example, heterologous (IN,LN) 93.33% and (LN,IN)
4

---

Under review as a conference paper at ICLR 2022
93.10% better than homologous (IN,IN) 92.88% and (LN,LN) 92.21%. When the batch size 4, as shown in Figure 2 the accuracy of (BN,BN) deteriorates seriously, suffering the issues caused by the small batch size. We find that heterologous normalization is generally valid. Most heterologous combinations not only surpass the homologous combinations but also outperform the corresponding ones with large batch size 128. Moreover, heterologous combinations (BN, all achieve decent performances. Among them, (BN,EN_GE achieves the best accuracy 95.04% comparable (BN,BN) 95.05% with large batch size 128.
BX
TX
LN
GN
EN
EN 632
std
EN G32
BN
194,
93.
94,
91.
IN
93. 52
TN
93. 42
93. 47
93. 22
93. 70
93. 24
94. 02
IN
84. 44
93,
92, 21
92.
85.
1N
94. 83
93. 66
92. 90
83. 74
93,
93,
ON
GN
94. 62
93. 60
80, 96
94. 06
90.
94.
EN
86.
53. 61
90.
91.
91.
92.
EN
94. 93
93. 62
93, 62
94. 37
93. 89
94. 37
EN
02.
03,
93.
(a) Batch size 128
(b) Batch size =
Figure 2: Test accuracy (%) by using different normalization combinations on ResNet18. Redder represents higher accuracy. Slash means the failure of training. The diagonal cells are the traditional homologous normalization, while other cells are heterologous normalization.
Ablation study. We investigate different configure of channel groups for EN ResNet18. When the mean and the std are computed from the same channel groups, the normalization is homologous. Otherwise, the normalization is heterologous. When the group is c, which means the number of channels, EN is equal to BN, As shown in Figure 3 the diagonal cells are the homologous EN, while other cells are heterologous EN. We can find heterologous normalization is generally valid, especially in the case of the small batch size. Moreover, we can find the model achieves better performance by setting the mean's group to and varying the std's group. For example, by setting the mean's group to and the std's 'ss group to 1, the model gets the best accuracy when the batch size is 128. By setting the mean's 's group to and the std's group to 8, the model gets the best accuracy when the batch size is 4. That is to say. there is need to extend the pixels to calculate the mean. It is enough to extend the pixels to calculate the std. is verified again that the heterologous combination (BN,EN) works well.
sid
32
16
8
std
35, 05
94. 33
53. 77
93. 54
93, 92
93. 41
32
96, 01
94. 17
94. 48
94. 67
94. 08
93. 70
16
94.
93.
93, 60
91. 96
92. 60
93
16
91.
93, 57
91.
92, 35
92.
8
85. 17
84. 69
94. 09
94. 28
94.
94. 07
4
$4. 72
94. 34
94. 27
93. 95
83, 86
93, 70
95.
18
92.
04
91. 17
88, 64
92. 98
91. 66
(a) Batch size 128
(b) Batch size
Figure 3: Test accuracy (%) by using different channel groups to calculate Extend Normalization's mean and std. Redder represents higher accuracy. When the group is c, which means the number of channels, Extend Normalization is equal to Batch Normalization.
Comparing to Switchable Normalization (SN). SN mixes both the mean and the std from BN, IN, and LN.. SN switches between them by learning their importance using trainable parameters. The selectivity of normalizers makes SN robust to batch size. We compare SN and HN using ResNet18 with various batch sizes. HN adopts the heterologous combination (BN.EN) where the group of EN 1. Figure and 2able?]shoo the results. We can find that HN outperforms SN on all batch sizes except batch size 8. For example, HN's accuracy is 0.50% higher than SN's with large batch size 128 and 0.58% higher with small batch size 2.
Different networks. We further evaluate HN different networks: MobileNetV22xx.33 Mo- bileNetV2 x1.0, ShuffleNetV2 x1.5, and Densenet121. BN, IN, LN, GN, and SN are homologous. HN adopts the heterologous combination (BN,EN) where the channel group of EN is 1. We also
5

---

Under review as a conference paper at ICLR 2022
Table 2: Test accuracies (%) of SN and HN with different batch sizes. Diff represents HN's s accuracy subtracts SN's.
RcCUrey (10
$4.80
$4.40
HN
Batch size
SN
HN
Diff
94.00
128
94.68
95.18
0.50
93.60
64
94.70
95.32
0.62
93.20
32
95.08
95.17
0.09
128
64
15
16
94.82
95.02
0.20
batch size
8
95.09
94.99
-0.10
4
94.61
94.93
0.32
93.99
94.57
0.58
Figure 4: Comparisons between SN and HN with different batch sizes.
evaluate HN_ 18,, which refers to heterologous combination (BN,NN G8) where the channel group of EN is 8 ( A larger group cannot be divided by these network channels since they have fewer channels.) Table shows the test accuracy of these networks with batch size 128. BN achieves much better performance than IN, LN, GN. SN improves the accuracy of MobileNetV2 x1.0, but degrades other networks. On MobileNerV2.x0.33. MobileNetV2 and ShuffeNNrV22x11.5 HN_G8 and HN achieve the best and the second best accuracy compared to other normalizations. On DenseNet121, HN G8 achieves comparable accuracy to BN, whose accuracy is the best.
Table 3: Test accuracies (%) on different networks. Slash means the failure of training.
BN
IN
LN
GN
SN
HN
HN.G8
89.60
82.96
1
80.40
88.95
89.91
90.20
MobileNetV2
90.92
87.42
53.48
84.93
91.40
91.49
91.92
ShuffleNetV2 x1.5
91.40
86.83
86.75
86.99
90.41
92.64
92.81
DenseNet121
95.20
92.99
86.69
91.08
94.39
94.03
95.11
4.2 CIFAR-100
CIFAR-100 100 is similar to CIFAR-10 but with 100 classes. The experiment settings the same as Section 4.1 Table 4 shows the test accuracy using ResNet18 on CIFAR-100. With the batch size 128, BN achieves the best performance 77.91
11991% %. HN and HN G32 have close accuracy, surpassing
other normalization except BN. HN's accuracy is 5. 1%, 8.32%, 3. 1..1%% %, and 0.90% higher than IN, LN, GN, and SN, respectively. With batch size HN and HN_G32 have close accuracy, surpassing all other normalization. HN_G32's accuracy is 0.55%, 5.45%, 4.69%, 2.67%, 1.57% higher than BN, IN, LN, GN, SN, respectively.
Table 4: Test accuracy (%) on CIFAR-100
BN
IN
LN
GN
SN
HN
HN_G32
Batch size = 128
77.91
71.65
68.44
73.65
75.86
76.76
76.60
Batch size 4
76.60
71.70
72.46
74.48
75.58
77.02
77.15
4.3 CALTECH256
CalTech256 has 30607 images and 256 classes. We use 80% images the training data and 20% images as the test data. Some images have only one channel, and we copy the one channel to form three channels. Other experiment settings the same as Section 4.1 except CalTech256's image size is 224x224. CalTech256 has more classes, and each class has fewer training examples. Table 5 shows the test accuracy using ResNet18 8 CalTech256. With batch size 128, SN achieves the best performance, and HN G32 achieves the second best performance. HN G32's accuracy is 0.75%, 5.61% % 7.44%, and 3.44% higher than BN, IN, LN, and GN, respectively. With batch size 4, HN

---

Under review as a conference paper at ICLR 2022
and HN_G32 surpass all other normalization. HN_G32's accuracy is 1.64%, 8.52%, 9.01 %6, 4.66%, 1.55% higher than BN, IN, LN, GN, SN, respectively.
Table 5: Test accuracy (%) on CalTech256.
BN
IN
I.N
GN
SN
HN
HN G32
Batch size 128
56.39
51.52
49.69
53.69
58.44
56.32
57.14
Batch size 4
58.512
51.64
51.151
55.492
58.60
59.491
60.16
4.4 IMAGENET
ImageNet classification dataset (Russakovsky et al. 2015 has 1. 1.28M training images and 50,000 validation images with 1000 classes. The training images are cropped with random size of 0.08 to 1.0 of the original size and a random aspect ratio of 3/4 4/3 of the original aspect ratio, and then resized to 224x224. Then random horizontal flipping is made. The validation image is resized to 256x256, and then cropped by 224x224 at the center. Each channel of the input is normalized into 0 mean and std globally. Weight decay of 0,0001, and SGD with 0.9 momentum are used. On ImageNet, we evaluate R with BN, GN, and HN. HN adopts the heterologous combination (BN.EN) where the channel group of EN is 1. Four GPUs are used. We consider batch sizes 256 and 16. The mean and the std of BN and HN are computed within each GPU. For the batch size of 256 (64 images per GPU), the initial learning rate is set to 0.1. And for the batch size of 16 (4 images per GPU), the initial learning rate is set to 0.1 * 16/ 256 following the linear scaling rule. We train the network for 100 epochs and decrease the learning rate by 10x at 30, 60, and 90 epochs.
The results of ImageNet are shown in Figure and Table 6 From Figure ((a) we can see HN achieves close performance to BN, better than GN, with large batch size. From Figure [(b) we can see HN achieves close performance to GN, better than BN, with small batch size. As shown in Table 6 with 64 images per GPU, BN achieves the best validation accuracy of 70.37%. HN gets 70.12%, worse than BN by 0.25%, but better than GN by 1.35%. With 4 images per GPU, the accuracy of BN decreases 65.78% GN achieves the best accuracy of 69.08%. HN achieves 68.54%, worse than GN by 0.54%, better than BN by 2.76%.
GN
75
70
65
scunsey
60
55
45
40
epochs
spache
(a) 64 images per GPU
(b) images per GPU
Figure 5: Test accuracy of ResNet18 on ImageNet vs. the number of training epoch.
4.5 COMBINATION WITH SWITCHABLE NORMALIZATION.
Though we have shown HN outperforms SN in most situations, we should know HN and SN are not mutually exclusive. We can combine HN and SN together to improve the performance. We evaluate two strategies for combining HN and SN. One is (BN,BN+EN), where the mean is calculated like BN, and the std is the combination of BN and EN. Another is (BN,BN+ where the mean is calculated like BN, and the std is the combination of BN, IN, and LN, as the same as SN. Table 7 shows R8 accuracies on CalTech256 with batch size 128. We can find that combinations of
7

---

Under review as a conference paper at ICLR 2022
HN and SN further improve accuracies with large and small batch sizes. (BN,BN+EN) outperforms both (BN,BN) and (BN.EN). Since the moving average value is used, there is no need to compute the mean and the std at inference time for (BN.BN+EN). (BN,BN+IN+LN) achieves the best accuracy, but the std needs to be calculated at inference time since it includes IN and L.N.
Table 6: Test accuracy (%) of ResNet18 on ImageNet.
BN
GN
HN
25
64 images per GPU
70.37
68.77
70.12
58
4 images per GPU
65.78
69.08
68.54
66
04
Table 7:
Test accuracy (%) on Cal- Tech256 when combining Heterologous Normalization with Switchable Nor- malization.
01
Batch size
mean
std
128
14
#
Adloo
BN
BN
56.39
58.51
BN
EN
56.32
59.49
BN, LN. IN
BN, LN. IN
58.44
58.60
Figure 6: Comparion of statistics' std which reflects the statistics' fluctuation quantita- tively. Neuron i-j represents the jth neuron of the inh layer, and bs refers to batch size.
BN BN
BN, EN BN, LN. IN
57.95
59.81
58.65
61.15
4.6 ANALYSIS
We train four-layer fully-comnected (FC) network on MNIST. Each FC layer have 128 neurons, BN and ReLU are used after the FC layer. We train the network with batch sizes 128 and 4. The learning rate is 0.1 when the batch size is 128. It is set to 0.1 4/128 when the batch size is 4, following the linear scaling rule. We train the network for 20 epochs. For each layer, we calculated each neuron's 's calculated the mean
mean and std along the batch dimension (BN_mean and BN_std). Moreover, and the std of the universal neurons (EN_ meaan and EN _td).. We use i-j to represent the Jrh. neuron of the deh layer. Figure 7 shows the statistics evolutions of different neurons over the last 200 training steps. To quantify the fluctuation, we show the std of these statistics themselves in Figure 6 We have the following observations:
Statistics of BN are more fluctuant than EN. That is to say, the noises brought by BN are more than EN since EN extends the pixel set for estimating statistics. When the batch size is 128, the noises brought by BN are moderate thus are helpful for training.
The difference between BN_mean and EN _eaan is larger than the difference between
BN_std and EN_std. This phenomenon is obvious in Figure 7(a) 7(e)
7(g)
That ex-
plains why replacing BN std with EN std is more effective than replacing BN mean with EN mean.
BN std have more fluctuations, while BN mean is flatter than BN std. When the batch size is 4, as shown in Figure 6 7 7(1, 7 BN_std's fluctuations become dramatic. That implies the small batch size problem BN is caused mainly by the fluctuation of BN_std. EN_std has smaller fluctuation, thus using EN_tsd instead of BN_stt can alleviate the small batch size problem of BN.
Statistics in the first layer have more significant fluctuations than following layers, as shown in Figure 6, 7(a) and [b)] especially in the case of small batch size. The lower layer is closer to the data layer, thus is influenced more easily by the noise in the data. After passing through multiple learnable layers, the noises diminish comparatively. That suggests different layers could use different strategies control the noise. For example, We may extend the pixel set for calculating both the mean and the std in the lower layer. Moreover, the extension degree (the group EN) could be different for different layers.
8

---

Under review as a conference paper at ICLR 2022
153
(a) Neuron 1-1, Batch size=128
(b) Neuron 1-1, Batch size=4
0.
255
25
04
01
322
201
(c) Neuron 2-1, Batch size=
(d) Neuron 2-1, Batch size=4
ZN, mrooe
DR
68
25
03
(c) Neuron 3-1, Batch sizc=128
(f) Neuron 3-1, Batch sizc=4
mea
DL
0.7
01
3
(g) Neuron 4-1, Batch sizc=128
(h) Neuron 4-1, Batch size=4
Figure 7: Statistics evolution over the course of training. Neuron i-j represents the jeh neuron of the ith layer. Horizontal axis represents training step. The small batch size problem Batch Normalization is caused mainly by the fluctuation of the std.
5 CONCLUSION
In this paper, we propose Heterologous Normalization (HN), which computes the mean and the std from different pixel sets to take advantage of different normalizations. HN is a general method that can use different strategies to compute the mean and the std. Moreover, we propose a particular combination that works well in most situations: calculating the mean along the (N, H, W) dimen- sions as the same as BN, while calculating the std along the (N,C,H,W) dimensions. maintains the advantage of Batch Normalization but also alleviates the problem caused by the small batch size. HN surpasses or achieves comparable performance than existing homologous methods with large small batch sizes various datasets. Besides, HN can be combined with BN together to improve the performance further. Moreover, we explain why HN conquered the small batch size problem by analyzing the evolution of the statistics over the course of training.
not only

---

Under review as a conference paper at ICLR 2022
REFERENCES
Devansh Arpit, Yingbo Zhou, Bhargava U Kota, and Venu Govindaraju. Normalization propagation: A parametric technique for removing internal covariate shift deep networks. arXiv preprint arXiv:1603.01431, 2016.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint (rXiv:1607.064500 2016.
Nils Bjorck, Carla P Gomes, Bart Selman, and Kilian Q Weinberger. Understanding batch normal- ization. In Advances in Neural Information Processing Systems, pp. 7694-7705, 2018.
Tim Cooijmans, Nicolas Ballas, César Laurent, Çağlar Gülçehre, and Aaron Courville. Recurrent batch normalization. arXiv preprint arXiv: 09025, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Elad Hoffer, Ron Banner, Itay Golan, and Daniel Soudry. Norm matters: efficient and accurate normalization schemes in deep networks. In Advances in Neural Information Processing Systems, pp. 2160-2170, 2018.
Sergey loffe. Batch renormalization: Towards reducing minibatch dependence in batch-normalized models. In Advances in neural information processing systems, pp. 1945-1953, 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by re- ducing internal covariate shift. In Proceedings of The 32nd International Conference on Machine Learning. pp. 448 456, 2015.
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self- neff-normalizing neural networks. In Advances in neural information processing systems, pp- 971-980, 2017.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. 2009.
Chunjie Luo, Jianfeng Zhan, Lei Wang, and Qiang Yang. Cosine normalization: Using cosine similarity instead of dot product in neural networks. arXiv preprint urXiv: 2017.
Ping Luo, Jiamin Ren, Zhanglin Peng, Ruimao Zhang, and Jingyu Li. Differentiable learning-to- normalize via switchable normalization. arXiv preprint arXiv:1806 10779, 2018a.
Ping Luo. Xinjiang Wang, Wenqi Shao, and Zhanglin Peng. Towards understanding regularization in batch normalization. 2018b.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprini arXiv:1802.05957 2018.
Hyeonseob Nam and Hyo-Eun Kim. Batch-instance normalization for adaptively style-invariant neural networks. In Advances in Neural Information Processing Systems, pp. 2558-2567, 2018.
Mengye Ren, Renjie Liao, Raquel Urtasun, Fabian H Sinz, and Richard S Zemel. Normaliz- ing the normalizers: Comparing and extending network normalization schemes. arXiv preprint arXiv: /tfv:/661..00 16110040, 2016.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-252, 2015.
Tim Salimans and Diederik P Kingma. Weight normalization: A simple reparameterization to accel- crate training of deep neural networks. In Advances in Neural Information Processing Systems, pp. 901-901, 2016.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems, pp. 2234-2242, 2016.
10

---

Under review as a conference paper at ICLR 2022
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch nor- malization help optimization? In Advances in Neural Information Processing Systems, pp. 2483- 2493, 2018.
Saurabh Singh and Abhinav Shrivastava. Evalnorm: Estimating batch normalization statistics for evaluation. arXiv preprini arXiv: Teti::9904000333 2019.
Cecilia Summers and M. Dinneen. Four things everyone should know to improve batch normaliza- tion, ArXiv, abs/ 090//900000544. 2020.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing in- 2016.
gredient for fast stylization. arXiv preprint
Guangrun Wang, Ping Luo, Xinjiang Wang, Liang Lin, al. Kalman normalization: Normalizing internal representations across network layers. In Advances in Neural Information Processing Systems, pp. 21-31, 2018.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 3-19, 2018.
Xiaoyong Yuan, Zheng Feng, Matthew Norton, and Xiaolin Li. Generalized batch normalization: Towards accelerating deep neural networks. Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 1682- 1689, 2019.
Ruimao Zhang, Zhanglin Peng, Lingyun Wu, Zhen Li, and Ping Luo. Exemplar normalization for learning deep representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17726-12735, 2020.


Reviews:
[{"review_id": "Uh3fP21yhSI", "reviewer": "Reviewer_Zi9s", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is easy to follow and the method seems somewhat novel.\n\nHowever, there are several weaknesses that lead me to give reject, not good enough in this stage.\n\n1) Although this method seems somewhat novel, the experiments do not support it enough. Specifically,  the results should be run at least 5 times, reporting it mean and std for CIFAR and CALTECH256. Besides, the results report for GN in CIFAR is inferior to BN, which is unreasonable to me. The author should check their parameters setting or use some official report results for double-checking.\n\n2) This paper lacks sufficient theory to explain why the combination strategy works well in most situations: calculating the mean along the (N, H, W) dimensions as the same as BN, while calculating the std along the (N,C,H,W) dimensions.", "summary_of_the_paper": "This paper proposes a new technique called Heterologous Normalization to help train modern deep networks. The main method is easy to follow. In contrast to conventional homologous normalization methods(e.g BN, IN), HN computes normalization’s mean and standard deviation from different pixel sets. The experiments show HN is slightly better than BN, and GN in some cases.\n\n\n\n\n", "summary_of_the_review": "Given the current weakness of experiments and theory part, I tend to weak reject this stage.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BNYKVju04Yk", "reviewer": "Reviewer_fNu6", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "** Pros\n\n1)\tThe proposed method is very simple and easy to implement. I particularly like the insights of using different sets to derive the statistics, which implies the mean and standard deviation terms in the normalization layer may work in different manners. It could be a good start to understand how normalization works in modern neural networks. \n2)\tWhen batch size varies, HN behaves more robust than the counterpart BN. Unlike instance-level normalizations such as IN, GN which require to calculate the statistics in inference time, HN can be totally cost-free (e.g. the configuration of BN+EN) for inference. \n\n** Cons\n\n1)\tUnfortunately, the proposed HN seems not to beat the state-of-the-art methods especially on large datasets like ImageNet. According to Table 6, HN is slightly worse than BN for large batch size, and worse than GN for small batch size. To stabilize the training on small batch sizes, a widely-used recipe GN+WS [*1] is mentioned in many pervious methods, but not compared in this paper. Moreover, as shown in Fig 4, it seems HN still suffers from performance drop as batch size decreases. Although it is better than the counterpart BN, a recent work MABN [*2] is able to eliminate the degradation even when the batch size is 1, which is also cost-free in inference time. So, I think HN may not be an optimal choice for practice use. \n2)\tIn Sec 4.6, to understand how HN works, that paper points that the std term of BN has more fluctuations than the mean term under small batch sizes. However, I find that in all cases EN introduces more stable statistics than BN, but according to Fig 2, EN+EN configuration performs not as good as EN+BN, which I think cannot be attributed to the statistic noise. I recommend the authors may investigate more in-depth interpretations. For example, does EN+BN recipe benefit from the implicit regularization effect of BN?\n\n[*1] Qiao, Siyuan, et al. \"Weight standardization.\" (2019).\n\n[*2] Yan, Junjie, et al. \"Towards stabilizing batch statistics in backward propagation of batch normalization.\" ICLR 2020.\n", "summary_of_the_paper": "The paper introduces Heterologous Normalization (HN), an alternative to normalization techniques in neural networks such as BN, LN, etc. The key insight is, the optimal statistics of mean and standard deviation may be derived from different pixel sets respectively. Based on the observation, the proposed HN calculates the mean in BN’s style, while following the way of EN to stabilizing the variance. Experiments shows HN achieves comparable or slightly better performances than BN’s in normal batch sizes. While for very small batch sizes (e.g. 4), HN significantly outperforms BN by a large margin. ", "summary_of_the_review": "Although the paper introduces some interesting insights, due to many similar works in this direction, I do not think the contribution of the paper is significant. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2qFGttHwa6", "reviewer": "Reviewer_5kEH", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The presented work is novel and well embedded \ninto other works on normalization methods. Overall, it is a \nvariation of an existing, but highly important, method. \nThe relevance of the work is somewhat limited since the \nreported performance improvements are relatively far from \ncurrent state-of-the-art. The work appears technically sound. \nAll reported performance values lack error bars. \nA hypothesis and/or explanation of why heterologous estimation\nof normalization parameters work is missing. \n\n### Strengths\n\na) The work is clearly written, well structured and well explained. \nThe structure and notation of the paper follow standards in the field.\n\n\nb) The aim of the paper is clearly written and the work is well embedded\ninto related works. Differences and similarities are explained and \nhow other normalization methods arises as special cases of heterologous \nnormalization. \n\nc) The authors perform experiments on relevant benchmarking datasets and\nalso perform and ablation study that is well-motivated. \n\n\n### Weaknesses\n\na) The presented work in the current form is limited to images with \ntheir specific data structure and layout, i.e. a sample dimension, \na channel, a width, and a height dimension. However, normalization methods\nplay a role in almost all data types and architectures (graph neural networks, \nrecurrent neural networks, transformers). The authors should at least\nprovide a perspective of the relevance and applicability of their method\nto other data types and/or a more general formulation of their approach. \n\n\nb) All experiments and reported performance values are presented without\nrepetitions and thus without error bars. Thus, difference in predictive \nperformance could arise just by chance. The authors should provide error\nbars for all experiments to demonstrate that their reported performance \nimprovements do not just arise by chance. \n\nc) The experiments are performaned within a setting that is very far \nfrom current state-of-the-art at predictive performance. E.g. on ImageNet, \nthe authors architectures operate at an accuracy of 65-70% (Table 6\nand Figure 5), whereas VGG was reported at 75% in 2014 and current\narchitectures such as NFNets [1] operate close to 90%. Therefore, the relevance\nof the proposed approach is relatively low. The authors should perform\nexperiments with current state-of-the-art models to assess whether \ntheir approach is relevant in those architectures and experiments.  \n\nd) Figures 4,5,6,7 could be improved to follow scientific standards. \n\ne) There is a lack of a hypothesis or at least explanation why this strategy\nto estimate mean and standard deviation based on different sample sets \nimproves batch normalization aside from the trivial fact that a larger\nnumber of samples are used. In that regard, also other strategies, \nsuch as using geometric averages of past parameter estimates \nshould be discussed. ", "summary_of_the_paper": "The authors present a strategy to calculate the normalizing parameters of batch normalization, i.e. mean and standard deviation, based on different sets of samples. ", "summary_of_the_review": "In the current form, I consider this work to be below the bar to \nbe accepted due to the problems and limitations mentioned above. \nI am willing to increase my score if the mentioned improvements\nare performed and presented in the paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2GShibkwZ9g", "reviewer": "Reviewer_cQjf", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThe proposed strategy is quite simple. All underlying computations are built on previous methods. It seems that there is not much to argue about on the implementation and realizability of the proposed strategy. Therefore, the main focus of the review would be more on the validity of the design principle and on the empirical evaluation and analysis:\n\n1. As the statistic of standard deviation measures how widely the values are spreading over around the mean of the set, it is odd that the mean $\\mu_{S_i}$ used in the normalization is different from the mean $\\mu_{S_i\\'}$ used for estimating the standard deviation $\\sigma_{S_i\\'}$. That is, the normalization formula actually includes two different *mean*s. It is not easy to rationalize what the formula actually calibrates. \n2. It is unclear why heterologous combinations $(BN, *)$ shown in Fig. 2 all achieve good performances on batch size 128 and what the results might imply. There are no further discussions on this point. \n3. The claim in the last paragraph of the introduction is also ambiguous and not well justified.\nWhat does it mean by \"*equilibrium between generalization and stability*\"? Is \"*controlling the **number** of pixels to calculate the statistics*\" really the key?\n4. It is reasonable that \"the difference between BN mean and EN mean is larger than the difference between BN std and EN std.\" However, it does not seem to be very meaningful to draw a conclusion as \"*that explains replacing BN std with EN std is more effective than replacing BN mean with EN mean*\". Having similar values does not justify the validity and effectiveness of replacing BN std with EN std. \nFurthermore, is the fluctuation in the standard deviation indeed the main issue for normalization with a small batch size? If that is the case, then what is the appropriate range for the standard deviation? the smaller the better?\n\n\n", "summary_of_the_paper": "This paper presents a heterologous normalization method that estimates the mean and the variance from different pixel sets for training deep networks. It is claimed in the paper that this kind of mixed statistics for normalization provides better and more stable performance on learning a deep model. Several experiments of classification tasks on CIFAR-10, CIFAR-100, Caltech-256, and ImageNet datasets are shown to support the claim. The paper also presents an analysis to illustrate the fluctuation in the statistics during training.", "summary_of_the_review": "The idea of this paper is simple, and the proposed method is straightforward and seems very easy to implement: Given all available options of existing normalization techniques, one may handily combine different techniques for estimating the mean and the variance. In practice, there might not be apparent drawbacks of trying such a heuristic strategy. On the other hand, the strategy still lacks more solid technical supports and insights to justify why it works and how to apply it for different scenarios. It seems that, for most cases, it would be much easier for the user to adopt a simpler strategy like \"*if memory is sufficient, use a large batch size + batch normalization; otherwise, use a small batch size + group normalization*\". The experimental results look fair, but the improvements are not very significant. Therefore, whereas this paper presents a new strategy to carry out normalization for training deep networks, its contributions are marginally significant and require further investigations. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers recommended reject. No responses from the authors.
Decision: Reject