Heterologous  Normalization

1 INTRODUCTION . Deep neural networks have received great success in many areas . Batch Normalization ( BN ) has become a standard technique for training modern deep networks . BN normalizes the features by the mean and the standard deviation ( std ) computed from a batch of samples . The coordination between examples helps the learning process . The random selection of examples in the minibatch brings the sampling noises , providing a regularization effect . However , its effectiveness diminishes when the batch size becomes smaller since the noises are too much and make inaccurate batch statistics estimation . That hinders BN ’ s usage for scenes lacking samples or memories . For example , federated learning , a hot topic in machine learning , aims to train a model across multiple decentralized edge devices or servers to address privacy and security issues . The heterogeneous environments require a robust training algorithm with large or small batch size . To address the small batch size problem , some methods try to avoid normalizing along the batch dimension . In the case of NCHW format feature map , let N refer to the batch dimension , C refer to the channel dimension , H and W refer to the spatial height and width dimensions . Layer Normalization ( LN ) ( Ba et al. , 2016 ) computes the mean and the std along ( C , H , W ) dimensions . Instance Normalization ( IN ) ( Ulyanov et al. , 2016 ) computes the mean and the std along ( H , W ) dimensions . Group Normalization ( GN ) ( Wu & He , 2018 ) is an intermediate state between Layer Normalization and Instance Normalization . It uses a group of channels within the sample itself to compute the mean and the std . Avoiding normalizing along the batch dimension gives up the advantages of BN , resulting in poor performances in many cases . To take advantage of different approaches , Switchable Normalization ( SN ) ( Luo et al. , 2018a ) combines different normalizations . It uses learnable parameters to combine BN , IN , and LN linearly . SN brings more statistics and computational complexity . Besides , the mean and the std of those methods need to be calculated at inference time . They slow down the inference comparing to BN since BN ’ s mean and std are pre-computed from the training data by the moving average . Although different normalizations compute the statistics from different pixel sets ( pixel refers to a element in the feature map ) , a specific normalization computes the mean and the std from the same pixel set . Thus existing methods can be viewed as homologous normalization . In this paper , we propose Heterologous Normalization ( HN ) , which computes the mean and the std from different pixel sets to take advantage of different normalization methods . Although HN is a general method that can use different strategies to compute the mean and the std , we find this combination strategy works well in most situations : calculating the mean along the ( N , H , W ) dimensions as the same as BN , while calculating the std along the ( N , C , H , W ) dimensions . On the one hand , it maintains the advantage of batch normalization when the batch size is large . On the other hand , it enlarges the number of pixels from which the std is calculated , thus alleviating the problem caused by the small batch size . At inference time , HN ’ s mean and std are pre-computed from the training set by moving average as the same as BN , keeping the inference efficiency . We evaluate HN on various datasets : CIFAR-10 , CIFAR-100 , CalTech256 , ImageNet . Experiments show that heterologous combination is valid for most situations . HN surpasses or achieves comparable performance than BN , IN , LN , GN , and SN , with large or small batch sizes on various datasets . Moreover , HN can be combined with SN together to improve the performance further . By analyzing the statistics ’ evolution over the course of training , we find that the noise of the small batch size is mainly caused by the std fluctuation rather than the mean . Enlarging the number of pixels from which the std is calculated can alleviate fluctuation successfully . That explains why HN works well with the small batch size . We conclude our key contributions as follows : 1 ) We show that it is unnecessary to estimate normalization statistics from the same pixel set and propose a general Heterologous Normalization that calculates normalization statistics from different pixel sets . 2 ) We find a special heterologous method that surpasses or achieves comparable performance to existing homologous methods , with large or small batch sizes on various datasets . 3 ) We find the noise of the small batch size is mainly caused by the std fluctuation rather than the mean . We should make an equilibrium between generalization and stability by controlling the number of pixels to calculate the statistics . 2 RELATED WORK . Batch Normalization ( BN ) ( Ioffe & Szegedy , 2015 ) is effective at accelerating and improving the training of deep neural networks by reducing internal covariate shift . It performs the normalization for each training minibatch along ( N , H , W ) dimensions in the case of NCHW format feature . Since BN uses the statistics on minibatch examples , its effect is dependent on the minibatch size . To mitigate this problem , Normalization Propagation ( Arpit et al. , 2016 ) uses a data-independent parametric estimate of the mean and standard deviation instead of explicitly calculating from data . Batch Renormalization ( Ioffe , 2017 ) introduces two extra parameters to correct the fact that the minibatch statistics differ from the population ones . It needs to train the model for a certain number of iterations with Batch Normalization alone , without the correction , then ramps up the amount of allowed correction . There is a family of methods that avoid normalizing along the batch dimension . Layer Normalization ( LN ) ( Ba et al. , 2016 ) computes the mean and standard deviation along ( C , H , W ) dimensions . Instance Normalization ( IN ) ( Ulyanov et al. , 2016 ) computes the mean and standard deviation along ( H , W ) dimensions . When the batch size is 1 , Batch Normalization is equivalent to Instance Normalization . Group Normalization ( GN ) ( Wu & He , 2018 ) is an intermediate state between Layer Normalization and Instance Normalization . It uses a group of channels to compute the mean and the std , while Layer Normalization uses all channels , and Instance Normalization uses one channel . To take advantage of different approaches , Switchable Normalization ( SN ) ( Luo et al. , 2018a ) , Exemplar Normalization Zhang et al . ( 2020 ) and Batch-Instance Normalization ( BIN ) ( Nam & Kim , 2018 ) try to combine different normalization together . Switchable and Exemplar Normalization use learnable parameters to combine Batch , Instance , and Layer Normalization . BIN uses a learnable gate parameter to combine Batch and Instance normalization . Weight normalization ( Salimans & Kingma , 2016 ) normalizes the filter weights instead of the activations by re-parameterizing the incoming weight vector . Cosine normalization ( Luo et al. , 2017 ) normalizes both the filter weights and the activations by using cosine similarity instead of dot product in neural networks . Some researchers try to use other statistics instead of the mean and the standard deviation in normalization . Instead of the standard L2 batch normalization , ( Hoffer et al. , 2018 ) uses normalization in L1 and L∞ spaces and shows that it can improve numerical stability in low-precision implementations as well as provide computational and memory benefits . Generalized batch normalization ( Yuan et al. , 2019 ) investigates a variety of alternative deviation measures for scaling and alternative mean measures for centering . Virtual batch normalization ( Salimans et al. , 2016 ) and spectral normalization ( Miyato et al. , 2018 ) focus on the normalization in generative adversarial networks . Self-Normalizing ( Klambauer et al. , 2017 ) focuses on standard feed-forward neural networks ( fully-connected networks ) . Recurrent batch normalization ( Cooijmans et al. , 2016 ) modifies batch normalization to use in recurrent networks . Kalman normalization ( Wang et al. , 2018 ) estimates the mean and standard deviation of a certain layer by considering the distributions of all its preceding layers , mimicking the merits of Kalman Filtering Process . EvalNorm ( Singh & Shrivastava , 2019 ) estimates corrected normalization statistics to use for batch normalization during evaluation . ( Ren et al. , 2016 ) provides a unifying view of the different normalization approaches . ( Santurkar et al. , 2018 ) , ( Luo et al. , 2018b ) and ( Bjorck et al. , 2018 ) try to explain how batch normalization works . Summers & Dinneen ( 2020 ) propose several useful techniques to improve batch normalization . 3 HETEROLOGOUS NORMALIZATION . We first describe some notations that will be used next . For NCHW format feature map , let U denote the universal pixel set in the same feature layer , UN denote the set of pixels that belong to the same example , UC denote the set of pixels that belong to the same channel , UG denote the set of pixels that belong to the same group of channels . A family of normalization can be formalized as : x̂i = 1 σS′i ( xi − µSi ) ( 1 ) yi = γx̂i + β ( 2 ) where xi is the input , and yi is the output of the normalization . µ is the mean , and σ is the standard deviation ( std ) . Si is the pixel set from which the mean is computed , and S ′ i is the pixel set from which the std is computed . γ and β are learned parameters for affine transformation . As shown in Table 1 , different normalization methods estimate statistics from different pixel sets . BN computes the mean and the std along the ( N , H , W ) dimensions . The random noise brought by batch statistics estimation is beneficial to the generalization . However , the batch statistics estimation brings too much noise when the batch size is small . IN , LN , and GN try to avoid estimate statistics along the batch dimension . Each example estimates statistics within its own pixel sets , ignoring the information of other examples . Since there are not enough pixels to estimate statistics for BN when batch size is small , a straightforward method is to extend the pixel set for statistics calculation . For example , we can use the universal pixels U or the pixels from a group of channels UG to calculate statistics . We use Extended Normalization ( EN ) to refer to that method . In the following , the default number of groups in EN is 1 ( the universal set U ) when there is no mention of it , and we use EN Gn to represent that the groups of channels are n in EN . Figure 1 shows the visualization of different normalizations . Although different normalizations compute the statistics from different pixel sets , a specific normalization computes the mean and the std from the same pixel set . Thus existing methods can be viewed as homologous normalization , which has Si = S ′ i ( 3 ) In this paper , we propose Heterologous Normalization ( HN ) , which computes the mean and the std from different of pixel sets to take advantage of different normalization methods . That is to say , in HN , we have Si 6= S ′ i ( 4 ) Although HN is a general method that can use different strategies to compute the mean and the std , we find this configure works well in most situations : Si = UC , S ′ i = UG ( 5 ) Specifically , the mean is calculated along the ( N , H , W ) dimensions as the same as BN , while the std is computed along the ( N , C , H , W ) dimensions as the same as EN . On the one hand , it maintains BN ’ s advantage when the batch size is large . On the other hand , it enlarges the number of pixels from which the std is calculated , thus alleviating the problem caused by the small batch size . Switchable Normalization ( SN ) ( Luo et al. , 2018a ) also try to take advantage of different normalizations by mixing them together . Its mean and std are both linear combinations of the batch , instance , and layer ones . The pixel sets , from which the mean and the std are calculated , are the same . Different from SN , HN calculates the mean and the std from heterogeneous pixel sets . HN has fewer statistics and computational complexity than SN . Besides , the mean and the std of HN are pre-computed from the training data by the moving average as the same as BN . There is no need to compute the mean and the std at inference time . Moreover , since the mean and the std are pre-computed and fixed at inference time , the normalization can further be fused into convolution operation . That is very helpful to speed up the inference , especially on mobile or embedded devices . As for IN , LN , GN , SN , the mean and the std need to be calculated at inference time . Finally , HN and SN are not mutually exclusive . We can combine HN and SN together . For example , we can use the mean of BN , and the linear combinations of the std from different normalizations .

Summary: The paper introduces Heterologous Normalization (HN), an alternative to normalization techniques in neural networks such as BN, LN, etc. The key insight is, the optimal statistics of mean and standard deviation may be derived from different pixel sets respectively. Based on the observation, the proposed HN calculates the mean in BN’s style, while following the way of EN to stabilizing the variance. Experiments shows HN achieves comparable or slightly better performances than BN’s in normal batch sizes. While for very small batch sizes (e.g. 4), HN significantly outperforms BN by a large margin.