Improving Batch Normalization with Skewness Reduction for Deep Neural Networks

1 INTRODUCTION . In recent years , deep neural networks have been applied to many visual computing tasks , such as image recognition ( Krizhevsky et al. , 2012 ; Huang et al. , 2017 ) , image super-resolution ( Tong et al. , 2017 ) , video-based activity recognition ( Feichtenhofer et al. , 2016 ) , etc . ( Ronneberger et al. , 2015 ; Feichtenhofer et al. , 2018 ) , achieving promising results . These models are usually trained with stochastic gradient descent or its variants . State-of-the-art neural networks often have many layers , which means they have a lot of parameters to learn , leading to practical issues including long training time and high risk of overfitting . To facilitate learning with gradient descent , Batch Normalization ( BN ) was proposed in ( Ioffe & Szegedy , 2015 ) , which has been found very effective in deep learning . A BN layer normalizes the batch input to zero mean and unit variance . ( In practice , a BN layer learns a mapping that does not necessarily maintain the “ zero mean , unit variance ” property for the outputs . But that level of detail will not affect the validity of the discussion here . ) This has been shown to improve the speed of convergence in training deep neural networks as well as improving the performance ( He et al. , 2016 ) , and hence BN has become one common component of many popular deep networks . We have discovered that , making the distributions of the features in the same layer more similar would make the network performs better . However , the standard BN procedure only normalizes the features to ensure that they have the same mean and variance . This does not necessarily make the distributions of the features in the same layer to become similar . For example , an exponential distribution can also have zero mean and unit variance . In other words , the standard BN , while performing normalization with respect to the mean and the variance , will not ensure the features of different layers to have similar distributions . Note that , the mean and the variance are only the first-order and second-order moments , respectively , for a distribution . To further encourage the distributions to become closer , we propose to introduce an extra dimension of normalization by mapping the data to ensure they have similar skewness . Skewness is a measure of the asymmetry of a distribution , and we hypothesize that including this measure will provide a much stronger constraint towards making these distributions become similar . From another point of view , modifying skewness requires nonlinear operations . Recent research ( Pascanu et al. , 2013b ; Montufar et al. , 2014 ) has shown that deep neural networks are more expressive while stacking up the nonlinear activation with more layers . The nonlinearity introduced for modifying skewness may further contribute to improving the network ’ s capacity in approaching any desired input-output mapping ( which is typically highly nonlinear ) , and thus making network learning more flexible . In this paper , we present a novel approach for improving BN with skewness reduction ( BNSR ) for training deep neural networks . We notice that , during training , our approach can make the feature distributions to be similar with fewer epochs . Also , we demonstrate that , it is more effective while applying BNSR on the layer with more dissimilar distributions of the features . We further compare our proposed method with other normalization schemes , including Batch Normlization ( BN ) , Layer Normlization ( LN ) and Instance Normlization ( IN ) on CIFAR-100 and ImageNet datasets . Experimental results show that BNSR outperforms all of them . Our contributions are summarized as follows : • We propose a new batch normalization scheme . To our best knowledge , this is the first work to consider skewness for normalization . • The scheme introduces a nonlinear function , which not only decreases the skewness of the feature distributions , but also increases the flexibility of the network . • We demonstrate that our approach outperforms other normalization approach on visual recognition tasks . 2 RELATED WORK . In this section , we first give a review for the related works on normalization , followed by a brief description of the recent understanding to BN . 2.1 NORMALIZATION . To shorten the training stage , researchers normalize the input data ( LeCun et al. , 2012 ) . Alternatively , some initialization methods ( LeCun et al. , 2012 ; Glorot & Bengio , 2010 ; Wiesler & Ney , 2011 ) are proposed . However , such methods have their limitations as they were proposed based on strong assumptions of the feature distributions . Algorithm 1 : Training stage of BNSR , applied to features x over a mini-batch Input : Values of x over a mini-batch : B = { x1 ... m } ; Parameters : Parameters to be learned : γ , β Output : yi = BNγ , β ( xi ) 1 µB ← 1m ∑m i=1 xi 2 σB 2 ← 1m ∑m i=1 ( xi − µB ) 2 3 x̂i ← ϕp ( x̂i ) 4 yi ← γx̂i + β ≡ BNSRγ , β ( xi ) Algorithm 2 : Testing stage of BNSR , applied to features x over a mini-batch Input : Values of x over a mini-batch : B = { x1 ... m } ; Output : yi = BNγ , β ( xi ) 1 Calculate the population µ , σ by unbias estimation or exponential moving average 2 for i = 1 . . .m do 3 x̂i ← xi−µ√σ2+ 4 x̂i ← ϕp ( x̂i ) 5 end 6 yi = γx̂i + β Before the proposal of BN , normalization layers like Local Response Normalization ( LRN ) ( Lyu & Simoncelli , 2008 ; Jarrett et al. , 2009 ; Krizhevsky et al. , 2012 ) , which computes the statistics of the local neighborhood for each pixel , was usually used in deep neural networks ( Krizhevsky et al. , 2012 ) . Unlike LRN , Batch Normalization ( Ioffe & Szegedy , 2015 ) normalizes the features along the batch axis , which makes the normalization more global . Besides , BN also allows higher learning rates . While large learning rates increase the scale of the weights of the network , back propagation with BN is unaffected by the scale of its weights . Also , for a given training sample , a neural network with BN does not always provide a fixed value , making BN serve as a regularization of the network . Another modification that BN made is the introduction of a pair of parameters β and γ , which shift and scale the normalized features . These processes allow the BN layer to represent identity mapping , and increase the flexibility of the network . Since placing a ReLU layer after the BN is common , to avoid half of the neurons to be mapped to zero , it is also beneficial to do this transformation between the normalization and activation . Meanwhile , it is worth mentioning that , in the inference stage , BN does not exactly normalize the input : the mean and variance it used are pre-computed from the training set , such that if only one sample is sent to the network ( the mini-batch contains only one sample , and thus the mean is the feature itself and variance equals to zero ) , the BN layers can still functioning correctly . Many normalization methods ( Ba et al. , 2016 ; Ulyanov et al. , 2016 ; Salimans & Kingma , 2016 ; Luo et al. , 2018 ; Wu & He , 2018 ) were proposed after BN . Layer Normalization ( LN ) ( Ba et al. , 2016 ) performs the normalization over all the hidden units in the same layer ; Instance Normalization ( IN ) ( Ulyanov et al. , 2016 ) proposes to normalize each sample ; Weight Normalization ( WN ) ( Salimans & Kingma , 2016 ) , instead of normalizing the input of the layers , operates the normalization on the filter weights ; Group Normalization ( GN ) ( Wu & He , 2018 ) divides channels into groups , and computes the normalization statistics of the features within each group . Comparing to BN , although these methods have their strength , they in general do not outperform BN in many visual classification/recognition problems . In addition , ELU ( Clevert et al. , 2015 ) , PoLU ( Li et al. , 2018 ) and SELU ( Klambauer et al. , 2017 ) were proposed as new nonlinear activation functions , which have negative saturation < 1 , and use this property to push the mean of the output closer to zero . Computing these activations are with lower computational complexity , and can be served as an alternative to the feature normalization . Table 1 : Comparison of error rates ( % ) of BNSR , BN , BN with noisy mean and variance , BN with noisy skewness on CIFAR-100 . The training loss and error rate curves are in Fig . 2 BNSR BN Noise ( µ , σ ) Noise ( ρ ) error 30.61 31.35 33.52 32.1 Figure 2 : Comparison of performance among ( 1 ) BNSR ; ( 2 ) BN ; ( 3 ) BN with noisy mean and variance ; ( 3 ) BN with noisy skewness on CIFAR-100 . We show ( a ) the training loss ; ( b ) the testing error v.s . numbers of training epochs . The model is VGG-19 . 2.2 RECENT RESEARCH ON BN . It has been shown that ( LeCun et al. , 2012 ; Wiesler & Ney , 2011 ) , in the training stage , if the features are whitened ( the inputs have zero means , unit variances , and decorrelated ) the training process can be speeded up . However , adding the whitening process for each layer is costly . To overcome this , Batch Normalization , which linear-transforms the features to have zero means and unit variances , was proposed to normalize the features . It was believed that , the benefits of doing BN supposedly come from the reduction of the internal covariate shift effect ( Ioffe & Szegedy , 2015 ) , which is defined as the change in the distribution of the features of the layers ( Ioffe & Szegedy , 2015 ) , due to the variation of the network parameters during learning . By reducing the internal covariate shift , training can be improved . However , a recent paper ( Santurkar et al. , 2018 ) demonstrated that , the internal covariate shift effect has little to do with the effectiveness of BN . They further point out that , by both experiments and theoretical analysis , the success of BN comes from smoothness of the loss surface . In this paper , we demonstrate that the distributions of the features in the same layer affect the performance of the network . The network is improved while these distributions are more similar . Nevertheless , as illustrated in Fig . 1a , we can not conclude that two distributions are similar if they have the same mean and variance . Therefore , we would like to develop a new normalization scheme such that the distributions have a higher chance to become closer after normalization . For our proposed approach , Batch Normalization with Skewness Reduction ( BNSR ) , we adopt the advantages of original BN mentioned above . First , the features are normalized to the same mean and variance to make the loss surface smoother . Second , BNSR adopts the re-scaling parameter γ and the re-centering parameter β such that the network has greater flexibility . We further impose a novel step - Skewness Reduction into normalization , to encourage the distributions of the features in the same layer to become further closer . More details are to be presented in the next section .

Summary: The paper proposes to add an extra nonlinearity function in batch normalization, between the normalization and affine scaling. The nonlinearity is a power function x ** p for x >= 0 and - (-x) ** p for x < 0 (python pseudo-code), where p is a constant, and the authors propose to keep it fixed to 1.01. The intuition behind is reducing skewness of activations, and the modification is evaluated on CIFAR and tiny ImageNet datasets. The authors also experiment with the part at which to insert the nonlinearity.