Diminishing Batch Normalization

1 INTRODUCTION . Deep neural networks ( DNN ) have shown unprecedented success in various applications such as object detection . However , it still takes a long time to train a DNN until it converges . Ioffe & Szegedy identified a critical problem involved in training deep networks , internal covariate shift , and then proposed batch normalization ( BN ) to decrease this phenomenon . BN addresses this problem by normalizing the distribution of every hidden layer ’ s input . In order to do so , it calculates the preactivation mean and standard deviation using mini-batch statistics at each iteration of training and uses these estimates to normalize the input to the next layer . The output of a layer is normalized by using the batch statistics , and two new trainable parameters per neuron are introduced that capture the inverse operation . It is now a standard practice Bottou et al . ( 2016 ) ; He et al . ( 2016 ) . While this approach leads to a significant performance jump , to the best of our knowledge , there is no known theoretical guarantee for the convergence of an algorithm with BN . The difficulty of analyzing the convergence of the BN algorithm comes from the fact that not all of the BN parameters are updated by gradients . Thus , it invalidates most of the classical studies of convergence for gradient methods . In this paper , we propose a generalization of the BN algorithm , diminishing batch normalization ( DBN ) , where we update the BN parameters in a diminishing moving average way . It essentially means that the BN layer adjusts its output according to all past mini-batches instead of only the current one . It helps to reduce the problem of the original BN that the output of a BN layer on a particular training pattern depends on the other patterns in the current mini-batch , which is pointed out by Bottou et al .. By setting the layer parameter we introduce into DBN to a specific value , we recover the original BN algorithm . We give a convergence analysis of the algorithm with a two-layer batch-normalized neural network and diminishing stepsizes . We assume two layers ( the generalization to multiple layers can be made by using the same approach but substantially complicating the notation ) and an arbitrary loss function . The convergence analysis applies to any activation function that follows our common assumption . The main result shows that under diminishing stepsizes on gradient updates and updates on mini-batch statistics , and standard Lipschitz conditions on loss functions DBN converges to a stationary point . As already pointed out the primary challenge is the fact that some trainable parameters are updated by gradient while others are updated by a minor recalculation . Contributions . The main contribution of this paper is in providing a general convergence guarantee for DBN . Specifically , we make the following contributions . • In section 4 , we show the sufficient and necessary conditions for the stepsizes and diminishing weights to ensure the convergence of BN parameters . • We show that the algorithm converges to a stationary point under a general nonconvex objective function . This paper is organized as follows . In Section 2 , we review the related works and the development of the BN algorithm . We formally state our model and algorithm in Section 3 . We present our main results in Sections 4 . In Section 5 , we numerically show that the DBN algorithm outperforms the original BN algorithm . Proofs for main steps are collected in the Appendix . 2 LITERATURE REVIEW . Before the introduction of BN , it has long been known in the deep learning community that input whitening and decorrelation help to speed up the training process . In fact , Orr & Müller show that preprocessing the data by subtracting the mean , normalizing the variance , and decorrelating the input has various beneficial effects for back-propagation . Krizhevsky et al . propose a method called local response normalization which is inspired by computational neuroscience and acts as a form of lateral inhibition , i.e. , the capacity of an excited neuron to reduce the activity of its neighbors . Gülçehre & Bengio propose a standardization layer that bears significant resemblance to batch normalization , except that the two methods are motivated by very different goals and perform different tasks . Inspired by BN , several new works are taking BN as a basis for further improvements . Layer normalization Ba et al . ( 2016 ) is much like the BN except that it uses all of the summed inputs to compute the mean and variance instead of the mini-batch statistics . Besides , unlike BN , layer normalization performs precisely the same computation at training and test times . Normalization propagation that Arpit et al . uses data-independent estimations for the mean and standard deviation in every layer to reduce the internal covariate shift and make the estimation more accurate for the validation phase . Weight normalization also removes the dependencies between the examples in a minibatch so that it can be applied to recurrent models , reinforcement learning or generative models Salimans & Kingma ( 2016 ) . Cooijmans et al . propose a new way to apply batch normalization to RNN and LSTM models . Given all these flavors , the original BN method is the most popular technique and for this reason our choice of the analysis . To the best of our knowledge , we are not aware of any prior analysis of BN . BN has the gradient and non-gradient updates . Thus , nonconvex convergence results do not immediately transfer . Our analysis explicitly considers the workings of BN . However , nonconvex convergence proofs are relevant since some small portions of our analysis rely on known proofs and approaches . Neural nets are not convex , even if the loss function is convex . For classical convergence results with a nonconvex objective function and diminishing learning rate , we refer to survey papers Bertsekas ( 2011 ) ; Bertsekas & Tsitsiklis ( 2000 ) ; Bottou et al . ( 2016 ) . Bertsekas & Tsitsiklis provide a convergence result with the deterministic gradient with errors . Bottou et al . provide a convergence result with the stochastic gradient . The classic analyses showing the norm of gradients of the objective function going to zero date back to Grippo ( 1994 ) ; Polyak & Tsypkin ( 1973 ) ; Polyak ( 1987 ) . For strongly convex objective functions with a diminishing learning rate , we learn the classic convergence results from Bottou et al .. 3 MODEL AND ALGORITHM . The optimization problem for a network is an objective function consisting of a large number of component functions , that reads : min f̄ ( θ , λ ) = N∑ i=1 fi ( Xi : θ , λ ) , subject to θ ∈ P , λ ∈ Q , ( 1 ) where fi : Rn1 × Rn2 → R , i = 1 , ... , N , are real-valued functions for any data record Xi . Index i associates with data recordXi and target response yi ( hidden behind the dependency of f on i ) in the training set . Parameters θ include the common parameters updated by gradients directly associated with the loss function , i.e. , behind the part that we have a parametric model , while BN parameters λ are introduced by the BN algorithm and not updated by gradient methods but by the mini-batch statistics . We define that the derivative of fi is always taken with respect to θ : ∇fi ( Xi : θ , λ ) : = ∇θfi ( Xi : θ , λ ) . ( 2 ) The deep network we analyze has 2 fully-connected layers with D1 neurons each . The techniques presented can be extended to more layers with additional notation . Each hidden layer computes y = a ( Wu ) with activation function a ( · ) and u is the input vector of the layer . We do not need to include an intercept term since the BN algorithm automatically adjusts for it . BN is applied to the output of the first hidden layer . We next describe the computation in each layer to show how we obtain the output of the network . The notations introduced here is used in the analysis . Figure 1 shows the full structure of the network . The input data is vector X , which is one of { Xi } Ni=1 . Vector λ = ( ( µj ) D j=1 , ( σj ) D j=1 ) is the set of all BN parameters and vector θ = ( W1 , W2 , ( β ( 1 ) j ) D j=1 , ( γ ( 1 ) j ) D j=1 ) is the set of all trainable parameters which are updated by gradients . Matrices W1 , W2 are the actual model parameters and β , γ are introduced by BN . The value of jth neuron of the first hidden layer is z ( 1 ) j ( X : θ ) = a ( W1 , j , ·X ) , ( 3 ) where W1 , j , · denotes the weights of the linear transformations for the jth neuron . The jth entry of batch-normalized output of the first layer is y ( 1 ) j ( X : θ , λ ) = γ ( 1 ) j ( z ( 1 ) j ( X : θ ) − µj σj + B ) + β ( 1 ) j , where β ( 1 ) j and γ ( 1 ) j are trainable parameters updated by gradient and µj and σj are batch normalization parameters for z ( 1 ) j . Trainable parameter µj is the mini-batch mean of z ( 1 ) j and trainable parameter σj is the mini-batch sample deviation of z ( 1 ) j . Constant B keeps the denominator from zero . The output of jth entry of the output layer is : z ( 2 ) j ( X : θ ) = a ( W2 , j , · [ γ ( 1 ) j ( z ( 1 ) j ( X : θ ) − µj σj + B ) + β ( 1 ) j ] ) ( 4 ) The objective function for the ith sample is fi ( Xi : θ , λ ) = li ( ( z ( 2 ) j ( Xi : θ , λ ) ) j ) , ( 5 ) where li ( · ) is the loss function associated with the target response yi . For sample i , we have the following complete expression for the objective function : fi ( Xi : θ , λ ) = li a ( D∑ j=1 W2 , k , j [ γ ( 1 ) j a ( W1 , j , ·Xi − µj ) σj + B + β ( 1 ) j ] ) k . ( 6 ) Function fi ( Xi : θ , λ ) is nonconvex with respect to θ and λ . 3.1 ALGORITHM . Algorithm 1 shows the algorithm studied herein . There are two deviations from the standard BN algorithm , one of them actually being a generalization . We use the full gradient instead of the more popular stochastic gradient ( SG ) method . It essentially means that each batch contains the entire training set instead of a randomly chosen subset of the training set . An analysis of SG is potential future research . Although the primary motivation for full gradient update is to reduce the burdensome in showing the convergence , the full gradient method is similar to SG in the sense that both of them go through the entire training set , while full gradient goes through it deterministically and the SG goes through it in expectation . Therefore , it is reasonable to speculate that the SG method has similar convergence property as the full algorithm studied herein . Algorithm 1 DBN : Diminishing Batch-Normalized Network Update Algorithm 1 : Initialize θ ∈ Rn1 and λ ∈ Rn2 2 : for iteration m=1,2 , ... do 3 : θ ( m+1 ) : = θ ( m ) − η ( m ) ∑N i=1∇fi ( Xi : θ ( m ) , λ ( m ) ) 4 : for j=1 , ... , D1 do 5 : µ ( m+1 ) j : = 1 N ∑N i=1 z ( 1 ) j ( Xi : θ ( m+1 ) ) 6 : σ ( m+1 ) j : = √ 1 N ∑N i=1 ( z ( 1 ) j ( Xi : θ ( m+1 ) ) − µ ( m+1 ) j ) 2 7 : λ ( m+1 ) : = α ( m+1 ) ( ( µ ( m+1 ) j ) D1 j=1 , ( σ ( m+1 ) j ) D1 j=1 ) + ( 1− α ( m+1 ) ) λ ( m ) The second difference is that we update the BN parameters ( θ , λ ) by their moving averages with respect to diminishing α ( m ) . The original BN algorithm can be recovered by setting α ( m ) = 1 for every m. After introducing diminishing α ( m ) , λ ( m ) and hence the output of the BN layer is determined by the history of all past data records , instead of those solely in the last batch . Thus , the output of the BN layer becomes more general that better reflects the distribution of the entire dataset . We use two strategies to decide the values of α ( m ) . One is to use a constant smaller than 1 for all m , and the other one is to decay the α ( m ) gradually , such as α ( m ) = 1/m . In our numerical experiment , we show that Algorithm 1 outperforms the original BN algorithm , where both are based on SG and non-linear activation functions with many layers FNN and CNN models .

Summary: The authors propose a momentum based approach for batch normalization and provide an asymptotic convergence analysis of the objective in terms of the first order criterion. To my understanding, the main effort in the analysis is to show that the sequences of interest are Cauchy. Some numerical results are reported to demonstrate that the proposed variant of BN slightly outperforms BN with careful adjustment of some hyper parameter. The proposed approach is incremental, and the theoretical results are somewhat weak.