Diminishing Batch Normalization

1 INTRODUCTION . Deep neural networks ( DNN ) have shown unprecedented success in various applications such as object detection . However , it still takes a long time to train a DNN until it converges . Ioffe & Szegedy identified a critical problem involved in training deep networks , internal covariate shift , and then proposed batch normalization ( BN ) to decrease this phenomenon . BN addresses this problem by normalizing the distribution of every hidden layer ’ s input . In order to do so , it calculates the preactivation mean and standard deviation using mini-batch statistics at each iteration of training and uses these estimates to normalize the input to the next layer . The output of a layer is normalized by using the batch statistics , and two new trainable parameters per neuron are introduced that capture the inverse operation . It is now a standard practice Bottou et al . ( 2016 ) ; He et al . ( 2016 ) . While this approach leads to a significant performance jump , to the best of our knowledge , there is no known theoretical guarantee for the convergence of an algorithm with BN . The difficulty of analyzing the convergence of the BN algorithm comes from the fact that not all of the BN parameters are updated by gradients . Thus , it invalidates most of the classical studies of convergence for gradient methods . In this paper , we propose a generalization of the BN algorithm , diminishing batch normalization ( DBN ) , where we update the BN parameters in a diminishing moving average way . It essentially means that the BN layer adjusts its output according to all past mini-batches instead of only the current one . It helps to reduce the problem of the original BN that the output of a BN layer on a particular training pattern depends on the other patterns in the current mini-batch , which is pointed out by Bottou et al .. By setting the layer parameter we introduce into DBN to a specific value , we recover the original BN algorithm . We give a convergence analysis of the algorithm with a two-layer batch-normalized neural network and diminishing stepsizes . We assume two layers ( the generalization to multiple layers can be made by using the same approach but substantially complicating the notation ) and an arbitrary loss function . The convergence analysis applies to any activation function that follows our common assumption . The main result shows that under diminishing stepsizes on gradient updates and updates on mini-batch statistics , and standard Lipschitz conditions on loss functions DBN converges to a stationary point . As already pointed out the primary challenge is the fact that some trainable parameters are updated by gradient while others are updated by a minor recalculation . Contributions . The main contribution of this paper is in providing a general convergence guarantee for DBN . Specifically , we make the following contributions . • In section 4 , we show the sufficient and necessary conditions for the stepsizes and diminishing weights to ensure the convergence of BN parameters . • We show that the algorithm converges to a stationary point under a general nonconvex objective function . This paper is organized as follows . In Section 2 , we review the related works and the development of the BN algorithm . We formally state our model and algorithm in Section 3 . We present our main results in Sections 4 . In Section 5 , we numerically show that the DBN algorithm outperforms the original BN algorithm . Proofs for main steps are collected in the Appendix . 2 LITERATURE REVIEW . Before the introduction of BN , it has long been known in the deep learning community that input whitening and decorrelation help to speed up the training process . In fact , Orr & Müller show that preprocessing the data by subtracting the mean , normalizing the variance , and decorrelating the input has various beneficial effects for back-propagation . Krizhevsky et al . propose a method called local response normalization which is inspired by computational neuroscience and acts as a form of lateral inhibition , i.e. , the capacity of an excited neuron to reduce the activity of its neighbors . Gülçehre & Bengio propose a standardization layer that bears significant resemblance to batch normalization , except that the two methods are motivated by very different goals and perform different tasks . Inspired by BN , several new works are taking BN as a basis for further improvements . Layer normalization Ba et al . ( 2016 ) is much like the BN except that it uses all of the summed inputs to compute the mean and variance instead of the mini-batch statistics . Besides , unlike BN , layer normalization performs precisely the same computation at training and test times . Normalization propagation that Arpit et al . uses data-independent estimations for the mean and standard deviation in every layer to reduce the internal covariate shift and make the estimation more accurate for the validation phase . Weight normalization also removes the dependencies between the examples in a minibatch so that it can be applied to recurrent models , reinforcement learning or generative models Salimans & Kingma ( 2016 ) . Cooijmans et al . propose a new way to apply batch normalization to RNN and LSTM models . Given all these flavors , the original BN method is the most popular technique and for this reason our choice of the analysis . To the best of our knowledge , we are not aware of any prior analysis of BN . BN has the gradient and non-gradient updates . Thus , nonconvex convergence results do not immediately transfer . Our analysis explicitly considers the workings of BN . However , nonconvex convergence proofs are relevant since some small portions of our analysis rely on known proofs and approaches . Neural nets are not convex , even if the loss function is convex . For classical convergence results with a nonconvex objective function and diminishing learning rate , we refer to survey papers Bertsekas ( 2011 ) ; Bertsekas & Tsitsiklis ( 2000 ) ; Bottou et al . ( 2016 ) . Bertsekas & Tsitsiklis provide a convergence result with the deterministic gradient with errors . Bottou et al . provide a convergence result with the stochastic gradient . The classic analyses showing the norm of gradients of the objective function going to zero date back to Grippo ( 1994 ) ; Polyak & Tsypkin ( 1973 ) ; Polyak ( 1987 ) . For strongly convex objective functions with a diminishing learning rate , we learn the classic convergence results from Bottou et al .. 3 MODEL AND ALGORITHM . The optimization problem for a network is an objective function consisting of a large number of component functions , that reads : min f̄ ( θ , λ ) = N∑ i=1 fi ( Xi : θ , λ ) , subject to θ ∈ P , λ ∈ Q , ( 1 ) where fi : Rn1 × Rn2 → R , i = 1 , ... , N , are real-valued functions for any data record Xi . Index i associates with data recordXi and target response yi ( hidden behind the dependency of f on i ) in the training set . Parameters θ include the common parameters updated by gradients directly associated with the loss function , i.e. , behind the part that we have a parametric model , while BN parameters λ are introduced by the BN algorithm and not updated by gradient methods but by the mini-batch statistics . We define that the derivative of fi is always taken with respect to θ : ∇fi ( Xi : θ , λ ) : = ∇θfi ( Xi : θ , λ ) . ( 2 ) The deep network we analyze has 2 fully-connected layers with D1 neurons each . The techniques presented can be extended to more layers with additional notation . Each hidden layer computes y = a ( Wu ) with activation function a ( · ) and u is the input vector of the layer . We do not need to include an intercept term since the BN algorithm automatically adjusts for it . BN is applied to the output of the first hidden layer . We next describe the computation in each layer to show how we obtain the output of the network . The notations introduced here is used in the analysis . Figure 1 shows the full structure of the network . The input data is vector X , which is one of { Xi } Ni=1 . Vector λ = ( ( µj ) D j=1 , ( σj ) D j=1 ) is the set of all BN parameters and vector θ = ( W1 , W2 , ( β ( 1 ) j ) D j=1 , ( γ ( 1 ) j ) D j=1 ) is the set of all trainable parameters which are updated by gradients . Matrices W1 , W2 are the actual model parameters and β , γ are introduced by BN . The value of jth neuron of the first hidden layer is z ( 1 ) j ( X : θ ) = a ( W1 , j , ·X ) , ( 3 ) where W1 , j , · denotes the weights of the linear transformations for the jth neuron . The jth entry of batch-normalized output of the first layer is y ( 1 ) j ( X : θ , λ ) = γ ( 1 ) j ( z ( 1 ) j ( X : θ ) − µj σj + B ) + β ( 1 ) j , where β ( 1 ) j and γ ( 1 ) j are trainable parameters updated by gradient and µj and σj are batch normalization parameters for z ( 1 ) j . Trainable parameter µj is the mini-batch mean of z ( 1 ) j and trainable parameter σj is the mini-batch sample deviation of z ( 1 ) j . Constant B keeps the denominator from zero . The output of jth entry of the output layer is : z ( 2 ) j ( X : θ ) = a ( W2 , j , · [ γ ( 1 ) j ( z ( 1 ) j ( X : θ ) − µj σj + B ) + β ( 1 ) j ] ) ( 4 ) The objective function for the ith sample is fi ( Xi : θ , λ ) = li ( ( z ( 2 ) j ( Xi : θ , λ ) ) j ) , ( 5 ) where li ( · ) is the loss function associated with the target response yi . For sample i , we have the following complete expression for the objective function : fi ( Xi : θ , λ ) = li a ( D∑ j=1 W2 , k , j [ γ ( 1 ) j a ( W1 , j , ·Xi − µj ) σj + B + β ( 1 ) j ] ) k . ( 6 ) Function fi ( Xi : θ , λ ) is nonconvex with respect to θ and λ . 3.1 ALGORITHM . Algorithm 1 shows the algorithm studied herein . There are two deviations from the standard BN algorithm , one of them actually being a generalization . We use the full gradient instead of the more popular stochastic gradient ( SG ) method . It essentially means that each batch contains the entire training set instead of a randomly chosen subset of the training set . An analysis of SG is potential future research . Although the primary motivation for full gradient update is to reduce the burdensome in showing the convergence , the full gradient method is similar to SG in the sense that both of them go through the entire training set , while full gradient goes through it deterministically and the SG goes through it in expectation . Therefore , it is reasonable to speculate that the SG method has similar convergence property as the full algorithm studied herein . Algorithm 1 DBN : Diminishing Batch-Normalized Network Update Algorithm 1 : Initialize θ ∈ Rn1 and λ ∈ Rn2 2 : for iteration m=1,2 , ... do 3 : θ ( m+1 ) : = θ ( m ) − η ( m ) ∑N i=1∇fi ( Xi : θ ( m ) , λ ( m ) ) 4 : for j=1 , ... , D1 do 5 : µ ( m+1 ) j : = 1 N ∑N i=1 z ( 1 ) j ( Xi : θ ( m+1 ) ) 6 : σ ( m+1 ) j : = √ 1 N ∑N i=1 ( z ( 1 ) j ( Xi : θ ( m+1 ) ) − µ ( m+1 ) j ) 2 7 : λ ( m+1 ) : = α ( m+1 ) ( ( µ ( m+1 ) j ) D1 j=1 , ( σ ( m+1 ) j ) D1 j=1 ) + ( 1− α ( m+1 ) ) λ ( m ) The second difference is that we update the BN parameters ( θ , λ ) by their moving averages with respect to diminishing α ( m ) . The original BN algorithm can be recovered by setting α ( m ) = 1 for every m. After introducing diminishing α ( m ) , λ ( m ) and hence the output of the BN layer is determined by the history of all past data records , instead of those solely in the last batch . Thus , the output of the BN layer becomes more general that better reflects the distribution of the entire dataset . We use two strategies to decide the values of α ( m ) . One is to use a constant smaller than 1 for all m , and the other one is to decay the α ( m ) gradually , such as α ( m ) = 1/m . In our numerical experiment , we show that Algorithm 1 outperforms the original BN algorithm , where both are based on SG and non-linear activation functions with many layers FNN and CNN models .

Summary: In this work, the authors propose a generalization of the batch normalization (BN) technique often used in training neural networks, and analyzed this convergence. In particular, a one hidden layer and one BN hidden layer fully connected network is considered, and a deterministic gradient descent algorithm with certain kind of BN has been considered in this work. The proposed “generalized” BN strategy is devised on the deterministic setting, but it is a slight generalization of the original BN by introducing a moving average operation. Classical results of Bertsekas is leveraged to show the asymptotic convergence of the algorithm.