Spherical Motion Dynamics: Learning Dynamics of Neural Network with Normalization, Weight Decay, and SGD

1 INTRODUCTION AND BACKGROUND . Normalization techniques ( e.g . Batch Normalization ( Ioffe & Szegedy , 2015 ) or its variants ) are one of the most commonly adopted techniques for training deep neural networks ( DNN ) . A typical normalization can be formulated as following : consider a single unit in a neural network , the input is X , the weight of linear layer is w ( bias is included in w ) , then its output is y ( X ; w ; γ ; β ) = g ( Xw − µ ( Xw ) σ ( wX ) γ + β ) , ( 1 ) where g is a nonlinear activation function like ReLU or sigmoid , µ , σ are mean and standard deviation computed across specific dimension of Xw ( like Batch Normalization ( Ioffe & Szegedy , 2015 ) , Layer Normalization Ba et al . ( 2016 ) , Group Normalization ( Wu & He , 2018 ) , etc. ) . β , γ are learnable parameters to remedy for the limited range of normalized feature map . Aside from normalizing feature map , Salimans & Kingma ( 2016 ) normalizes weight by l2 norm instead : y ( X ; w ; γ ; β ) = g ( X w ||w||2 γ + β ) , ( 2 ) where || · ||2 denotes l2 norm of a vector . Characterizing evolving of networks during training . Though formulated in different manners , all normalization techniques mentioned above share an interesting property : the weight w affiliated with a normalized unit is scale-invariant : ∀α ∈ R+ , y ( X ; αW ; γ , β ) = y ( X ; w ; γ , β ) . Due to the scale-invariant property of weight , the Euclidean distance defined in weight space completely fails to measure the evolving of DNN during learning process . As a result , original definition of learning rate η can not sufficiently represent the update efficiency of normalized DNN . To deal with such issue , van Laarhoven ( 2017 ) ; Hoffer et al . ( 2018 ) ; Zhang et al . ( 2019 ) propose “ effective learning rate ” as a substitute for learning rate to measure the update efficiency of normalized neural network with stochastic gradient descent ( SGD ) , defined as ηeff = η ||w||22 . ( 3 ) Joint effects of normalization and weight decay . van Laarhoven ( 2017 ) explores the joint effect of normalization and weight decay ( WD ) , and obtains the magnitudes of weight by assuming the convergence of weight , i.e . if wt = wt+1 , the weight norm can be approximated as ||wt||2 = O ( 4 √ η/λ ) , where λ is WD coefficient . Combining with Eq . ( 3 ) , we have ηeff = √ ηλ . A more intuitive demonstration about relationship between normalization and weight decay is presented in Chiley et al . ( 2019 ) ( see Figure 1 ) : due to the fact that the gradient of scale invariant weight ∂L/∂w ( L is the loss function of normalized network without WD part ) is always perpendicular to weight w , one can infer that gradient component ∂L/∂w always tends to increase the weight norm , while the gradient component provided by WD always tends to reduce weight norm . Thus if weight norm remains unchanged , or “ equilibrium has been reached ” 1 , one can obtain wt −wt+1 ||wt||2 = √ 2ηλ ∂L/∂w E||∂L/∂w||2 . ( 4 ) Eq . ( 4 ) implies the magnitude of update is scale-invariant of gradients , and effective learning rate should be √ 2ηλ . Li & Arora ( 2020 ) manages to estimate the magnitude of update in SGDM , their result is presented in limit and accumulation manner : if both limT−→∞RT = 1/T ∑T t=0 ||wt|| and limT−→∞DT = 1/T ∑T t=0 ||wt −wt+1|| exist , then we have lim T−→∞ DT RT = 2ηλ 1 + α . ( 5 ) Though not rigorously , one can easily speculate from Eq . ( 5 ) the magnitude of update in SGDM cases should be √ 2ηλ/ ( 1 + α ) in equilibrium condition . But proof of Eq . ( 5 ) requires more strong assumptions : not only convergence of weight norm , but also convergence of update norm ‖wt+1 − wt‖2 ( both in accumulation manner ) . As discussed above , all previous qualitative results about “ effective learning rate ” ( van Laarhoven , 2017 ; Chiley et al. , 2019 ; Li & Arora , 2020 ) highly rely on equilibrium condition , but none of them explores why such equilibrium condition can be achieved . Only van Laarhoven ( 2017 ) simply interprets the occurrence of equilibrium as a natural result of convergence of optimization , i.e . when optimization is close to be finished , wt = wt+1 , resulting in equilibrium condition . However , this interpretation has an apparent contradiction : according to Eq . ( 4 ) and ( 5 ) , when equilibrium condition is reached , the magnitude of update is constant , only determined by hyper-parameters , which means optimization process has not converged yet . Li & Arora ( 2020 ) also notices the non-convergence of SGD with BN and WD , so they do not discuss reasonableness of assumptions adopted by Eq. ( 5 ) . In a word , previous results about “ effective learning rate ” in equilibrium condition can only provide vague insights , they are difficult to be connected with empirical observation . In this work , we comprehensively reveal the learning dynamics of normalized neural network using stochastic gradient descent without/with momentum ( SGD/SGDM ) and weight decay , named as Spherical Motion Dynamics ( SMD ) . Our investigation aims to answer the following question : 1 ” Weight norm remains unchanged ” means ||wt||2 ≈ ||wt+1||2 , Chiley et al . ( 2019 ) calls this condition as “ equilibrium ” , which will also be used in the following context of this paper . Note equilibrium condition is not mathematically rigorous , we only use it for intuitive analysis . Why and how equilibrium condition can be reached in Spherical Motion Dynamics ? Specifically , our contributions are • We introduce the assumptions that can lead to equilibrium condition in SMD , and justify their reasonableness by sufficient experiments . We also prove under given assumptions , equilibrium condition can be reached as weight norm approach to its theoretical value in a linear rate regime in SMD . Our assumptions show the equilibrium condition can occur long before the finish of the whole optimization ; • We define a novel index , angular update , to measure the change of normalized neural network in a single iteration , and derive its theoretical value under equilibrium condition in SMD . We also prove angular update can approach to its theoretical value in a linear regime along with behavior of weight norm . Our results imply that the update efficiency of SGD/SGDM on normalized neural network only depends on pre-defined hyper-parameters in both SGD and SGDM case ; • We verify our theorems on different computer vision tasks ( including one of most challenging datasets ImageNet ( Russakovsky et al. , 2015 ) and MSCOCO ( Lin et al. , 2014 ) ) with various networks structures and normalization techniques . Experiments show the theoretical value of angular update and weight norm agree well with empirical observation . Recently , a parallel work ( Li et al. , 2020b ) is published to analyze the equilibrium condition of normalized neural network with weight decay in SGD case . They model the learning dynamic of normalized network as a SDE , and propose the concept “ intrinsic learning rate ” ( λe = λη ) to measure the converge rate of weight norm and effective learning rate . Their main result , in which converging time to equilibrium in SGD case is O ( 1/ ( λη ) ) , is consistent with our theory . The major difference between Li et al . ( 2020b ) and our work is that all results from Li et al . ( 2020b ) are qualitatively results or conjectures limited on SGD case . While our work derives the quantitative results on both SGD/SGDM cases , our analysis allows us to precisely predict the value of weight norm/angular update in large scale data experiments , and clarify the difference of approaching rate between SGD and SGDM case . Besides , Li et al . ( 2020b ) conducts empirical study to discuss the connection between the equilibrium state and accuracy of trained model . Our work mainly focuses on the equilibrium condition itself , connection between equilibrium condition and performance of trained model is beyond our discussion in this paper . The experiments we present in main text are only for verification of our theory . Our theory on equilibrium condition in Spherical Motion Dynamics implies equilibrium condition mostly relies on the update rules of SGD/SGDM with WD , and scale-invariant property . The cause of equilibrium condition is independent of decrease of loss or trajectory of optimization , but equilibrium condition turns out to significantly affect update efficiency of normalized network by controlling the relative update ( Eq. ( 4 ) ) . We believe equilibrium condition is one of the key reason why learning dynamic of normalized neural network is not consistent with traditional optimization theory ( Li et al. , 2020b ) . We think it has great potential to study the leaning dynamic of normalized network or develop novel efficient learning strategy under the view of Spherical Motion Dynamics . 2 RELATED WORK . Normalization techniques Batch normalization ( BN ( Ioffe & Szegedy , 2015 ) ) is proposed to deal with gradient vanishing/explosion , and accelerate the training of DNN . Rapidly , BN has been widely used in almost all kinds of deep learning tasks . Aside from BN , more types of normalization techniques have been proposed to remedy the defects of BN ( Ioffe , 2017 ; Wu & He , 2018 ; Chiley et al. , 2019 ; Yan et al. , 2020 ) or to achieve better performance ( Ba et al. , 2016 ; Ulyanov et al. , 2016 ; Salimans & Kingma , 2016 ; Shao et al. , 2019 ; Singh & Krishnan , 2020 ) . Though extremely effective , the mechanism of BN still remains as a mystery . Existing works attempt to analyze the function of BN : Ioffe & Szegedy ( 2015 ) claims BN can reduce the Internal Covariance Shift ( ICS ) of DNN ; Santurkar et al . ( 2018 ) argue that the effectiveness of BN is not related to ICS , but the smoothness of normalized network ; Luo et al . ( 2019 ) shows BN can be viewed as an implicit regularization technique ; Cai et al . ( 2019 ) proves that with BN orthogonal least square problem can converge at linear rate ; Dukler et al . ( 2020 ) proves weight normalization can speed up training in a two-layer ReLU network . Weight decay Weight decay ( WD ) is well-known as l2 regularization , or ridge regression , in statistics . WD is also found to be extreme effective when applied in deep learning tasks . Krizhevsky & Geoffrey ( 2009 ) shows WD sometimes can even improve training accuracy not just generalization performance ; Zhang et al . ( 2019 ) show WD can regularize the input-output Jacobian norm and reduce the effective damping coefficient ; Li et al . ( 2020a ) discusses the disharmony between WD and weight normalization . A more recent work Lewkowycz & Gur-Ari ( 2020 ) empirically finds the number of SGD steps T until a model achieves maximum performance satisfies T ∝ 1λη , where λ , η are weight decay factor and learning rate respectively , they interpret this phenomenon under the view of Neural Tangent Kernel ( Jacot et al. , 2018 ) , showing that weight decay can accelerate the training process . Notice their result has no connection with equilibrium condition discussed in this work . Our results shows the cause of equilibrium condition can be reached long before neural network can get its highest performance . Effective learning rate Due to the scale invariant property caused by normalization , researchers start to study the behavior of effective learning rate . van Laarhoven ( 2017 ) ; Chiley et al . ( 2019 ) estimate the magnitude of effective learning rate under equilibrium assumptions in SGD case ; Hoffer et al . ( 2018 ) quantify effective learning rate without equilibrium assumptions , so their results are much weaker ; Arora et al . ( 2019 ) proves that without WD , normalized DNN can still converge with fixed/decaying learning rate in GD/SGD cases respectively ; Zhang et al . ( 2019 ) shows WD can increase effective learning rate ; Li & Arora ( 2020 ) proves standard multi-stage learning rate schedule with BN and WD is equivalent to an exponential increasing learning rate schedule without WD . As a proposition , Li & Arora ( 2020 ) quantifies the magnitude of effective learning rate in SGDM case . But none of them have ever discussed why equilibrium condition can be reached . A recent work Li et al . ( 2020b ) studies the convergence of effective learning rate by SDE , proving that the convergence time is of O ( 1/ ( λη ) ) , where λ , η are weight decay factor and learning rate respectively . Their result can only provide intuitive understanding , and is limited on SGD case .

Summary: The main goal of the paper is to establish theoretically some previous known results that for scale invariant networks the weight norm has a fixed point with ||w||^4=eta/lambda ||\tilde{g}|| . They also discuss the angular update, which because of scale invariance is basically equivalent to arccos (1-eta lambda) |w_t|^2/|w_t+1|^2 and it thus comes mainly from the gradient. They have some experiments which they compare with the predicted equilibrium values for the angular update/ weight norm.