Momentum Doesn't Change The Implicit Bias

1 INTRODUCTION . It is widely believed that the optimizers have implicit bias in terms of selecting output parameters among all the local minima on the landscape ( Neyshabur et al. , 2015 ; Keskar et al. , 2017 ; Wilson et al. , 2017 ) . It is shown in the analysis of Adaboost that the coordinate descent would converge to the L1 max-margin solution for the linear classification task with exponetial-tailed loss ( ( Schapire & Freund , 2013 ; Telgarsky , 2013 ) ) . Latter , Soudry et al . ( 2018 ) shows that gradient descent would converge to the L2 max-margin solution under the same setting , which mirrors its good generalization property in practice . Since then , many efforts have been taken on analyzing the implicit bias of various local-search optimizers , including stochastic gradient descent ( Nacson et al. , 2019 ) , steepest descent ( Gunasekar et al. , 2018a ) , AdaGrad ( Qian & Qian , 2019 ) and optimizers for homogeneous neural networks ( Lyu & Li , 2019 ; Ji & Telgarsky , 2020 ; Wang et al. , 2021 ) . However , though the momentum acceleration technique is widely adopted in the optimization algorithms in both convex and non-convex learning tasks ( Sutskever et al. , 2013 ; Vaswani et al. , 2017 ; Tan & Le , 2019 ) , the understanding on how the momentum would affect generalization performance of the optimization algorithms is still unclear . A natural question is : Can we theoretically analyze the implicit bias of momentum-based optimizers ? In this paper , we take the first step to analyze the convergence of momentum based optimizers and unveil their implicit bias . Specifically , we study the classification problem with linear model and exponential-tailed loss using Stochastic Gradient Descent with Momentum ( SGDM ) and Adam optimizers . We consider the optimizers with constant learning rate and constant momentum hyperparameters , which are widely adopted in practice , e.g. , the default setting in popular machine learning frameworks ( Paszke et al. , 2019 ) and in experiments ( Xie et al. , 2017 ) . We note that Gradient Descent with Momentum ( GDM ) can be viewed as a special case of SGDM , and naturally share the properties for SGDM . Our main results are summarized in Theorem 1 . Theorem 1 ( informal ) . With linear separable dataset S , for SGDM and Adam ( without stochasticity , abbreviated as w/s latter ) , the loss converges to 0 with rate O ( 1t ) where t is the number of iterations , the parameter norm diverges to infinity , and direction of parameters converges to the direction of the L2 max-margin solution . Theorem 1 states SGDM converges to the L2 max-margin solution , which is the same as SGD , indicating that momentum does not affect the convergent direction . The good generalization behavior of the output parameters of SGDM is well validated as the margin of a classifier is positively correlated with its generalization error ( Jiang et al. , 2019 ) . This is supported by existing experimental observations ( c.f . Figure 1 , ( Soudry et al. , 2018 ) and Figure 2 , ( Nacson et al. , 2019 ) ) . Similar claims hold for Adam ( w/s ) , which is also well supported by empirical results in Wang et al . ( 2021 ) . Our contributions are significant in terms of the following aspects : • We establish the implicit bias of the momentum based optimizers , an open problem since the initial work Soudry et al . ( 2018 ) . The momentum based optimizers are widely used in practice and our theoretical characterization deepens the understanding on their generalization property , which is important by its own . • Technically , we propose a new Lyapunov function to analyze the convergence of SGDM , which helps to bound the sum of squared gradients along the training trajectory . Compared to the usual one , the new Lyapunov function depends on a middle variable of an alternative update rule of SGDM , which helps to capture the historical dependence in the momentum update . To our knowledge , such a technique has not been exploited ever before and can be of independent interest for convergence analysis of momentum-based optimizers . We then construct a new Lyapunov function to bound the difference of learned parameters and the scaled max-margin solution , which finally leads to the direction convergence . This Lyapunov function provides a direct way to establish the convergence to the desired direction . Organization of This Paper . Section 2 collects further related works on the implicit bias of first order optimizers and convergence of momentum-based optimizers . Section 3 shows basic settings and assumptions which will be used throughout this paper . Section 4 studies the implicit bias of GDM as a warm up , while Section 5 and Section 6 explore respectively the implicit bias of SGDM and Adam ( w/s ) . Discussions of these results are put in Section 7 . 2 FURTHER RELATED WORKS . Implicit Bias of First-order Optimization Methods . Soudry et al . ( 2018 ) prove that gradient descent on linear classification problem with exponential-tailed loss converges to the direction of the max L2 margin solution of the corresponding hard-margin Support Vector Machine . Nacson et al . ( 2019 ) extend the results in ( Soudry et al. , 2018 ) to the stochastic case , proving that the convergent direction of SGD is the same as GD almost surely . Qian & Qian ( 2019 ) go beyond the vanilla gradient descent methods and consider the AdaGrad optimizer instead . They prove that the convergent direction of AdaGrad has a dependency on the optimizing trajectory , which varies according to the initialization . Ji & Telgarsky ( 2021 ) propose a primal-dual analysis framework for the linear classification models , and prove a faster convergent rate of the margin by increasing the learning rate according to the loss . Based on ( Ji & Telgarsky , 2021 ) , ( Ji et al. , 2021 ) design another algorithm with an even faster convergent rate of margin by applying the Nesterov ’ s Acceleration Method on the dual space . However , the corresponding form of the algorithm on the primal space is no longer a Nesterov ’ s Acceleration Method nor GDM , which is significantly different from our settings . On the other hand , there is another line of work trying to extend the result in the linear case to deep neural networks . Ji & Telgarsky ( 2018 ) ; Gunasekar et al . ( 2018b ) study the deep linear network and Soudry et al . ( 2018 ) study the two-layer neural network with ReLU activation . Lyu & Li ( 2019 ) propose a framework to analyze the asymptotic direction of GD on homogeneous neural networks , proving that given there exists a time the network achieves 100 % training accuracy , GD will converge to some KKT point of the L2 max-margin problem . Wang et al . ( 2021 ) extend the framework of Lyu & Li ( 2019 ) to adaptive optimizers , and prove RMSProp and Adam without momentum have the same convergent direction as GD , while AdaGrad doesn ’ t . The results ( Lyu & Li , 2019 ; Wang et al. , 2021 ) indicate that results in the linear model can be extended to deep homogeneous neural networks , and suggest that the linear model is a proper start point to study the implicit bias . There are also works on the implicit bias of regression problems with bounded optimal points and interesting readers can refer to ( Rosasco & Villa , 2015 ; Lin & Rosasco , 2017 ; Ali et al. , 2020 ) etc . for details . Convergence of Momentum-Based Optimization Methods . For convex optimization problems , the convergence rate of Nesterov ’ s Acceleration Method has been proved in Nesterov ( 1983 ) . In contrast , although GDM ( Polyak ’ s Heavy-Ball Method ) was proposed in ( Polyak , 1964 ) prior to the Nesterov ’ s Acceleration Method , the convergence of GDM on convex loss with Lipschitz gradient was not solved until Ghadimi et al . ( 2015 ) provides an ergodic convergent result for GDM , i.e. , the convergent result for the running average of the iterates . However , the ergodic result is undesired under many learning scenarios , e.g. , in classification tasks , the optimization algorithms usually output the parameters of the last step . To the best of our knowledge , there is only two works on the non-ergodic analysis of ( S ) GDM : Sun et al . ( 2019 ) and Tao et al . ( 2021 ) . Sun et al . ( 2019 ) prove that if the training loss is coercive ( the training loss goes to infinity whenever parameter norm goes to infinity ) , convex , and globally smooth , then , with constant momentum hyper-parameter , the training loss converges to minima with rateO ( t−1 ) . Tao et al . ( 2021 ) analyze a case when momentum coefficient increases to 1 and the gradient is bounded all over the parameter space , showing SGDM can improve the convergence rate of SGD by a factor log ( t ) . There are also works on the gradient norm convergence of SGDM under various settings ( Yan et al. , 2018 ; Yu et al. , 2019 ; Liu et al. , 2020 ) and on the investigation of momentum-based method from the view point of dynamics ( Sarao Mannelli & Urbani , 2021 ) . However , there is no existing work on the implicit bias of momentum-based optimizers for the classification problem , which is first analyzed by this paper . 3 PRELIMINARIES . In this paper , we focus on the linear model with exponential-tailed loss . We first derive the results for binary classification , then we show that the methodology can be easily extended to the multi-class classification problem . Problem setting . The dataset used for training is defined as S = ( xi , yi ) Ni=1 , where xi ∈ Rd is the i-th feature , and yi ∈ R is the i-th label ( i = 1 , 2 , · · · , N ) . We will use the linear model to fit the label : for any feature x ∈ Rd and parameter w ∈ Rd , the prediction is given by 〈w , x〉 . For binary classification , given any data zi = ( xi , yi ) ∈ S , the individual loss for parameter w is given as ` ( yi〈w , xi〉 ) . A common setting is to ensemble the feature and label together as x̃i = yixi ( there is a mapping T : ( x , y ) → yx , and x̃i is T ( ( xi , yi ) ) ) . The individual loss can be rewritten as ˜̀ ( w , ( xi , yi ) ) = ` ( yi〈w , xi〉 ) = ` ( 〈w , x̃i〉 ) . The optimization target is defined as the averaged loss : L ( w ) = ∑N i=1 ˜̀ ( w , ( xi , yi ) ) N . Without loss of generality , we consider the case with normalized data1 , that is , ‖x̃i‖ ≤ 1 , ∀i ∈ [ N ] . Optimizer . Here we will introduce the update rules of SGDM and Adam ( w/s ) . SGDM can be viewed as a stochastic version of GDM by randomly choosing a subset of the dataset to update . Specifically , the update rule of SGDM is given as ( SGDM ) : w ( t+ 1 ) −w ( t ) = −η∇LB ( t ) ( w ( t ) ) + β ( w ( t ) −w ( t− 1 ) ) , ∀t ≥ 1 , ( 1 ) where B ( t ) is a subset of S with size b which is sampled independently and uniformly with replacement , and LB ( t ) is defined as LB ( t ) ( w ) = ∑ z∈B ( t ) ˜̀ ( w , z ) b . We also define Ft as the sub-sigma field such that { w ( t ) } ∞t=1 is adapted with respect to Filtration { Ft } ∞t=1 . 1The proof can be naturally applied to unnormalized data by letting ` ( x ) = ` ( maxx̃∈T ( S ) ‖x̃‖ · x ) and x̃i = x̃i maxx̃∈T ( S ) ‖x̃‖ . The Adam ( w/s ) can be viewed as a variant of GDM in which the preconditioner is adopted , whose form is characterized as follows : m ( 0 ) = 0 , m ( t ) = β1m ( t− 1 ) + ( 1− β1 ) ∇L ( w ( t ) ) , m̂ ( t ) = 1 1− βt1 m ( t ) , ∀t ≥ 1 , ν ( 0 ) = 0 , ν ( t ) = β2ν ( t− 1 ) + ( 1− β2 ) ( ∇L ( w ( t ) ) ) 2 , ν̂ ( t ) = 1 1− βt2 ν ( t ) , ∀t ≥ 1 , ( Adam ( w/s ) ) : w ( t ) = w ( t− 1 ) − η m̂ ( t− 1 ) √ ν̂ ( t− 1 ) + ε1d , ∀t ≥ 1 , ( 2 ) where 1√ ν̂ ( t−1 ) +ε1d ) is called the preconditioner at step t. Assumptions : The analysis of this paper are based on three common assumptions in existing literature ( first proposed by ( Soudry et al. , 2018 ) ) , respectively on the separability of the dataset , the individual loss behaviour at the tail , and the smoothness of the individual loss . We list them as follows : Assumption 1 ( Linearly Separable Dataset ) . There exists one parameter w ∈ Rd , such that 〈w , x̃i〉 > 0 , ∀i ∈ [ N ] . Assumption 2 ( Exponential-tailed Loss ) . The individual loss ` is exponential-tailed , i.e. , • Differentiable and monotonically decreasing to zero , with its derivative also converging to zero , i.e. , limx→∞ ` ( x ) = limx→∞ ` ′ ( x ) = 0 , and ` ′ ( x ) < 0 ∀x ; • Close to exponential loss when x is large enough , i.e. , there exist positive constants c , a , K , µ+ , µ− , x+ , x− and x0 , such that , ∀x > x+ : − ` ′ ( x ) ≤ c ( 1 + e−µ+x ) e−ax , ( 3 ) ∀x > x− : − ` ′ ( x ) ≥ c ( 1− e−µ−x ) e−ax . ( 4 ) Assumption 3 ( Smooth Loss ) . Either of the following assumptions holds regarding the case : ( D ) : ( Deterministic Case ) The individual loss ` is locally smooth , i.e. , for any s0 ∈ R , there exists a positive real Hs0 , such that ∀x , y ≥ s0 , | ` ′ ( x ) − ` ′ ( y ) | ≤ Hs0 |x− y| . ( S ) : ( Stochastic Case ) The individual loss ` is globally smooth , i.e. , there exists a positive real H , such that ∀x , y ∈ R , | ` ′ ( x ) − ` ′ ( y ) | ≤ H|x− y| . We provide explanations of these three assumptions respectively . Based on Assumption 1 , we can formally define the margin and the maximum margin solution of an optimization problem , which is introduced in Definition 1 Definition 1 . Let the margin γ ( w ) of parameter w defined as the lowest score of the prediction of w over the dataset S , i.e. , γ ( w ) = minx̃i∈T ( S ) 〈w , x̃i〉 . By Assumption 1 and the positive homogeneous of γ , γ ( ŵγ ( ŵ ) ) = 1 , and thus we define the maximum margin solution ŵ and the L 2 max margin γ of the dataset S as follows : ŵ 4 = arg min γ ( w ) ≥1 ‖w‖2 , γ 4= 1 ‖ŵ‖ Since ‖ · ‖2 is strongly convex and set { w : γ ( w ) ≥ 1 } is convex , ŵ is uniquely defined . Assumption 2 constraints the loss to be exponential-tailed , which is satisfied by many popular choices of ` , including the ( ` exp ( x ) = e−x ) and the logistic loss ( ` log ( x ) = log ( 1 + e−x ) ) . Also , as c and a can be respectively absorbed by resetting the learning rate and data as η = cη and xi = axi , without loss of generality , in this paper we only analyze the case that c = a = 1 . The globally smooth assumption ( Assumption 3 . ( S ) ) is strictly stronger than the locally smooth assumption ( Assumption 3 . ( D ) ) . One can easily verify that both the exponential loss and the logistic loss meet Assumption 3 . ( D ) , and the logistic loss also meets Assumption 3 . ( S ) .

Summary: The authors study the convergence of stochastic gradient descent momentum (SGDM) in a problem with of binary classification with separable data.  Interestingly, the authors observed that SGDM and GD converge to the same solution (i.e. the max margin solution) with the same convergence rate O(1/t).  The paper proposed a variation of [Soudry et al. 2018] solution, that allows for the analysis of the more complex SGDM.