Training By Vanilla SGD with Larger Learning Rates

1 INTRODUCTION . We are interested in minimizing a function f : Rd → R with an expectation form : min x∈Rd f ( x ) : = Eξ [ K ( x , ξ ) ] , ( 1a ) where the subscript indicates expectation is computed on the random variable ξ . Specially , if the ξ probability distribution is clear and random variable ξ is uniformly distributed on N terms , the objective function f can be expressed into a finite-sum form : min x∈Rd f ( x ) : = 1 N N∑ i=1 fi ( x ) , ( 1b ) where fi ( x ) : Rd → R is the i-th component function . The optimization of Eqs . ( 1a ) and ( 1b ) is widely encountered in machine learning tasks ( Goodfellow et al. , 2016 ; Simonyan & Zisserman , 2014 ) . To solve this problem given in Eq . ( 1b ) , we can compute the gradient of objective function directly with a classic GD ( Gradient Descent ) algorithm . However , this method suffers from expensive gradient computation for extremely large N , and hence people apply stochastic gradient descent method ( SGM ) to address this issue . Incremental gradient descent ( IGD ) is a primary version of SGM , where its calculation of gradient proceeds on single component fi at each iteration , instead of the whole . As a special case of IGD , SGD ( Robbins & Monro , 1951 ) , a fundamental method to train neural networks , always updates parameters by the gradient computed on a minibatch . There exists an intuitive view that SGD with constant step size ( SGD-CS ) potentially leads to faster convergence rate . Some studies ( Solodov , 1998 ; Tseng , 1998 ) show that , under a so-called strong growth condition ( SGC ) , SGD-CS converges to an optimal point faster than SGD with a diminishing step size . In this work , we study the local convergence ( or last-iterate convergence as is called in Jain et al . ( 2019 ) ) of SGD-CS on nonconvex functions . Note that SGD-CS does not provide a guarantee of converge when starting from any initialization point . Therefore , a useful strategy is to use SGD with a decreasing step size at the beginning , and then switch to SGD-CS in a neighborhood of a minimizer . Fig.1 illustrates the range of the step size of SGD versus the number of iterations . Our main theoretical and experimental contributions are as follows . • We establish local ( or last-iterate ) convergence of SGD with a constant step size ( SGDCS ) on nonconvex functions under the interpolation condition . We note that previous results are mostly for strongly convex functions under strong ( or weak ) growth condition . Our result is much closer to common situations in practice . • We discover that on linear regression problems with ` 2 regularization , the size of convergent learning rates can be quite large for incremental gradient descent ( IGD ) . Our numerical results show that , within a fairly large range , the larger step size is , the smaller spectral radius is , and the faster the convergence rate is . • Based on the above observations , we further propose a strategy called SGDL that uses the SGD with a large initial learning rate ( more than 10 times larger than the learning rate in SGD with momentum ) , while still being a vanilla SGD . We conduct extensive experiments on various popular deep-learning tasks and models in computer vision , audio recognition and natural language processing . Our results show that the method converges successfully and has a strong generalization performance , and sometimes outperforms its advanced variant ( SGDM ) and other several popular adaptive methods ( e.g. , Adam , AdaDelta , etc ) . 2 RELATED WORK . There are many papers on stochastic optimization and we summarize typical ones which are most relevant with our work . The convergence of SGD for over-parameterized models is analyzed in ( Vaswani et al. , 2018 ; Mai & Johansson , 2020 ; Allen-Zhu et al. , 2019 ; Li & Liang , 2018 ) . The power of interpolation is studied in ( Ma et al. , 2018 ; Vaswani et al. , 2019 ) . The work ( Jastrzębski et al. , 2017 ) investigates a large ratio of learning rate to batch size often leads to a wide endpoint . The work ( Smith & Topin , 2019 ) shows a phenomenon called super-convergence which is in contrast to the results in ( Bottou et al. , 2018 ) . More recently , several new learning rate schedules have proposed for SGD ( Loshchilov & Hutter , 2016 ; Smith , 2017 ; Agarwal et al. , 2017 ; Carmon et al. , 2018 ) . Adaptive gradient methods are widely in deep learning application . Popular solutions include AdaDelta ( Zeiler , 2012 ) , RMSProp ( Hinton et al. , 2012 ) , Adam ( Kingma & Ba , 2014 ) and so on . Unfortunately , it is believed that the adaptive methods may have a poor empirically performance . For example , Wilson et al . ( 2017 ) have observed that Adam hurts generalization performance in comparison to SGD with or without momentum . The work ( Schmidt & Roux , 2013 ) introduces SGD-CS that attains linear convergence rate for strongly convex functions under the strong growth condition ( SGC ) , and sublinear convergence rate for convex functions under the SGC . The work ( Cevher & Vu , 2018 ) investigates the weak growth condition ( WGC ) , the necessary condition for linear convergence of SGD-CS . For a general convex function , the work ( Ma et al. , 2018 ) shows that SGD-CS attains linear convergence rate under the interpolation property . For a finite sum f , the interpolation property means ∇fi ( x∗ ) = 0 for i = 1 , . . . , N , which is held apparently in over-parameterized DNN . However , the theoretical analysis of SGD-CS for nonconvex functions is less as mature as in the convex case . 3 PRELIMINARIES . 3.1 NOTATIONS . We denote the minimizer of object function f as x∗ and use usual partial ordering for symmetric matrices : A B means A−B is positive semidefinite ; similar for the relations , , ≺ . The norm ‖ · ‖ either denotes the Euclidean norm for vectors or Frobenius norm for matrices . The Hessian matrix of each component function fi ( x ) is denoted as Hi ( x ) . With slight abuse of notation , the Hessian matrix Hi at x∗ is denoted as H∗i . We also denote the mean and variance of { H∗1 , . . . , H∗N } as H∗ ≡ 1N ∑N i=1H ∗ i and Σ ∗ ≡ 1N ∑N i=1 ( H ∗ i ) 2 − ( H∗ ) 2 , and define a ∧ b = min ( a , b ) . X P−→ Y means random variable X converges to random variable Y in probability , while X L 1 −−→ Y means X converges to random variable Y in L1 norm . 3.2 ASSUMPTIONS . Our results are under the following mild assumptions : ( A.1 ) The objective function f has the same structure as given in Eq. ( 1b ) . ( A.2 ) Each component function fi in Eq . ( 1b ) is twice continously differentiable , with the corresponding Hessian matrix Hi being Lipschitz continuous for constant Li at the minimizer x∗ . Without loss of generality , we suppose that ‖Hi ( x ) −Hi ( x∗ ) ‖ ≤ L‖x− x∗‖ , for i = 1 , . . . , N , where L = max i=1 , ... , N Li . ( A.3 ) As in ( Zhang et al. , 2000 , Remark 5.2 ) , we assume the point of interest x∗ is a strong minimizer ( i.e . H ( x∗ ) 0 ) . Assumption ( A.3 ) restricts that the objective function is essentially strongly convex in a sufficiently small neighborhood of a minimizer . The work ( Liu et al. , 2020 ) investigates that the loss function is typically nonconvex in any neighborhood of a minimizer . However , this assumption is not restrictive in the regularization setting . Remark 1 . Even if the Hessian matrix of the point of interest x∗ has zero eigenvalues , we can study the ` 2 regularized problem instead , and then the Hessian matrix becomes positive definite . This phenomenon is also validated in our numerical experiment in Section 5 , and we add ` 2 regularizer in DNNs in Section 6 . Based on the above technical assumptions , we focus on the convergence analysis of SGD-CS by discussing the convergence behavior precisely : xk+1 = xk − α 1 m m∑ j=1 ∇f ξ ( j ) k ( xk ) . ( 2 ) Next , we introduce the notion of point attraction to help us understand the convergence behavior to Eq. ( 2 ) . Definition 1 . We say x∗ is a point of attraction to Eq . ( 2 ) if there is an open ball N ( x∗ , ) of x∗ , such that for any x0 ∈ N , { xk } generated by Eq . ( 2 ) all lie inN and converges to x∗ in the ` 1 norm . The Ostrowski Theorem says a sufficient condition for x∗ to be a point of attraction of the deterministic iteration xk+1 = T ( xk ) is that the spectral radius of T ′ ( x ) is strictly less than one , which further implies the condition that there exists an open ball N such that once the initial x0 ∈ N , the iterates { xk } will converge to x∗ with linear rate . We define the strong minimizer x∗ as a point of strong attraction following the similar manner . Definition 2 . We say x∗ is a point of strong attraction to Eq . ( 2 ) if there exists a neighborhoodN of x∗ , such that for any x0 ∈ N , the sequence { xk } generated by Eq . ( 2 ) all lie in N and satisfy E‖xk − x∗‖2 ≤ ρk · ‖x0 − x∗‖2 , for some ρ ∈ [ 0 , 1 ) . The following discussion studies necessary and sufficient conditions for the minimizer x∗ being a point of attraction of SGD with the constant step size α . However , starting from any initialization point , SGD-CS can not be guaranteed to find such a local neighborhood around x∗ , except in some special cases including the loss function studied in Section 5 and Appendix A.2 . Therefore , a practical strategy is to use diminishing step size when starting from a random initial point ( Allen-Zhu , 2018 ) , and as long as the iteration points belong to a neighborhood of a point of attraction , one can use a constant step size . 4 RESULTS FOR NONCONVEX FUNCTIONS . In this section , we rigorously show the necessary condition for the minimizer x∗ being a point of attraction , and Theorem 1 provides a sufficient condition for the strong minimizer x∗ to be a point of strong attraction with high probability . The detailed proofs are in Appendix.A.1 . 4.1 NECESSARY CONDITION . Lemma 1 . Suppose that the assumptions ( A.1 ) and ( A.2 ) hold , then ‖∇fi ( x ) ‖ < L 2 + ‖H∗i ‖ for i = 1 , . . . , N , provided that x ∈ N ( x∗ , ) . Theorem 1 . Suppose that the assumptions ( A.1 ) and ( A.2 ) hold . If the minimizer x∗ is a point of attraction of Eq . ( 2 ) , then the interpolation property is satisfied , i.e. , ∇fi ( x∗ ) = 0 for i = 1 , . . . , N .

Summary: This paper presents a theoretical analysis of SGD with constant step size (SGD-CS) and presents conditions under which SGD-CS leads to parameter updates that converge to a local minima, including parameters of non-convex functions.  The authors then show, in context of some special functions, that the step size can be fairly large yet convergence is achieved.  This is followed by a number of empirical studies of SGD with large (but annealed) step-size (SGDL) on a variety of tasks.