On the Convergence of Step Decay Step-Size for Stochastic Optimization

√ T ) rate . We also provide near-optimal ( and sometimes provably tight ) convergence guarantees for general , possibly non-smooth , convex and strongly convex problems . The practical efficiency of the step decay step-size is demonstrated in several large-scale deep neural network training tasks . 1 Introduction . We focus on stochastic programming problems on the form min x∈X f ( x ) : = Eξ∼Ξ [ f ( x ; ξ ) ] . ( 1 ) Here , ξ is a random variable drawn from some source distribution Ξ over an arbitrary probability space and X is a closed , convex subset of Rd . This problem is often encountered in machine learning applications , such as training of deep neural networks . Depending on the specifics of the application , the function f can either be non-convex , convex or strongly convex ; it can be smooth or non-smooth ; and it may also have additional structure that can be exploited . Despite the many advances in the field of stochastic programming , the stochastic gradient descent ( SGD ) method [ 37 , 32 ] remains important and is arguably still the most popular method for solving ( 1 ) . The SGD method updates the decision vector x using the following recursion xt+1 = ΠX ( xt − ηtĝt ) ( 2 ) where ĝt is an unbiased estimate of the gradient ( or subgradient ) at xt and ΠX is the Euclidean projection onto X . The step-size parameter ( learning-rate ) ηt > 0 is critical to control the rate at which the model learns and to guarantee that the SGD iterates converge to an optimizer of ( 1 ) . Setting the step-size too large will result in iterates which never converge ; and setting it too small leads to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) slow convergence and may even cause the iterates to get stuck at bad local minima . As long as the iterates do not diverge , a large constant step-size promotes fast convergence but only to a ( large ) neighborhood of the optimal solution . To increase the accuracy , we have to decrease the step-size . The traditional approach is to decrease the step-size in every iteration , typically as η0/t or η0/ √ t. Both these step-size schedules have been studied extensively and guarantee a non-asymptotic convergence of SGD [ 31 , 26 , 36 , 19 , 38 , 15 ] . However , from a practical perspective , these step-size policies often perform poorly , since they begin to decrease too early . For non-convex problems , such as those which arise in training of deep neural networks , the most successful and popular step-size policy in practice is the step decay step-size [ 25 , 21 , 22 ] . This step-size policy starts with a relatively large constant step-size and then cuts the step-size by a fixed number ( called decay factor ) at after a given number of epochs . Not only does this step-size result in a faster initial convergence , but it also guarantees that the SGD iterates eventually converge to an exact solution . The step decay step-size is the default choice in many deep learning libraries , such as TensorFlow [ 4 ] and PyTorch [ 35 ] ; both use a decay rate of 0.1 and user-defined milestones when the step-size is decreased . Although some recent studies attempt to monitor the optimization process to trigger the milestones when certain conditions are met [ 27 , 50 ] , such approaches are difficult to analyze . Another common approach is to simply divide the targeted number of iterations into N intervals of equal length , and trigger a milestone at the end of each such interval . However , the theoretical properties of this simpler step-size policy is also not yet well understood . To make the problem more manageable theoretically , some related step-size policies that let the length of each segment increase linearly [ 6 ] or exponentially [ 19 , 8 , 49 ] are analyzed . On the contrary , we consider the more practical case when the horizon T of the step-decay step-size is divided into several equal parts . If the number of iterations or horizon T is known apriori , reference [ 11 ] analyzes a step decay step-size with a decay rate of 1/2 applied every T/ log2 T iterations and establishes a near-optimal O ( log2 T/T ) convergence rate for vanilla SGD on least-squares problems . However , to our knowledge , there are no convergence guarantees in the literature under more general conditions , e.g. , for general strongly-convex , general convex , or non-convex problems . Motivated by this , in this paper , we study the non-asymptotic convergence of SGD with the step decay step-size in more general settings . 1.1 Main Contributions . This work establishes novel convergence guarantees for SGD with the step decay step-size on nonconvex , convex and strongly convex optimization problems . We make the following contributions : • We propose a non-uniform probability rule Pt ∝ 1/ηt for selecting the output in the smooth non-convex setting . Based on this rule , we ( i ) establish a near-optimal O ( lnT/ √ T ) rate for SGD with step decay step-size ; ( ii ) improve the results for exponential decay step-size [ 28 ] ; ( iii ) remove the lnT factor in the best known convergence rate for the 1/ √ t step-size . • For strongly convex problems , we establish the following error bounds for the last iterate under step-decay : ( i ) O ( lnT/T ) for smooth problem , which we also prove to be tight ; ( ii ) O ( ln2 T/T ) without the smoothness assumption . • For the general convex case , we prove that the step decay step-size at the last iterate can achieve a near-optimal convergence rate ( up to a lnT factor ) . 1.2 Related Work . For SGD , the best known bound for the expected error of the T th iterate is of O ( 1/ √ T ) when the objective is convex and smooth [ 32 , 12 ] , and of O ( 1/T ) when the objective is also strongly convex [ 31 , 36 ] . Without any further assumptions , these rates are known to be optimal . If we restrict our attention to diminishing step-sizes , ηt = η0/t , the best known error bound for strongly convex and non-smooth problems is of O ( lnT/T ) [ 38 ] , which is also tight [ 17 ] . This rate can be improved to O ( 1/T ) by averaging strategies [ 36 , 26 , 38 ] or a step decay step-size [ 19 ] . For smooth non-convex functions , Reference [ 12 ] established anO ( 1/ √ T ) rate for SGD with a constant step-size ηt = O ( 1/ √ T ) . Recently , this error bound has proven to be tight up to a constant , unless additional assumptions are made [ 9 ] . Step decay step-sizes were probably first considered for deterministic subgradient methods [ 14 ] and [ 40 ] . More recently step decay step-size schemes have been employed to improve the convergence rates under various conditions : local growth ( convex ) [ 45 ] , Polyak-Lójasiewicz ( PL ) condition [ 49 ] , sharp growth ( non-convex ) [ 8 ] . Most of these references consider proximal algorithms ( which add a quadratic term λ ‖·‖2 to the original loss function ) and , in addition , perform an extra averaging step between stages . In particular , if the function f is µ-weakly convex ( i.e . f + µ2 ‖·‖ 2 is convex ) , λ is required to be larger than µ2 [ 49 , 6 ] . In contrast , we study the performance of step decay step-size for the standard SGD algorithm . Moreover , in [ 45 ] the output of each inner-loop is an uniform average . So their complexity reflects the average performance over iterations . But for convex and strongly convex problems , we focus on the final iterate which is the most preferred choice in practice . In fact , most practitioners just use the final iterate . Reference [ 6 ] considers slowly decaying step-sizes , ∼ 1/t where t is the stage index . In other words , the step-size is not cut by a constant factor between stages as we do , but it decays ( very ) slowly . Compared to [ 19 ] and [ 49 ] , where the inner-loop size S grows exponentially , we use a constant value of S , which is known to work better in practice . In the extreme case when S = 1 , step-decay reduces to the exponentially decaying step-size recently studied under the PL condition and a general smoothness assumption [ 28 ] . A number of adaptive step-size selection strategies have been proposed for SGD ( e.g. , [ 10 , 42 , 23 , 30 , 29 ] ) , some of which result in step-decay policies [ 44 , 27 , 50 ] . For example , Reference [ 27 ] develops a statistical procedure to automatically determine when the SGD iterates with a constant step-size no longer make progress , and then halve the step-size . Empirically , this automatic scheme is competitive with the best hand-tuned step-decay schedules , but no formal guarantees for this observed behaviour are given . The paper is organized as follows . Notation and basic definitions are introduced in Section 2 . In Section 3 , we analyze the convergence rates of step decay step-size on nonconvex case and propose a novel non-uniform sampling rule for the algorithm output . The convergence for general convex and strongly convex functions are investigated in Sections A and 4 , respectively . Numerical results of our algorithms are presented and discussed in Section 5 . Finally , conclusions are made in Section 6 . 2 Preliminaries . In this part , we will give some definitions and notations used throughout the paper . Definition 1 . ( 1 ) The stochastic gradient oracle Õ is variance-bounded if for any input vector x̂ , it returns a random vector ĝ such that E [ ‖ĝ − E [ ĝ ] ‖2 ] ≤ V 2 . ( 2 ) Õ is bounded if for any input vector x̂ , it returns a random vector ĝ such that E [ ‖ĝ‖2 ] ≤ G2 for some fixed G > 0 . Definition 2 . When the function f is differentiable on X , we say that f is L-smooth on X if there exists a constant L > 0 such that ‖∇f ( x ) −∇f ( y ) ‖ ≤ L ‖x− y‖ . It also implies that f ( y ) ≤ f ( x ) + 〈∇f ( x ) , y − x〉+ L2 ‖x− y‖ 2 for any x , y ∈ X . If f is not differentiable on X , we say that f is L-smooth with respect to x∗ if f ( x ) − f ( x∗ ) ≤ L2 ‖x− x ∗‖2 , ∀x ∈ X , with L > 0 . The definition of L-smoothness for non-convex functions has been considered by [ 36 ] . Definition 3 . The function f is µ-strongly convex on X ⊆ Rd if f ( y ) ≥ f ( x ) + 〈g , y − x〉 + µ 2 ‖y − x‖ 2 , ∀x , y ∈ X , g ∈ ∂f ( x ) , with µ > 0 . Definition 4 . The function f is convex on X if f ( y ) ≥ f ( x ) + 〈g , y − x〉 for any g ∈ ∂f ( x ) and x , y ∈ X . Throughout the paper , we assume the objective function f is bounded below on X and let f∗ denote its infimum . If f is strongly convex , x∗ is the unique minimum point of f and f∗ = f ( x∗ ) . Notation : Let [ n ] denote the set of { 1 , 2 , · · · , n } and ‖·‖ : = ‖·‖2 without specific mention . We use brc and dre to denote the nearest integer to the real number r from below and above . For simplicity of notation , we assume that S , N , ( logα T ) /2 , logα T and T/ logα T are all integers . The subgradient of the function f on x is denoted by ∂f ( x ) : = { v | f ( y ) ≥ f ( x ) + 〈v , y − x〉 , ∀y } .

Summary: This paper proved convergence results for SGD with step decay step-size (the step size is decayed by some factor after certain iterations).   In the smooth non-convex setting, this paper sampled the output from $\\{x_t\\}$ with probability of $x_t$ being $O(1/\eta_t).$ Based on this novel approach of sampling the output, the authors  1. proved the first convergence bound for SGD with step-decay step-sizes on non-convex problems and achieved a $O(\ln T/\sqrt{T})$ rate towards a stationary point, which is comparable to the folklore results for $\eta_t = O(1/\sqrt{t})$ step-size; 2. improved the convergence results for exponentially decaying step-size, which allows larger initial step size compared with previous work; 3. improved the convergence rate from $O(\ln T/\sqrt{T})$ to $O(1/\sqrt{T})$ for the $1/\sqrt{t}$ step-size.  For the strongly convex and smooth functions, this paper proved an $O(\ln T/T)$ convergence rate for the last iteration, which generalized the result of Ge et al. The paper also proved a $O(\ln^2 T/T)$ convergence rate for the strongly convex and non-smooth functions.  In the experiments, the authors verified that step-decay step-size scheduling have better performance in training neural networks compared with constant step size or polynomially decaying step-size.