On the Convergence of Step Decay Step-Size for Stochastic Optimization

√ T ) rate . We also provide near-optimal ( and sometimes provably tight ) convergence guarantees for general , possibly non-smooth , convex and strongly convex problems . The practical efficiency of the step decay step-size is demonstrated in several large-scale deep neural network training tasks . 1 Introduction . We focus on stochastic programming problems on the form min x∈X f ( x ) : = Eξ∼Ξ [ f ( x ; ξ ) ] . ( 1 ) Here , ξ is a random variable drawn from some source distribution Ξ over an arbitrary probability space and X is a closed , convex subset of Rd . This problem is often encountered in machine learning applications , such as training of deep neural networks . Depending on the specifics of the application , the function f can either be non-convex , convex or strongly convex ; it can be smooth or non-smooth ; and it may also have additional structure that can be exploited . Despite the many advances in the field of stochastic programming , the stochastic gradient descent ( SGD ) method [ 37 , 32 ] remains important and is arguably still the most popular method for solving ( 1 ) . The SGD method updates the decision vector x using the following recursion xt+1 = ΠX ( xt − ηtĝt ) ( 2 ) where ĝt is an unbiased estimate of the gradient ( or subgradient ) at xt and ΠX is the Euclidean projection onto X . The step-size parameter ( learning-rate ) ηt > 0 is critical to control the rate at which the model learns and to guarantee that the SGD iterates converge to an optimizer of ( 1 ) . Setting the step-size too large will result in iterates which never converge ; and setting it too small leads to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) slow convergence and may even cause the iterates to get stuck at bad local minima . As long as the iterates do not diverge , a large constant step-size promotes fast convergence but only to a ( large ) neighborhood of the optimal solution . To increase the accuracy , we have to decrease the step-size . The traditional approach is to decrease the step-size in every iteration , typically as η0/t or η0/ √ t. Both these step-size schedules have been studied extensively and guarantee a non-asymptotic convergence of SGD [ 31 , 26 , 36 , 19 , 38 , 15 ] . However , from a practical perspective , these step-size policies often perform poorly , since they begin to decrease too early . For non-convex problems , such as those which arise in training of deep neural networks , the most successful and popular step-size policy in practice is the step decay step-size [ 25 , 21 , 22 ] . This step-size policy starts with a relatively large constant step-size and then cuts the step-size by a fixed number ( called decay factor ) at after a given number of epochs . Not only does this step-size result in a faster initial convergence , but it also guarantees that the SGD iterates eventually converge to an exact solution . The step decay step-size is the default choice in many deep learning libraries , such as TensorFlow [ 4 ] and PyTorch [ 35 ] ; both use a decay rate of 0.1 and user-defined milestones when the step-size is decreased . Although some recent studies attempt to monitor the optimization process to trigger the milestones when certain conditions are met [ 27 , 50 ] , such approaches are difficult to analyze . Another common approach is to simply divide the targeted number of iterations into N intervals of equal length , and trigger a milestone at the end of each such interval . However , the theoretical properties of this simpler step-size policy is also not yet well understood . To make the problem more manageable theoretically , some related step-size policies that let the length of each segment increase linearly [ 6 ] or exponentially [ 19 , 8 , 49 ] are analyzed . On the contrary , we consider the more practical case when the horizon T of the step-decay step-size is divided into several equal parts . If the number of iterations or horizon T is known apriori , reference [ 11 ] analyzes a step decay step-size with a decay rate of 1/2 applied every T/ log2 T iterations and establishes a near-optimal O ( log2 T/T ) convergence rate for vanilla SGD on least-squares problems . However , to our knowledge , there are no convergence guarantees in the literature under more general conditions , e.g. , for general strongly-convex , general convex , or non-convex problems . Motivated by this , in this paper , we study the non-asymptotic convergence of SGD with the step decay step-size in more general settings . 1.1 Main Contributions . This work establishes novel convergence guarantees for SGD with the step decay step-size on nonconvex , convex and strongly convex optimization problems . We make the following contributions : • We propose a non-uniform probability rule Pt ∝ 1/ηt for selecting the output in the smooth non-convex setting . Based on this rule , we ( i ) establish a near-optimal O ( lnT/ √ T ) rate for SGD with step decay step-size ; ( ii ) improve the results for exponential decay step-size [ 28 ] ; ( iii ) remove the lnT factor in the best known convergence rate for the 1/ √ t step-size . • For strongly convex problems , we establish the following error bounds for the last iterate under step-decay : ( i ) O ( lnT/T ) for smooth problem , which we also prove to be tight ; ( ii ) O ( ln2 T/T ) without the smoothness assumption . • For the general convex case , we prove that the step decay step-size at the last iterate can achieve a near-optimal convergence rate ( up to a lnT factor ) . 1.2 Related Work . For SGD , the best known bound for the expected error of the T th iterate is of O ( 1/ √ T ) when the objective is convex and smooth [ 32 , 12 ] , and of O ( 1/T ) when the objective is also strongly convex [ 31 , 36 ] . Without any further assumptions , these rates are known to be optimal . If we restrict our attention to diminishing step-sizes , ηt = η0/t , the best known error bound for strongly convex and non-smooth problems is of O ( lnT/T ) [ 38 ] , which is also tight [ 17 ] . This rate can be improved to O ( 1/T ) by averaging strategies [ 36 , 26 , 38 ] or a step decay step-size [ 19 ] . For smooth non-convex functions , Reference [ 12 ] established anO ( 1/ √ T ) rate for SGD with a constant step-size ηt = O ( 1/ √ T ) . Recently , this error bound has proven to be tight up to a constant , unless additional assumptions are made [ 9 ] . Step decay step-sizes were probably first considered for deterministic subgradient methods [ 14 ] and [ 40 ] . More recently step decay step-size schemes have been employed to improve the convergence rates under various conditions : local growth ( convex ) [ 45 ] , Polyak-Lójasiewicz ( PL ) condition [ 49 ] , sharp growth ( non-convex ) [ 8 ] . Most of these references consider proximal algorithms ( which add a quadratic term λ ‖·‖2 to the original loss function ) and , in addition , perform an extra averaging step between stages . In particular , if the function f is µ-weakly convex ( i.e . f + µ2 ‖·‖ 2 is convex ) , λ is required to be larger than µ2 [ 49 , 6 ] . In contrast , we study the performance of step decay step-size for the standard SGD algorithm . Moreover , in [ 45 ] the output of each inner-loop is an uniform average . So their complexity reflects the average performance over iterations . But for convex and strongly convex problems , we focus on the final iterate which is the most preferred choice in practice . In fact , most practitioners just use the final iterate . Reference [ 6 ] considers slowly decaying step-sizes , ∼ 1/t where t is the stage index . In other words , the step-size is not cut by a constant factor between stages as we do , but it decays ( very ) slowly . Compared to [ 19 ] and [ 49 ] , where the inner-loop size S grows exponentially , we use a constant value of S , which is known to work better in practice . In the extreme case when S = 1 , step-decay reduces to the exponentially decaying step-size recently studied under the PL condition and a general smoothness assumption [ 28 ] . A number of adaptive step-size selection strategies have been proposed for SGD ( e.g. , [ 10 , 42 , 23 , 30 , 29 ] ) , some of which result in step-decay policies [ 44 , 27 , 50 ] . For example , Reference [ 27 ] develops a statistical procedure to automatically determine when the SGD iterates with a constant step-size no longer make progress , and then halve the step-size . Empirically , this automatic scheme is competitive with the best hand-tuned step-decay schedules , but no formal guarantees for this observed behaviour are given . The paper is organized as follows . Notation and basic definitions are introduced in Section 2 . In Section 3 , we analyze the convergence rates of step decay step-size on nonconvex case and propose a novel non-uniform sampling rule for the algorithm output . The convergence for general convex and strongly convex functions are investigated in Sections A and 4 , respectively . Numerical results of our algorithms are presented and discussed in Section 5 . Finally , conclusions are made in Section 6 . 2 Preliminaries . In this part , we will give some definitions and notations used throughout the paper . Definition 1 . ( 1 ) The stochastic gradient oracle Õ is variance-bounded if for any input vector x̂ , it returns a random vector ĝ such that E [ ‖ĝ − E [ ĝ ] ‖2 ] ≤ V 2 . ( 2 ) Õ is bounded if for any input vector x̂ , it returns a random vector ĝ such that E [ ‖ĝ‖2 ] ≤ G2 for some fixed G > 0 . Definition 2 . When the function f is differentiable on X , we say that f is L-smooth on X if there exists a constant L > 0 such that ‖∇f ( x ) −∇f ( y ) ‖ ≤ L ‖x− y‖ . It also implies that f ( y ) ≤ f ( x ) + 〈∇f ( x ) , y − x〉+ L2 ‖x− y‖ 2 for any x , y ∈ X . If f is not differentiable on X , we say that f is L-smooth with respect to x∗ if f ( x ) − f ( x∗ ) ≤ L2 ‖x− x ∗‖2 , ∀x ∈ X , with L > 0 . The definition of L-smoothness for non-convex functions has been considered by [ 36 ] . Definition 3 . The function f is µ-strongly convex on X ⊆ Rd if f ( y ) ≥ f ( x ) + 〈g , y − x〉 + µ 2 ‖y − x‖ 2 , ∀x , y ∈ X , g ∈ ∂f ( x ) , with µ > 0 . Definition 4 . The function f is convex on X if f ( y ) ≥ f ( x ) + 〈g , y − x〉 for any g ∈ ∂f ( x ) and x , y ∈ X . Throughout the paper , we assume the objective function f is bounded below on X and let f∗ denote its infimum . If f is strongly convex , x∗ is the unique minimum point of f and f∗ = f ( x∗ ) . Notation : Let [ n ] denote the set of { 1 , 2 , · · · , n } and ‖·‖ : = ‖·‖2 without specific mention . We use brc and dre to denote the nearest integer to the real number r from below and above . For simplicity of notation , we assume that S , N , ( logα T ) /2 , logα T and T/ logα T are all integers . The subgradient of the function f on x is denoted by ∂f ( x ) : = { v | f ( y ) ≥ f ( x ) + 〈v , y − x〉 , ∀y } .

Summary: The authors present a new analysis of SGD with the very popular step decay implemented in all DL framework for instance. They provide convergence guarantee in various setting and show that the classical exponential decay is a very special case of their result, still achieving the state of the art result in this special case. They also provide some lower bound to show tightness of their results and finally propose a new sampling of the iterates that allows to remove some annoying log term in the convergence bound.