Rethinking learning rate schedules for stochastic optimization

1 INTRODUCTION . The recent advances in machine learning and deep learning rely almost exclusively on stochastic optimization methods , primarily SGD and its variants . Here , these large scale stochastic optimization methods are manually ( and often painstakingly ) tuned to the problem at hand ( often with parallelized hyper-parameter searches ) , where there is , as of yet , no class of “ universal methods ” which uniformly work well on a wide range of problems with little to no hyper-parameter tuning . This is in stark contrast to non-stochastic numerical optimization methods , where it is not an overstatement to argue that the l-BFGS and non-linear conjugate gradient methods ( with no hyper-parameter tuning whatsoever ) have provided nearly unbeatable procedures ( for a number of decades ) on nearly every unconstrained convex and non-convex problem . In the land of stochastic optimization , there are two dominant ( and somewhat compatible approaches ) : those methods which often manually tune learning rate schedules to achieve the best performance ( Krizhevsky et al. , 2012 ; Sutskever et al. , 2013 ; Kingma & Ba , 2014 ; Kidambi et al. , 2018 ) and those methods which rely on various forms of approximate preconditioning ( Duchi et al. , 2011 ; Tieleman & Hinton , 2012 ; Kingma & Ba , 2014 ) . This works examines the former class of methods , where we seek a more refined understanding of the issues of learning rate scheduling , through both theoretical analysis and empirical studies . Learning rate schedules for SGD is a rather enigmatic topic since there is a stark disparity between what is considered admissible in theory and what is employed in practice to achieve the best re- sults . Let us elaborate on this distinction more clearly . In theory , a vast majority of works starting with Robbins & Monro ( 1951 ) ; Polyak & Juditsky ( 1992 ) consider learning rates that have the form of ηt = ab+tα for some a , b ≥ 0 and 1/2 < α ≤ 1 – we call these polynomial decay schemes . The key property enjoyed by these polynomial decay schemes is that they are not summable but are square summable . A number of works obtain bounds on the asymptotic convergence rates of such schemes . Note that the focus of these works is to design learning rate schemes that work well for all large values of t. In contrast , practitioners are interested in achieving the best performance given a computational budget or equivalently a fixed time horizon T e.g. , 100 passes on training dataset with a batch size of 128 . The corresponding practically best performing learning rate scheme is often one where the step size is cut by a constant factor once every few epochs , or , equivalently , when no progress is made on a validation set ( Krizhevsky et al. , 2012 ; He et al. , 2016b ) ( often called a dev set based decay scheme ) . Such schemes are widely popular to the extent that they are available as schemes in deep learning libraries such as PyTorch 1 and several such useful tools of the trade are taught on popular deep learning courses 2 . Furthermore , what is ( often ) puzzling ( from a theory perspective ) is the emphasis that is laid on “ babysitting ” the learning rates 3 to achieve the best performance . Why do practitioners use constant and cut learning rate schemes while most of the theory work routinely works with polynomial decaying schemes ? Of course , implicit to this question is the view that both of these schemes are not equivalent . Indeed if both of these were equivalent , one could parameterize the learning rate as ab+tα and do hyperparameter search over a , b and α . In practice , this simply does not give results comparable to the constant and cut schemes.4 One potential explanation for this could be that , in the context of neural network training , local minima found by constant and cut schemes are of much better quality than those found by polynomial decay schemes , while for convex problems , polynomial decay schemes are indeed optimal . The primary contribution of this work is to show that this is simply not the case . We concretely show how minimax optimal theoretical learning rates ( i.e . polynomial decay schemes for wide classes of convex optimization problems ) may be misleading ( and sub-optimal for locally quadratic problems ) , and the story in practice is more nuanced . There important issues at play with regards to this suboptimality . First , even for the simple case of stochastic linear regression , with a fixed time horizon , the rate achieved by any polynomial decay scheme ( i.e. , any choice of a , b and α ) is suboptimal compared to the statistical minimax rate ( i.e. , information theoretically best possible rate achievable by any algorithm ) by a factor of condition number κ ( see Section 3 for definitions ) , while there exist constant and cut schemes that are suboptimal only by a factor of log κ . Second , this work shows that a factor of κ suboptimality is unavoidable if we wish to bound the error of each iterate of SGD . In other words , we show that the convergence rate of lim sup of the error , as t→∞ , has to be necessarily suboptimal by a factor of Ω̃ ( κ ) compared to the statistical minimax rate , for any learning rate sequence ( polynomial or not ) . In fact , at least Ω̃1/κ fraction of the iterates have this suboptimality . With this result , things become quite clear – all the works in stochastic approximation try to bound the error of each iterate of SGD asymptotically ( or lim sup of the error in other words ) . Since this necessarily has to be suboptimal by a factor of Ω̃ ( κ ) compared to the statistical minimax rates , the suboptimality of polynomial decay rates is not an issue . However , with a fixed time horizon , there exist learning rate schemes with much better convergence rates , while polynomial decay schemes fail to get better rates in this simpler setting ( of known time horizon ) . Thirdly , the work shows that , for stochastic linear regression , if we consider lim inf ( rather than lim sup ) of the error , it is possible to design schemes that are suboptimal by only a factor of log κ compared to the minimax rates . Variants of the constant and cut schemes achieve this guarantee . In summary , the contributions of this paper are showing how widely used pratical learning rate schedules are , in fact , highly effective even in the convex case . In particular , our theory and empirical results demonstrate this showing that : 1https : //pytorch.org/docs/stable/optim.html # torch.optim.lr_scheduler . ReduceLROnPlateau 2http : //cs231n.github.io/ 3http : //cs231n.github.io/neural-networks-3/ 4In fact , this work shows an instance where there is a significant ( provable ) difference between the perfor- mance of these two schemes . • For a fixed time horizon , constant and cut schemes are provably , significantly better than polynomial decay schemes . • There is a fundamental difference between fixed time horizon and infinite time horizon . • The above difference can be mitigated by considering lim inf of error instead of lim sup . • In addition to our theoretical contributions , we empirically verify the above claims for neural network training on cifar-10 . Extending results on the performance of constant and cut schemes to more general convex optimization problems , beyond stochastic linear regression , is an important future direction . However , the fact that the suboptimality of polynomial decay schemes even for the simple case of stochastic linear regression , has not been realized after decades of research on stochastic approximation is striking . In summary , the results of this paper show that , even for stochastic linear regression , the popular in practice , constant and cut learning rate schedules are provably better than polynomial decay schemes popular in theory and that there is a need to rethink learning rate schemes and convergence guarantees for stochastic approximation . Our results also suggest that current approaches to hyperparameter tuning of learning rate schedules might not be right headed and further suggest potential ways of improving them . Paper organization : The paper is organized as follows . We review related work in Section 2 . Section 3 describes the notation and problem setup . Section 4 presents our results on the suboptimality of both polynomial decay schemes and constant and cut schemes . Section 5 presents results on infinite horizon setting . Section 6 presents experimental results and Section 7 concludes the paper . 2 RELATED WORK . We will split related work into two parts , one based on theory and the other based on practice . Related efforts in theory : SGD and the problem of stochastic approximation was introduced in the seminal work of Robbins & Monro ( 1951 ) ; this work also elaborates on stepsize schemes that are satisfied by asymptotically convergent stochastic gradient methods : we refer to these schemes as “ convergent ” stepsize sequences . The ( asymptotic ) statistical optimality of iterate averaged SGD with larger stepsize schemes ofO ( 1/nα ) with α ∈ ( 0.5 , 1 ) was proven in the seminal works of Ruppert ( 1988 ) ; Polyak & Juditsky ( 1992 ) . The notions of convergent learning rate schemes in stochastic approximation literature has been studied in great detail ( Ljung et al. , 1992 ; Kushner & Yin , 2003 ; Bharath & Borkar , 1999 ; Lai , 2003 ) . Nearly all of the aforementioned works rely on function value sub-optimality to measure convergence and rely on the notion of asymptotic convergence ( i.e . in the limit of the number of updates of SGD tending to infinity ) to derive related “ convergent stepsize schedules ” . Along this line of thought , there are several efforts that prove ( minimax ) optimality of the aforementioned rates ( in a worst case sense and not per problem sense ) e.g. , Nemirovsky & Yudin ( 1983 ) ; Raginsky & Rakhlin ( 2011 ) ; Agarwal et al . ( 2012 ) . An alternative viewpoint is to consider gradient norm as a means to measure the progress of an algorithm . Along this line of thought are several works including the stochastic process viewpoint considered by Polyak & Juditsky ( 1992 ) and more recently , the work of Nesterov ( 2012 ) ( working with deterministic ( exact ) gradients ) . The work of Allen-Zhu ( 2018 ) considers questions relating to making the gradient norm small when working with stochastic gradients , and provides an improved rate . We return to this criterion in Section 7 . In terms of oracle models , note that both this paper , as well as other results ( Lacoste-Julien et al. , 2012 ; Rakhlin et al. , 2012 ; Bubeck , 2014 ) , work in an oracle model that assumes bounded variance of stochastic gradients or similar assumptions . There is an alternative oracle model for analyzing SGD as followed in papers includingBach & Moulines ( 2013 ) ; Bach ( 2014 ) ; Jain et al . ( 2017 ) which is arguably more reflective of SGD ’ s behavior in practice . For more details , refer to Jain et al . ( 2017 ) . It is an important direction to prove the results of this paper working in the alternative practically more applicable oracle model . Efforts in practice : As highlighted in the introduction , practical efforts in stochastic optimization have diverged from the classical theory of stochastic approximation , with several deep learning libraries like pytorch 5 providing unconventional alternatives such as cosine/sawtooth/dev set decay schemes , or even exponentially decaying learning rate schemes . In fact , a natural scheme used in training convolutional neural networks for vision is where the learning rate is cut by a constant factor after a certain number of epochs . Such schemes are essentially discretized variants of exponentially decaying learning rate schedules . We note that there are other learning rate schedules that have been recently proposed such as sgd with warm restarts ( Loshchilov & Hutter , 2016 ) , oscillating learning rates ( Smith & Topin , 2017 ) etc. , that are unconventional and have attracted a fair bit of attention . Furthermore , exponential learning rates appear to be considered in more recent NLP papers ( see for e.g. , Krishnamurthy et al . ( 2017 ) ) 6 .

Summary: This work provides theoretical insights on recent learning rate proposals such as Cyclical Learning Rates (Smith et al.). The authors focus on stochastic approximation i.e. how large is the SGD loss as a function of condition number and horizon. The critical contribution is the theoretical benefit of oscillating learning rates over more traditional learning rate schemes. Authors provide novel upper/lower bounds to establish benefit of oscillating LR, support their theory with experiments and provide insights on finite horizon learning rate selection. An important drawback is that results only apply to linear regression which is a fairly simple setup.