Adaptive Gradient Methods Converge Faster with Over-Parameterization (and you can do a line-search)

1 INTRODUCTION . Adaptive gradient methods such as AdaGrad ( Duchi et al. , 2011 ) , RMSProp ( Tieleman & Hinton , 2012 ) , AdaDelta ( Zeiler , 2012 ) , Adam ( Kingma & Ba , 2015 ) , and AMSGrad ( Reddi et al. , 2018 ) are popular optimizers for training deep neural networks ( Goodfellow et al. , 2016 ) . These methods scale well and exhibit good performance across problems , making them the default choice for many machine learning applications . Theoretically , these methods are usually studied in the non-smooth , online convex optimization setting ( Duchi et al. , 2011 ; Reddi et al. , 2018 ) with recent extensions to the strongly-convex ( Mukkamala & Hein , 2017 ; Wang et al. , 2020 ; Xie et al. , 2020 ) and non-convex settings ( Li & Orabona , 2019 ; Ward et al. , 2019 ; Zhou et al. , 2018 ; Chen et al. , 2019 ; Wu et al. , 2019 ; Défossez et al. , 2020 ; Staib et al. , 2019 ) . An online–batch reduction gives guarantees similar to stochastic gradient descent ( SGD ) in the offline setting ( Cesa-Bianchi et al. , 2004 ; Hazan & Kale , 2014 ; Levy et al. , 2018 ) . However , there are several discrepancies between the theory and application of these methods . Although the theory advocates for using decreasing step-sizes for Adam , AMSGrad and its variants ( Kingma & Ba , 2015 ; Reddi et al. , 2018 ) , a constant step-size is typically used in practice ( Paszke et al. , 2019 ) . Similarly , the standard analysis of these methods assumes a decreasing momentum parameter , however , the momentum is fixed in practice . On the other hand , AdaGrad ( Duchi et al. , 2011 ) has been shown to be “ universal ” as it attains the best known convergence rates in both the stochastic smooth and non-smooth settings ( Levy et al. , 2018 ) , but its empirical performance is rather disappointing when training deep models ( Kingma & Ba , 2015 ) . Improving the empirical performance was indeed the main motivation behind Adam and other methods ( Tieleman & Hinton , 2012 ; Zeiler , 2012 ) that followed AdaGrad . Although these methods have better empirical performance , they are not guaranteed to converge to the solution with a constant step-size and momentum parameter . Another inconsistency is that although the standard theoretical results are for non-smooth functions , these methods are also extensively used in the easier , smooth setting . More importantly , adaptive gradient methods are generally used to train highly expressive , large over-parameterized models ( Zhang et al. , 2017 ; Liang & Rakhlin , 2018 ) capable of interpolating the data . However , the standard theoretical analyses do not take advantage of these additional properties . On the other hand , a line of recent work ( Schmidt & Le Roux , 2013 ; Jain et al. , 2018 ; Ma et al. , 2018 ; Liu & Belkin , 2020 ; Cevher & Vũ , 2019 ; Vaswani et al. , 2019a ; b ; Wu et al. , 2019 ; Loizou et al. , 2020 ) focuses on the convergence of SGD in this interpolation setting . In the standard finite-sum case , interpolation implies that all the functions in the sum are minimized at the same solution . Under this additional assumption , these works show SGD with a constant step-size converges to the minimizer at a faster rate for both convex and non-convex smooth functions . In this work , we aim to resolve some of the discrepancies in the theory and practice of adaptive gradient methods . To theoretically analyze these methods , we consider a simplistic setting - smooth , convex functions under interpolation . Using the intuition gained from theory , we propose better techniques to adaptively set the step-size for these methods , dramatically improving their empirical performance when training over-parameterized models . 1.1 BACKGROUND AND CONTRIBUTIONS . Constant step-size . We focus on the theoretical convergence of two adaptive gradient methods : AdaGrad and AMSGrad . For smooth , convex functions , Levy et al . ( 2018 ) prove that AdaGrad with a constant step-size adapts to the smoothness and gradient noise , resulting in an O ( 1/T + ζ/ √ T ) convergence rate , where T is the number of iterations and ζ2 is a global bound on the variance in the stochastic gradients . This convergence rate matches that of SGD under the same setting ( Moulines & Bach , 2011 ) . In Section 3 , we show that constant step-size AdaGrad also adapts to interpolation and prove an O ( 1/T + σ/ √ T ) rate , where σ is the extent to which interpolation is violated . In the over-parameterized setting , σ2 can be much smaller than ζ2 ( Zhang & Zhou , 2019 ) , implying a faster convergence . When interpolation is exactly satisfied , σ2 = 0 , we obtain an O ( 1/T ) rate , while ζ2 can still be large . In the online convex optimization framework , for smooth functions , we show that the regret of AdaGrad improves from O ( √ T ) to O ( 1 ) when interpolation is satisfied and retains its O ( √ T ) -regret guarantee in the general setting ( Appendix C.2 ) . Assuming its corresponding preconditioner remains bounded , we show that AMSGrad with a constant step-size and constant momentum parameter also converges at the rate O ( 1/T ) under interpolation ( Section 4 ) . However , unlike AdaGrad , it requires specific step-sizes that depend on the problem ’ s smoothness . More generally , constant step-size AMSGrad converges to a neighbourhood of the solution , attaining an O ( 1/T + σ2 ) rate , which matches the rate of constant step-size SGD in the same setting ( Schmidt & Le Roux , 2013 ; Vaswani et al. , 2019a ) . When training over-parameterized models , this result provides some justification for the faster ( O ( 1/T ) vs. O ( 1/ √ T ) ) convergence of the AMSGrad variant typically used in practice . Adaptive step-size . Although AdaGrad converges at the same asymptotic rate for any step-size ( up to constants ) , it is unclear how to choose this step-size without manually trying different values . Similarly , AMSGrad is sensitive to the step-size , converging only for a specific range in both theory and practice . In Section 5 , we experimentally show that even for simple , convex problems , the step-size has a big impact on the empirical performance of AdaGrad and AMSGrad . To overcome this limitation , we use recent methods ( Vaswani et al. , 2019a ; Loizou et al. , 2020 ) that automatically set the step-size for SGD . These works use stochastic variants of the classical Armijo line-search ( Armijo , 1966 ) or the Polyak step-size ( Polyak , 1963 ) in the interpolation setting . We combine these techniques with adaptive gradient methods and show that a variant of stochastic line-search ( SLS ) enables AdaGrad to adapt to the smoothness of the underlying function , resulting in faster empirical convergence , while retaining its favourable convergence properties ( Section 3 ) . Similarly , AMSGrad with variants of SLS and SPS can match the convergence rate of its constant step-size counterpart , but without knowledge of the underlying smoothness properties ( Section 4 ) . Experimental results . Finally , in Section 5 , we benchmark our results against SGD variants with SLS ( Vaswani et al. , 2019b ) , SPS ( Loizou et al. , 2020 ) , tuned Adam and its recently proposed variants ( Luo et al. , 2019 ; Liu et al. , 2020 ) . We demonstrate that the proposed techniques for setting the step-size improve the empirical performance of adaptive gradient methods . These improvements are consistent across tasks , ranging from binary classification with a kernel mapping to multi-class classification using standard deep neural network architectures . 2 PROBLEM SETUP . We consider the unconstrained minimization of an objective f : Rd → R with a finite-sum structure , f ( w ) = 1n ∑n i=1 fi ( w ) . In supervised learning , n represents the number of training examples , and fi is the loss function on training example i . Although we focus on the finite-sum setting , our results can be easily generalized to the online optimization setting . The objective of our analysis is to better understand the effect of the step-size and line-searches when interpolation is ( almost ) satisfied . This is complicated by the fact that adaptive methods are still poorly understood ; state-of-the-art analyses do not show an improvement over gradient descent in the worst-case . To focus on the effect of step-sizes , we make the simplifying assumptions described in this section . We assume f and each fi are differentiable , convex , and lower-bounded by f∗ and f∗i , respectively . Furthermore , we assume that each function fi in the finite-sum is Li-smooth , implying that f is Lmax-smooth , where Lmax = maxi Li . We also make the standard assumption that the iterates remain bounded in a ball of radius D around a global minimizer , ‖wk − w∗‖ ≤ D for all wk ( Ahn et al. , 2020 ) . We remark that the bounded iterates assumption simplifies the analysis but is not essential , and similar to Reddi et al . ( 2018 ) ; Duchi et al . ( 2011 ) ; Levy et al . ( 2018 ) , our theoretical results can be extended to include a projection step . We include the formal definitions of these properties ( Nemirovski et al. , 2009 ) in Appendix A . The interpolation assumption means that the gradient of each fi in the finite-sum converges to zero at an optimum . If the overall objective f is minimized at w∗ , ∇f ( w∗ ) = 0 , then for all fi we have ∇fi ( w∗ ) = 0 . The interpolation condition can be exactly satisfied for many over-parameterized machine learning models such as non-parametric kernel regression without regularization ( Belkin et al. , 2019 ; Liang & Rakhlin , 2018 ) and over-parameterized deep neural networks ( Zhang et al. , 2017 ) . We measure the extent to which interpolation is violated by the disagreement between the minimum overall function value f∗ and the minimum value of each individual functions f∗i , σ2 : = Ei [ f∗ − f∗i ] ∈ [ 0 , ∞ ) ( Loizou et al. , 2020 ) . The minimizer of f need not be unique for σ2 to be uniquely defined , as it only depends on the minimum function values . Interpolation is said to be exactly satisfied if σ2 = 0 , and we also study the setting when σ2 > 0 . For a preconditioner matrix Ak and a constant momentum parameter β ∈ [ 0 , 1 ) , the update for a generic adaptive gradient method at iteration k can be expressed as : wk+1 = wk − ηk A−1k mk ; mk = βmk−1 + ( 1− β ) ∇fik ( wk ) ( 1 ) Here , ∇fik ( wk ) is the stochastic gradient of a randomly chosen function fik , and ηk is the step-size . Adaptive gradient methods typically differ in how their preconditioners are constructed and whether or not they include the momentum term βmk−1 ( see Table 1 for a list of common methods ) . Both RMSProp and Adam maintain an exponential moving average of past stochastic gradients , but as Reddi et al . ( 2018 ) pointed out , unlike AdaGrad , the corresponding preconditioners do not guarantee that Ak+1 Ak and the resulting per-dimension step-sizes do not go to zero . This can lead to large fluctuations in the effective step-size and prevent these methods from converging . To mitigate this problem , they proposed AMSGrad , which ensures Ak+1 Ak and the convergence of iterates . Consequently , our theoretical results focus on AdaGrad , AMSGrad and other adaptive gradient methods that ensure this monotonicity . However , we also considered RMSProp and Adam in our experimental evaluation . Although our theory holds for both the full matrix and diagonal variants ( where Ak is a diagonal matrix ) of these methods , we use only the latter in experiments for scalability . The diagonal variants perform a per-dimension scaling of the gradient and avoid computing the full matrix inverse , so their per-iteration cost is the same as SGD , although with an additional O ( d ) memory . For AMSGrad , we assume that the corresponding preconditioners are well-behaved in the sense that their eigenvalues are bounded in an interval [ amin , amax ] . This is a common assumption made in the analysis of adaptive methods . Moreover , for diagonal preconditioners , such a boundedness property is easy to verify , and it is also inexpensive to maintain the desired range by projection . Our main theoretical results for AdaGrad ( Section 3 ) and AMSGrad ( Section 4 ) are summarized in Table 2 .

Summary: This paper studies adaptive gradient methods under the over-parametrized settings, where the authors study the converge in the interpolation setting. In this setting, the optimal objective is 0. The authors show that the convergence rate is O(1/T). In addition, when the interpolation is approximately satisfied, the authors show the convergence to a neighborhood of the solution. The authors also provide theoretical justifications for popular line search methods.