Adaptive Gradient Methods Converge Faster with Over-Parameterization (and you can do a line-search)

1 INTRODUCTION . Adaptive gradient methods such as AdaGrad ( Duchi et al. , 2011 ) , RMSProp ( Tieleman & Hinton , 2012 ) , AdaDelta ( Zeiler , 2012 ) , Adam ( Kingma & Ba , 2015 ) , and AMSGrad ( Reddi et al. , 2018 ) are popular optimizers for training deep neural networks ( Goodfellow et al. , 2016 ) . These methods scale well and exhibit good performance across problems , making them the default choice for many machine learning applications . Theoretically , these methods are usually studied in the non-smooth , online convex optimization setting ( Duchi et al. , 2011 ; Reddi et al. , 2018 ) with recent extensions to the strongly-convex ( Mukkamala & Hein , 2017 ; Wang et al. , 2020 ; Xie et al. , 2020 ) and non-convex settings ( Li & Orabona , 2019 ; Ward et al. , 2019 ; Zhou et al. , 2018 ; Chen et al. , 2019 ; Wu et al. , 2019 ; Défossez et al. , 2020 ; Staib et al. , 2019 ) . An online–batch reduction gives guarantees similar to stochastic gradient descent ( SGD ) in the offline setting ( Cesa-Bianchi et al. , 2004 ; Hazan & Kale , 2014 ; Levy et al. , 2018 ) . However , there are several discrepancies between the theory and application of these methods . Although the theory advocates for using decreasing step-sizes for Adam , AMSGrad and its variants ( Kingma & Ba , 2015 ; Reddi et al. , 2018 ) , a constant step-size is typically used in practice ( Paszke et al. , 2019 ) . Similarly , the standard analysis of these methods assumes a decreasing momentum parameter , however , the momentum is fixed in practice . On the other hand , AdaGrad ( Duchi et al. , 2011 ) has been shown to be “ universal ” as it attains the best known convergence rates in both the stochastic smooth and non-smooth settings ( Levy et al. , 2018 ) , but its empirical performance is rather disappointing when training deep models ( Kingma & Ba , 2015 ) . Improving the empirical performance was indeed the main motivation behind Adam and other methods ( Tieleman & Hinton , 2012 ; Zeiler , 2012 ) that followed AdaGrad . Although these methods have better empirical performance , they are not guaranteed to converge to the solution with a constant step-size and momentum parameter . Another inconsistency is that although the standard theoretical results are for non-smooth functions , these methods are also extensively used in the easier , smooth setting . More importantly , adaptive gradient methods are generally used to train highly expressive , large over-parameterized models ( Zhang et al. , 2017 ; Liang & Rakhlin , 2018 ) capable of interpolating the data . However , the standard theoretical analyses do not take advantage of these additional properties . On the other hand , a line of recent work ( Schmidt & Le Roux , 2013 ; Jain et al. , 2018 ; Ma et al. , 2018 ; Liu & Belkin , 2020 ; Cevher & Vũ , 2019 ; Vaswani et al. , 2019a ; b ; Wu et al. , 2019 ; Loizou et al. , 2020 ) focuses on the convergence of SGD in this interpolation setting . In the standard finite-sum case , interpolation implies that all the functions in the sum are minimized at the same solution . Under this additional assumption , these works show SGD with a constant step-size converges to the minimizer at a faster rate for both convex and non-convex smooth functions . In this work , we aim to resolve some of the discrepancies in the theory and practice of adaptive gradient methods . To theoretically analyze these methods , we consider a simplistic setting - smooth , convex functions under interpolation . Using the intuition gained from theory , we propose better techniques to adaptively set the step-size for these methods , dramatically improving their empirical performance when training over-parameterized models . 1.1 BACKGROUND AND CONTRIBUTIONS . Constant step-size . We focus on the theoretical convergence of two adaptive gradient methods : AdaGrad and AMSGrad . For smooth , convex functions , Levy et al . ( 2018 ) prove that AdaGrad with a constant step-size adapts to the smoothness and gradient noise , resulting in an O ( 1/T + ζ/ √ T ) convergence rate , where T is the number of iterations and ζ2 is a global bound on the variance in the stochastic gradients . This convergence rate matches that of SGD under the same setting ( Moulines & Bach , 2011 ) . In Section 3 , we show that constant step-size AdaGrad also adapts to interpolation and prove an O ( 1/T + σ/ √ T ) rate , where σ is the extent to which interpolation is violated . In the over-parameterized setting , σ2 can be much smaller than ζ2 ( Zhang & Zhou , 2019 ) , implying a faster convergence . When interpolation is exactly satisfied , σ2 = 0 , we obtain an O ( 1/T ) rate , while ζ2 can still be large . In the online convex optimization framework , for smooth functions , we show that the regret of AdaGrad improves from O ( √ T ) to O ( 1 ) when interpolation is satisfied and retains its O ( √ T ) -regret guarantee in the general setting ( Appendix C.2 ) . Assuming its corresponding preconditioner remains bounded , we show that AMSGrad with a constant step-size and constant momentum parameter also converges at the rate O ( 1/T ) under interpolation ( Section 4 ) . However , unlike AdaGrad , it requires specific step-sizes that depend on the problem ’ s smoothness . More generally , constant step-size AMSGrad converges to a neighbourhood of the solution , attaining an O ( 1/T + σ2 ) rate , which matches the rate of constant step-size SGD in the same setting ( Schmidt & Le Roux , 2013 ; Vaswani et al. , 2019a ) . When training over-parameterized models , this result provides some justification for the faster ( O ( 1/T ) vs. O ( 1/ √ T ) ) convergence of the AMSGrad variant typically used in practice . Adaptive step-size . Although AdaGrad converges at the same asymptotic rate for any step-size ( up to constants ) , it is unclear how to choose this step-size without manually trying different values . Similarly , AMSGrad is sensitive to the step-size , converging only for a specific range in both theory and practice . In Section 5 , we experimentally show that even for simple , convex problems , the step-size has a big impact on the empirical performance of AdaGrad and AMSGrad . To overcome this limitation , we use recent methods ( Vaswani et al. , 2019a ; Loizou et al. , 2020 ) that automatically set the step-size for SGD . These works use stochastic variants of the classical Armijo line-search ( Armijo , 1966 ) or the Polyak step-size ( Polyak , 1963 ) in the interpolation setting . We combine these techniques with adaptive gradient methods and show that a variant of stochastic line-search ( SLS ) enables AdaGrad to adapt to the smoothness of the underlying function , resulting in faster empirical convergence , while retaining its favourable convergence properties ( Section 3 ) . Similarly , AMSGrad with variants of SLS and SPS can match the convergence rate of its constant step-size counterpart , but without knowledge of the underlying smoothness properties ( Section 4 ) . Experimental results . Finally , in Section 5 , we benchmark our results against SGD variants with SLS ( Vaswani et al. , 2019b ) , SPS ( Loizou et al. , 2020 ) , tuned Adam and its recently proposed variants ( Luo et al. , 2019 ; Liu et al. , 2020 ) . We demonstrate that the proposed techniques for setting the step-size improve the empirical performance of adaptive gradient methods . These improvements are consistent across tasks , ranging from binary classification with a kernel mapping to multi-class classification using standard deep neural network architectures . 2 PROBLEM SETUP . We consider the unconstrained minimization of an objective f : Rd → R with a finite-sum structure , f ( w ) = 1n ∑n i=1 fi ( w ) . In supervised learning , n represents the number of training examples , and fi is the loss function on training example i . Although we focus on the finite-sum setting , our results can be easily generalized to the online optimization setting . The objective of our analysis is to better understand the effect of the step-size and line-searches when interpolation is ( almost ) satisfied . This is complicated by the fact that adaptive methods are still poorly understood ; state-of-the-art analyses do not show an improvement over gradient descent in the worst-case . To focus on the effect of step-sizes , we make the simplifying assumptions described in this section . We assume f and each fi are differentiable , convex , and lower-bounded by f∗ and f∗i , respectively . Furthermore , we assume that each function fi in the finite-sum is Li-smooth , implying that f is Lmax-smooth , where Lmax = maxi Li . We also make the standard assumption that the iterates remain bounded in a ball of radius D around a global minimizer , ‖wk − w∗‖ ≤ D for all wk ( Ahn et al. , 2020 ) . We remark that the bounded iterates assumption simplifies the analysis but is not essential , and similar to Reddi et al . ( 2018 ) ; Duchi et al . ( 2011 ) ; Levy et al . ( 2018 ) , our theoretical results can be extended to include a projection step . We include the formal definitions of these properties ( Nemirovski et al. , 2009 ) in Appendix A . The interpolation assumption means that the gradient of each fi in the finite-sum converges to zero at an optimum . If the overall objective f is minimized at w∗ , ∇f ( w∗ ) = 0 , then for all fi we have ∇fi ( w∗ ) = 0 . The interpolation condition can be exactly satisfied for many over-parameterized machine learning models such as non-parametric kernel regression without regularization ( Belkin et al. , 2019 ; Liang & Rakhlin , 2018 ) and over-parameterized deep neural networks ( Zhang et al. , 2017 ) . We measure the extent to which interpolation is violated by the disagreement between the minimum overall function value f∗ and the minimum value of each individual functions f∗i , σ2 : = Ei [ f∗ − f∗i ] ∈ [ 0 , ∞ ) ( Loizou et al. , 2020 ) . The minimizer of f need not be unique for σ2 to be uniquely defined , as it only depends on the minimum function values . Interpolation is said to be exactly satisfied if σ2 = 0 , and we also study the setting when σ2 > 0 . For a preconditioner matrix Ak and a constant momentum parameter β ∈ [ 0 , 1 ) , the update for a generic adaptive gradient method at iteration k can be expressed as : wk+1 = wk − ηk A−1k mk ; mk = βmk−1 + ( 1− β ) ∇fik ( wk ) ( 1 ) Here , ∇fik ( wk ) is the stochastic gradient of a randomly chosen function fik , and ηk is the step-size . Adaptive gradient methods typically differ in how their preconditioners are constructed and whether or not they include the momentum term βmk−1 ( see Table 1 for a list of common methods ) . Both RMSProp and Adam maintain an exponential moving average of past stochastic gradients , but as Reddi et al . ( 2018 ) pointed out , unlike AdaGrad , the corresponding preconditioners do not guarantee that Ak+1 Ak and the resulting per-dimension step-sizes do not go to zero . This can lead to large fluctuations in the effective step-size and prevent these methods from converging . To mitigate this problem , they proposed AMSGrad , which ensures Ak+1 Ak and the convergence of iterates . Consequently , our theoretical results focus on AdaGrad , AMSGrad and other adaptive gradient methods that ensure this monotonicity . However , we also considered RMSProp and Adam in our experimental evaluation . Although our theory holds for both the full matrix and diagonal variants ( where Ak is a diagonal matrix ) of these methods , we use only the latter in experiments for scalability . The diagonal variants perform a per-dimension scaling of the gradient and avoid computing the full matrix inverse , so their per-iteration cost is the same as SGD , although with an additional O ( d ) memory . For AMSGrad , we assume that the corresponding preconditioners are well-behaved in the sense that their eigenvalues are bounded in an interval [ amin , amax ] . This is a common assumption made in the analysis of adaptive methods . Moreover , for diagonal preconditioners , such a boundedness property is easy to verify , and it is also inexpensive to maintain the desired range by projection . Our main theoretical results for AdaGrad ( Section 3 ) and AMSGrad ( Section 4 ) are summarized in Table 2 .

Summary: This paper analyzes adaptive algorithms such as adagrad and AMSGrad in a finite-sum optimization problem. The proofs appear to treat this setting through online convex optimization and online-to-batch conversion. It is shown that both AdaGrad and AMSGrad improve when the individual losses are all minimized at the same point. Further, line search techniques are analyzed in conjunction with these algorithms, and empirical results show that in practice the line searches speed up convergence.