Domain-Independent Dominance of Adaptive Methods

1 INTRODUCTION . Deep network architectures are becoming increasingly complex , often containing parameters that can be grouped according to multiple functionalities , such as gating , attention , convolution , and generation . Such parameter groups should arguably be treated differently during training , as their gradient statistics might be highly distinct . Adaptive gradient methods designate parameter-wise learning rates based on gradient histories , treating such parameters groups differently and , in principle , promise to be better suited for training complex neural network architectures . Nonetheless , advances in neural architectures have not been matched by progress in adaptive gradient descent algorithms . SGD is still prevalent , in spite of the development of seemingly more sophisticated adaptive alternatives , such as RMSProp ( Dauphin et al. , 2015 ) and Adam ( Kingma & Ba , 2015 ) . Such adaptive methods have been observed to yield poor generalization compared to SGD in classification tasks ( Wilson et al. , 2017 ) , and hence have been mostly adopted for training complex models ( Vaswani et al. , 2017 ; Arjovsky et al. , 2017 ) . For relatively simple architectures , such as ResNets ( He et al. , 2016a ) and DenseNets ( Huang et al. , 2017 ) , SGD is still the dominant choice . At a theoretical level , concerns have also emerged about the current crop of adaptive methods . Recently , Reddi et al . ( 2018 ) has identified cases , even in the stochastic convex setting , where Adam ( Kingma & Ba , 2015 ) fails to converge . Modifications to Adam that provide convergence guarantees have been formulated , but have shortcomings . AMSGrad ( Reddi et al. , 2018 ) requires non-increasing learning rates , while AdamNC ( Reddi et al. , 2018 ) and AdaBound ( Luo et al. , 2019 ) require that adaptivity be gradually eliminated during training . Moreover , while most of the recently proposed variants do not provide formal guarantees for non-convex problems , the few current convergence rate analyses in the literature ( Zaheer et al. , 2018 ; Chen et al. , 2019 ) do not match SGD ’ s . Section 3 fully details the convergence rates of the most popular Adam variants , along with their shortcomings . Our contribution is marked improvements to adaptive optimizers , from both theoretical and practical perspectives . At the theoretical level , we focus on convergence guarantees , deriving new algorithms : • Delayed Adam . Inspired by Zaheer et al . ( 2018 ) ’ s analysis of Adam , Section 4 proposes a simple modification for adaptive gradient methods which yields a provable convergence rate ofO ( 1/ √ T ) in the stochastic non-convex setting – the same as SGD . Our modification can be implemented by swapping two lines of code and preserves adaptivity without incurring extra memory costs . To illustrate these results , we present a non-convex problem where Adam fails to converge to a stationary point , while Delayed Adam – Adam with our proposed modification – provably converges with a rate of O ( 1/ √ T ) . • AvaGrad . Inspecting the convergence rate of Delayed Adam , we show that it would improve with an adaptive global learning rate , which self-regulates based on global statistics of the gradient second moments . Following this insight , Section 5 proposes a new adaptive method , AvaGrad , whose hyperparameters decouple learning rate and adaptability . Through extensive experiments , Section 6 demonstrates that AvaGrad is not merely a theoretical exercise . AvaGrad performs as well as both SGD and Adam in their respectively favored usage scenarios . Along this experimental journey , we happen to disprove some conventional wisdom , finding adaptive optimizers , including Adam , to be superior to SGD for training CNNs . The caveat is that , excepting AvaGrad , these methods are sensitive to hyperparameter values . AvaGrad is a uniquely attractive adaptive optimizer , yielding near best results over a wide range of hyperparameters . 2 PRELIMINARIES . 2.1 NOTATION . For vectors a = [ a1 , a2 , . . . ] , b = [ b1 , b2 , . . . ] ∈ Rd , we use the following notation : 1a for elementwise division ( 1a = [ 1 a1 , 1a2 , . . . ] ) , √ a for element-wise square root ( √ a = [ √ a1 , √ a2 , . . . ] ) , a+ b for element-wise addition ( a+ b = [ a1 + b1 , a2 + b2 , . . . ] ) , a b for element-wise multiplication ( a b = [ a1b1 , a2b2 , . . . ] ) . Moreover , ‖a‖ is used to denote the ` 2-norm : other norms will be specified whenever used ( e.g. , ‖a‖∞ ) . For subscripts and vector indexing , we adopt the following convention : the subscript t is used to denote an object related to the t-th iteration of an algorithm ( e.g. , wt ∈ Rd denotes the iterate at time step t ) ; the subscript i is used for indexing : wi ∈ R denotes the i-th coordinate of w ∈ Rd . When used together , t precedes i : wt , i ∈ R denotes the i-th coordinate of wt ∈ Rd . 2.2 STOCHASTIC NON-CONVEX OPTIMIZATION . In the stochastic non-convex setting , we are concerned with the optimization problem : min w∈Rd f ( w ) = Es∼D [ fs ( w ) ] ( 1 ) whereD is a probability distribution over a set S of “ data points ” . We also assume that f isM -smooth in w , as is typically done in non-convex optimization : ∀ w , w′ f ( w′ ) ≤ f ( w ) + 〈∇f ( w ) , w′ − w〉+ M 2 ‖w − w′‖2 ( 2 ) Methods for stochastic non-convex optimization are evaluated in terms of number of iterations or gradient evaluations required to achieve small loss gradients . This differs from the stochastic convex setting where convergence is measured w.r.t . suboptimality f ( w ) − minw∈Rd f ( w ) . We assume that the algorithm takes a sequence of data points S = ( s1 , . . . , sT ) from which it deterministically computes a sequence of parameter settingsw1 , . . . , wT together with a distributionP over { 1 , . . . , T } . We say an algorithm has a convergence rate of O ( g ( T ) ) if E S∼DT t∼P ( t|S ) [ ‖∇f ( wt ) ‖2 ] ≤ O ( g ( T ) ) where , as defined above , f ( w ) = Es∼D [ fs ( w ) ] . We also assume that the functions fs have bounded gradients : there exists some G∞ such that ‖∇fs ( w ) ‖∞ ≤ G∞ for all s ∈ S and w ∈ Rd . Throughout the paper , we also let G2 denote an upper bound on ‖∇fs ( w ) ‖ . 3 RELATED WORK . Here we present a brief overview of optimization methods commonly used for training neural networks , along with their convergence rate guarantees for stochastic smooth non-convex problems . We consider methods which , at each iteration t , receive or compute a gradient estimate : gt : = ∇fst ( wt ) , st ∼ D ( 3 ) and perform an update of the form : wt+1 = wt − αt · ηt mt ( 4 ) where αt ∈ R is the global learning rate , ηt ∈ Rd are the parameter-wise learning rates , and mt ∈ Rd is the update direction , typically defined as : mt = β1 , tmt−1 + ( 1− β1 , t ) gt and m0 = 0 . ( 5 ) Non-momentum methods such as SGD , AdaGrad , and RMSProp ( Dauphin et al. , 2015 ; Duchi et al. , 2011 ) have mt = gt ( i.e. , β1 , t = 0 ) , while momentum SGD and Adam ( Kingma & Ba , 2015 ) have β1 , t ∈ ( 0 , 1 ) . Note that while αt can always be absorbed into ηt , representing the update in this form will be convenient throughout the paper . SGD uses the same learning rate for all parameters , i.e. , ηt = ~1 . Although SGD is simple and offers no adaptation , it has a convergence rate of O ( 1/ √ T ) with either constant , increasing , or decreasing learning rates ( Ghadimi & Lan , 2013 ) , and is widely used when training deep networks , especially CNNs ( He et al. , 2016a ; Huang et al. , 2017 ) . At the heart of its convergence proof is the fact that Est [ αt · ηt gt ] = αt · ∇f ( wt ) . Popular adaptive methods such as RMSProp ( Dauphin et al. , 2015 ) , AdaGrad ( Duchi et al. , 2011 ) , and Adam ( Kingma & Ba , 2015 ) have ηt = 1√vt+ , where vt ∈ R d is given by : vt = β2 , tvt−1 + ( 1− β2 , t ) g2t and v0 = 0 . ( 6 ) As vt is an estimate of the second moments of the gradients , the optimizer designates smaller learning rates for parameters with larger uncertainty in their stochastic gradients . However , in this setting ηt and st are no longer independent , hence Est [ αt · ηt gt ] 6= αt · Est [ ηt ] ∇f ( wt ) . This “ bias ” can cause RMSProp and Adam to present convergence issues , even in the stochastic convex setting ( Reddi et al. , 2018 ) . Recently , Zaheer et al . ( 2018 ) showed that , with a constant learning rate , RMSProp and Adam have a convergence rate of O ( σ2 + 1/T ) , where σ2 = supw∈Rd Es∼D [ ‖∇fs ( w ) −∇f ( w ) ‖2 ] , hence their result does not generally guarantee convergence . Chen et al . ( 2019 ) showed that AdaGrad and AMSGrad enjoy a convergence rate of O ( log T/ √ T ) when a decaying learning rate is used . Note that both methods constrain ηt in some form , the former with β2 , t = 1− 1/t ( adaptability diminishes with t ) , and the latter explicitly enforces vt ≥ vj for all j < t ( ηt is point-wise non-increasing ) . In both cases , the method is less adaptive than Adam , and yet analyses so far have not delivered a convergence rate that matches SGD ’ s . 4 SGD-LIKE CONVERGENCE WITHOUT CONSTRAINED RATES . Algorithm 1 DELAYED ADAM Input : w1 ∈ Rd , αt , > 0 , β1 , t , β2 , t ∈ [ 0 , 1 ) 1 : Set m0 = 0 , v0 = 0 2 : for t = 1 to T do 3 : Draw st ∼ D 4 : Compute gt = ∇fst ( wt ) 5 : mt = β1 , tmt−1 + ( 1− β1 , t ) gt 6 : ηt = 1√ vt−1+ 7 : wt+1 = wt − αt · ηt mt 8 : vt = β2 , tvt−1 + ( 1− β2 , t ) g2t 9 : end for We first take a step back to note the following : to show that Adam might not converge in the stochastic convex setting , Reddi et al . ( 2018 ) provide a stochastic linear problem where Adam fails to converge w.r.t . suboptimality . Since non-convex optimization is evaluated w.r.t . norm of the gradients , a different instance is required to characterize Adam ’ s behavior in this setting . The following result shows that even for a quadratic problem , Adam indeed does not converge to a stationary point : Theorem 1 . For any ≥ 0 and constant β2 , t = β2 ∈ [ 0 , 1 ) , there is a stochastic convex optimization problem for which Adam does not converge to a stationary point . Proof . The full proof is given in Appendix A . The argument follows closely from Reddi et al . ( 2018 ) , where we explicitly present a stochastic optimization problem : min w∈ [ 0,1 ] f ( w ) : = Es∼D [ fs ( w ) ] fs ( w ) = { C w 2 2 , with probability p : = 1+δ C+1 −w , otherwise ( 7 ) We show that , for large enough C ( as a function of δ , , β2 ) , Adam will move towards w = 1 where ∇f ( 1 ) = δ , and that the constraint w ∈ [ 0 , 1 ] does not make w = 1 a stationary point . This result , like the one in Reddi et al . ( 2018 ) , relies on the fact that ηt and st are correlated : upon a draw of the rare sample C w 2 2 , the learning rate ηt decreases significantly and Adam takes a small step in the correct direction . On the other hand , a sequence of common samples increases ηt and Adam moves faster towards w = 1 . Instead of enforcing ηt to be point-wise non-increasing in t ( Reddi et al. , 2018 ) , which forces the optimizer to take small steps even for a long sequence of common samples , we propose to simply have ηt be independent of st. As an extra motivation for this approach , note that successful proof strategies ( Zaheer et al. , 2018 ) to analyzing adaptive methods include the following step : Est [ ηt gt ] = Est [ ( ηt−1 + ηt − ηt−1 ) gt ] = ηt−1 ∇f ( wt ) + Est [ ( ηt − ηt−1 ) gt ] ( 8 ) where bounding Est [ ( ηt − ηt−1 ) gt ] , seen as a form of bias , is a key part of recent convergence analyses . Replacing ηt by ηt−1 in the update equation of Adam removes this bias and can be implemented by simply swapping lines of code ( updating η after w ) , yielding a simple convergence analysis without hindering the adaptability of the method in any way . Algorithm 1 provides pseudocode when applying this modification , highlighted in red , to Adam , yielding Delayed Adam . The following Theorem shows that this modification is enough to guarantee a SGD-like convergence rate of O ( 1/ √ T ) in the stochastic non-convex setting for general adaptive gradient methods . Theorem 2 . Consider any optimization method which updates parameters as follows : wt+1 = wt − αt · ηt gt ( 9 ) where gt : = ∇fst ( wt ) , st ∼ D , and αt , ηt are independent of st . Assume that f ( w1 ) − f ( w ? ) ≤ D , f ( w ) = Es∼D [ fs ( w ) ] is M -smooth , and ‖∇fs ( w ) ‖∞ ≤ G∞ for all s ∈ S , w ∈ Rd . Moreover , let Z = ∑T t=1 αt mini ηt , i . For αt = γt √ 2D TMG2∞ , if p ( Z|st ) = p ( Z ) for all st ∈ S , then : E S∼DT t∼P ( t|S ) [ ‖∇f ( wt ) ‖2 ] ≤ √ MDG2∞ 2T · ES∼DT [ ∑T t=1 1 + γ 2 t ‖ηt‖ 2∑T t=1 γt mini ηt , i ] ( 10 ) where P assigns probabilities p ( t ) ∝ αt ·mini ηt , i . Proof . The full proof is given in Appendix B , along with analysis for the case with momentum β1 , t ∈ ( 0 , 1 ) in Appendix B.1 , and in particular β1 , t = β1/ √ t , which yields a similar rate . The convergence rate depends on ‖ηt‖ and mini ηt , i , which are random variables for Adam-like algorithms . However , if there are constants H and L such that 0 < L ≤ ηt , i ≤ H < ∞ for all i and t , then a rate of O ( 1/ √ T ) is guaranteed . This is the case for Delayed Adam , where 1/ ( G2 + ) ≤ ηt , i ≤ 1/ for all t and i. Theorem 2 also requires that αt and ηt are independent of st , which can be assured to hold by applying a “ delay ” to their respective computations , if necessary ( i.e. , replacing ηt by ηt−1 , as in Delayed Adam ) . Additionally , the assumption that p ( Z|st ) = p ( Z ) , meaning that a single sample should not affect the distribution of Z = ∑T t=1 αt mini ηt , i , is required since P is conditioned on the samples S ( unlike in standard analysis , where Z = ∑T t=1 αt and αt is deterministic ) , and is expected to hold as T →∞ . Practitioners typically use the last iterate wT or perform early-stopping : in this case , whether the assumption holds or not does not affect the behavior of the algorithm . Nonetheless , we also show in Appendix B.2 a similar rate that does not require this assumption to hold , which also yields a O ( 1/ √ T ) convergence rate taken that the parameter-wise learning rates are bounded from above and below .

Summary: This paper proposes a new adaptive method, which is called AvaGrad. The authors first show that Adam may not converge to a stationary point for a stochastic convex optimization in Theorem1, which is closely related to [1]. They then show that by simply making $eta_t$ to be independent of the sample $s_t$, Adam is able to converge just like SGD in Theorem2. Theorem2 follows the standard SGD techniques. Next, they propose AVAGRAD, which is based on the idea of getting rid of the effect of $\epsilon$.