Revisiting the Generalization of Adaptive Gradient Methods

1 INTRODUCTION . Adaptive gradient methods have remained a cornerstone of optimization for deep learning . They revolve around a simple idea : scale the step sizes according to the observed gradients along the execution . It is generally believed that these methods enjoy accelerated optimization , and are more robust to hyperparameter choices . For these reasons , adaptive optimizers have been applied across diverse architectures and domains . However , in recent years , there has been renewed scrutiny on the distinction between adaptive methods and “ vanilla ” stochastic gradient descent ( SGD ) . Namely , several lines of work have purported that SGD , while often slower to converge , finds solutions that generalize better : for the same optimization error ( training error ) , adaptive gradient methods will produce models with a higher statistical error ( holdout validation error ) . This claim , which can be shown to be true in a convex overparameterized examples , has perhaps muddled the consensus between academic research and practitioners pushing the empirical state of the art . For the latter group , adaptive gradient methods have largely endured this criticism , and remain an invaluable instrument in the deep learning toolbox . In this work , we revisit the generalization performance of adaptive gradient methods from an empirical perspective , and examine several often-overlooked factors which can have a significant effect on the optimization trajectory . Addressing these factors , which does not require trying yet another new optimizer , can often account for what appear to be performance gaps between adaptive methods and SGD . Our experiments suggest that adaptive gradient methods do not necessarily incur a generalization penalty : if an experiment indicates as such , there are a number of potential confounding factors and simple fixes . We complete the paper with a discussion of inconsistent evidence for the generalization penalty of adaptive methods , from both experimental and theoretical viewpoints . 1.1 OUR CONTRIBUTIONS . Our work investigates generalization of adaptive gradient methods , and constructively comments on the following : The brittleness of simple experiments and simple abstractions . We attempt a replication of the experiments from Wilson et al . ( 2017 ) , finding that they have not stood up to unknown hardware and software differences . We show simple theoretical settings where adaptive methods can either fail or succeed dramatically , as compared to SGD . Though each can shed interesting insights , neither abstraction is reflective of the truth . The perils of choosing a large . The innocuous initial accumulator value hyperparameter destroys adaptivity at parameter scales smaller than √ . This really matters in large-scale NLP ; a foolproof solution is to use our proposed “ ε = 0 ” variant of AdaGrad . The subtleties in conducting a proper optimizer search . The differences between Adam , AdaGrad , and RMSprop are not fundamental ; some , like AdaGrad ’ s lack of momentum , are easily fixable . Upon disentangling these differences , and with enough tuning of the learning rate schedule , we suggest that all three are equally good candidates in optimizer search , and can match or beat SGD . 1.2 RELATED WORK . Adaptive regularization was introduced along the AdaGrad algorithm in parallel in ( Duchi et al. , 2011 ; McMahan & Streeter , 2010 ) . A flurry of extensions , heuristics and modifications followed , most notably RMSprop ( Tieleman & Hinton , 2012 ) and Adam ( Kingma & Ba , 2014 ) . Today , these papers have been cited tens of thousands of times , and the algorithms they propose appear in every deep learning framework . For an in-depth survey of the theory of adaptive regularization and its roots in online learning , see ( Hazan , 2016 ) . Upon a quick perusal of recent literature , there is plenty of evidence that adaptive methods continue to be relevant in the state of the art . Adam in particular remains a staple in recent developments in fields such as NLP ( Devlin et al. , 2018 ; Yang et al. , 2019 ; Liu et al. , 2019 ) , deep generative modeling ( Karras et al. , 2017 ; Brock et al. , 2018 ; Kingma & Dhariwal , 2018 ) , and deep reinforcement learning ( Haarnoja et al. , 2018 ) . Adaptive methods have seen adoption in extremely large-scale settings , necessitating modifications to reduce memory consumption ( Shazeer & Stern , 2018 ; Anil et al. , 2019 ; Chen et al. , 2019 ) . In recent years , there have been various works attempting to quantify the generalization properties of SGD . These varied perspectives include general analyses based on stability and early stopping ( Hardt et al. , 2015 ) , a characterization of the implicit bias in special separable cases ( Gunasekar et al. , 2018a ; b ) , and a more fine-grained analysis for neural networks exploiting their specific structure ( Allen-Zhu & Li , 2019 ; Arora et al. , 2019 ) . More recently , there has been a growing interest towards understanding the interpolation regime for overparameterized function fitting , where SGD is often the basic object of analysis ( Belkin et al. , 2019 ; Mei & Montanari , 2019 ) . Finally , empirical questions on the generalization of adaptive gradient methods were brought to the forefront by Wilson et al . ( 2017 ) , who exhibit empirical and theoretical situations where adaptive methods generalize poorly . Building on this premise , Keskar & Socher ( 2017 ) suggest switching from Adam and SGD during training . Smith & Topin ( 2019 ) develop a doctrine of “ superconvergence ” which eschews adaptive methods . Reddi et al . ( 2018 ) point out some pathological settings where Adam fails to converge , and amends the algorithm accordingly . Schneider et al . ( 2019 ) note some sociological problems leading to misleading research on optimizer selection , providing a benchmarking suite for fairer hyperparameter searches , with mixed preliminary conclusions . 2 BACKGROUND . We begin by reviewing the stochastic optimization setting , and giving rigorous definitions of the adaptive gradient methods commonly used in practice . 2.1 OPTIMIZATION SETTING . We will focus on stochastic optimization tasks of the form minimize F ( w ) def= Ez∼D [ f ( w ; z ) ] , where the expectation is over a random variable z whose distribution D is initially unknown ; in machine learning , z often represents a pair ( x , y ) of an example x and its corresponding label y , drawn from an unknown population . A stochastic optimization algorithm is given a sample z1 , . . . , zT ∼ D from the underlying distribution , and produces a point w̄ ∈ Rd whose population loss F ( w̄ ) is as close as possible to that of the minimizer w ? = arg minw F ( w ) . Often , iterative ( first-order ) optimization methods maintain a sequence of iterates w1 , . . . , wT and , at each step t , use the stochastic gradient gt = ∇f ( wt , zt ) to form the next iterate wt+1 . The simplest stochastic optimization method is Stochastic Gradient Descent ( SGD ) , whose update rule at step t takes the form wt+1 ← wt − ηt gt , where ηt > 0 is a step size ( or learning rate ) parameter , whose scale and time-varying behavior are typically determined via hyperparameter search . 2.2 ADAPTIVE GRADIENT METHODS . Adaptive gradient methods comprise a general family of iterative optimization algorithms which attempt to automatically adapt to anisotropic gradient and parameter sizes . Often , an adaptive method will incorporate a different ( adaptively computed ) step size for each entry of the gradient vector . More specifically , the most common adaptive methods divide each parameter ’ s gradient update by a second-moment-based estimate of the scale of its historical gradients . A concise way to unify this family of adaptive methods is given by following update equation ( starting from an arbitrary initializer w0 ) : wt+1 ← wt − αtH−1t gt + βtH−1t Ht−1 ( wt − wt−1 ) . ( 1 ) The above update expresses a broad family of methods including SGD , momentum ( i.e. , Polyak ’ s heavy-ball method ) , AdaGrad , RMSprop , and Adam . The particular instantiations of the parameters αk , βk are summarized below : tion 1 . Here , Dt def = diag ( g 2t ) and Gt def = H 2t ; here , 2 denotes the entrywise square of a vector or matrix . We omit the parameters in the adaptive methods , see the discussion in Section 3.1 . 3 PRACTICAL ASPECTS OF ADAPTIVE OPTIMIZATION . In this section , we compile some lesser-known practices in the usage of adaptive methods , which we have found to help consistently across large-scale experiments . We emphasize that this collection is restricted to simple ideas , which do not add extraneous hyperparameters or algorithmic alternatives . 3.1 THE HYPERPARAMETER . The general AdaGrad update , as originally proposed by Duchi et al . ( 2011 ) , includes a parameter to allow for convenient inversions . Formally , the update looks like:1 wt+1 = wt − η H−1t gt ; Ht = [ t∑ k=1 diag ( g 2t ) ] 1/2 + · I The inclusion of in the the original proposal of the above updates seems to have been made for convenience of presenting the theoretical results . However , in practice , this parameter often turns 1Adam handles the parameter slightly differently ; see Kingma & Ba ( 2014 ) . 30 20 10 0 10 log10 ( Gt [ i ] ) 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 co un t i n em be dd in g la ye r 1e6 30 20 10 0 10 log10 ( Gt [ i ] ) 101 102 103 104 105 106 Figure 1 : Distribution of nonzero accumulator values Gt [ i ] = ∑t τ=1 gτ [ i ] 2 in the embedding layer of the Transformer from Section 4.1 , after t = 116200 training iterations . Left : Histogram of accumulator values , with counts on a linear scale . Right : Same plot as left , except with logarithmic vertical scale , showing prevalence of extremely small values . out to be a parameter that should be tuned depending on the problem instance . The default value of this parameter in the standard implementations of the algorithm tend to be quite high ; e.g. , in Tensorflow ( Abadi et al. , 2016 ) it is 0.1 which can be quite high . As large values would result in AdaGrad reducing to SGD with an implicit 1/ √ learning rate , and losing out on all adaptive properties ( RMSprop and Adam implementations also have an equivalent epsilon parameter ) . The effect can be seen in Figure 4 which shows that along many coordinates the accumulated second moments of the gradient are very small , even in the middle of training . At least one work ( Agarwal et al. , 2019 ) remarks that the ability to choose a large ε in a secondmoment-based adaptive method might be a feature rather than a shortcoming ; the smooth interpolation with SGD may improve the stability of more exotic optimizers . This does not appear to be the case for diagonal-matrix adaptive methods , in the NLP setting investigated in this paper . Instead , we suggest removing this hyperparameter altogether and justify it in Section 4.2 , and performing the AdaGrad update with the pseudoinverse instead of the full inverse . Then , the update is given by the following : wt+1 = wt − η H†t gt , ( 2 ) where A† denotes the Moore-Penrose pseudoinverse of A and with the preconditioning matrices updated as before . The above means that if there is a coordinate for which the gradient has been 0 thus far we make no movement in that coordinate . This fix which can similarly be applied to the full matrix version of AdaGrad , does not affect the regret guarantees of AdaGrad . We provide an analysis in the Appendix B , verifying as a sanity check that the standard AdaGrad regret bounds continue to hold when ε is completely removed .

Summary: This paper revisits the question of whether adaptive methods deliver solutions with larger generalization errors than those of solutions found using SGD. The conclusion is that "which algorithm will perform better on a given problem, depends on various properties of the precise instance." They reach this conclusion by rerunning the same experiments with the same settings as in the 2017 Wilson et al. paper which showed empirical evidence that adaptive methods exhibit worse generalization error than SGD, and showing that the results have changed, due to extraneous factors like the fact that they run on different hardware. The only advantage of adaptive methods seems to be that the amount of tuning is less intensive (e.g., no need to handtune staircase decaying learning rates). They provide two mathematical examples to show that SGD (with a particular decay rate) and Agagrad can beat the other in terms of generalization error, depending on the setting.