Revisiting the Generalization of Adaptive Gradient Methods
Authors: Naman Agarwal
Rohan Anil
Elad Hazan
Tomer Koren
Cyril Zhang
Venue:  (2020)
Keywords: Adaptive Methods
AdaGrad
Generalization

Abstract: A commonplace belief in the machine learning community is that using adaptive gradient methods hurts generalization. We re-examine this belief both theoretically and experimentally, in light of insights and trends from recent years.
We revisit some previous oft-cited experiments and theoretical accounts in more depth, and provide a new set of experiments in larger-scale, state-of-the-art settings. We conclude that with proper tuning, the improved training performance of adaptive optimizers does not in general carry an overfitting penalty, especially in contemporary deep learning. Finally, we synthesize a ``user's guide'' to adaptive optimizers, including some proposed modifications to AdaGrad to mitigate some of its empirical shortcomings.

Under review as a conference paper at ICLR 2020
REVISITING THE GENERALIZATION OF ADAPTIVE GRADIENT METHODS
Anonymous authors Paper under double-blind review
ABSTRACT
A common belief in the machine learning community is that using adaptive gra- dient methods hurts generalization. We re-examine this belief both theoretically and experimentally, in light of insights and trends from recent years. We revisit some previous oft-cited experiments and theoretical accounts in more depth, and provide new set experiments in larger-scale, Wtateoffthe-aa settings. We con- clude that with proper tuning, the improved training performance of adaptive opti- mizers does not in general carry an overfitting penalty, especially in contemporary deep learning. Finally, we synthesize a "user's 's guide" to adaptive optimizers, in- cluding some proposed modifications to AdaGrad to mitigate some of empirical shortcomings.
INTRODUCTION
Adaptive gradient methods have remained a cornerstone of optimization for deep learning. They revolve around a simple idea: scale the step sizes according to the observed gradients along the execution. It is generally believed that these methods enjoy accelerated optimization, and are more robust hyperparameter choices. For these reasons, adaptive optimizers have been applied across diverse architectures and domains.
However, recent years, there has been renewed scrutiny on the distinction between adaptive meth- ods and "vanilla" stochastic gradient descent (SGD). Namely, several lines of work have purported that SGD, while often slower to converge, finds solutions that generalize better: for same op- timization enror (training error), adaptive gradient methods will produce models with higher sta- tistical еπтог (holdout validation error). This claim, which can be shown to be true in a convex overparameterized examples. has perhaps muddled the consensus between academic research and practitioners pushing the empirical state of the art. For the latter group, adaptive gradient meth- ods have largely endured this criticism, and remain invaluable instrument in the deep learning toolbox.
In this work, we revisit the generalization performance of adaptive gradient methods from an em- pirical perspective, and examine several often-overlooked factors which can have significant effect on the optimization trajectory. Addressing these factors, which doey not require trying yet another new optimizer, can often account for what appear to be performance gaps between adaptive methods and SGD. Our experiments suggest that adaptive gradient methods do necessarily incur a gen- eralization penalty: if an experiment indicates as such, there are number of potential confounding factors and simple fixes. We complete the paper with discussion of inconsistent evidence for the generalization penalty of adaptive methods, from both experimental and theoretical viewpoints.
1.1
OUR CONTRIBUTIONS
Our work investigates generalization of adaptive gradient methods, and constructively comments on the following:
The brittleness of simple experiments and simple abstractions. We attempt a replication of the experiments from Wilson et al. (2017). finding that they have not stood up to unknown hardware and software differences. We show simple theoretical settings where adaptive methods either

---

Under review as a conference paper at ICLR 2020
fail succeed dramatically, compared to SGD. Though each can shed interesting insights, neither abstraction is reflective the truth.
The perils of choosing large The innocuous initial accumulator value hyperparameter de- stroys adaptivity at parameter scales smaller than vc. This really matters in large-scale NLP; a foolproof solution to use our proposed "s 0" variant of AdaGrad.
The subtleties in conducting a proper optimizer search. The differences between Adam, Ada- Grad, and RMSprop are not fundamental: some, like AdaGrad's lack of momentum, are easily fixable. Upon disentangling these differences, and with enough tuning of the learning rate schedule, we suggest that all three are equally good candidates in optimizer search, and can match beat SGD.
1.2 RELATED WORK
Adaptive regularization was introduced along the AdaGrad algorithm in parallel in (Duchi et al, 2011 McMahan & Streeter 2010)_ A flurry of extensions, heuristics and modifications followed, most notably RMSprop (Tieleman & Hinton (012)) and Adam (Kingma & Ba (2014). Today, these papers have been cited tens of thousands of times, and the algorithms they propose appear in every deep learning framework. For an in-depth survey of the theory of adaptive regularization and its roots in online learning, see (Hazan 2016).
Upon a quick perusal of recent literature, there is plenty of evidence that adaptive methods continue be relevant in the state of the Adam in particular remains a staple in recent developments in
fields such as NLP DDellin et al. 2018 Yang et al., 2019 Liu et al. 2019), deep generative modeling
(Karras et al., 2017| Brock et al. 2018 Kingma & Dhariwal 2018), and deep reinforcement learning
(Haarooa et al. 2018). Adaptive methods have seen adoption in extremely large-scale settings, necessitating modifications to reduce memory consumption (Shazeer & Stern 2018 Anil et al.)
2019 Chen et al.] 2019).
In recent years, there have been various works attempting to quantify the generalization properties of SGD. These varied perspectives include general analyses based on stability and early stopping (Hardt et al. (2015), a characterization of the implicit bias in special separable cases (Gunasekar al. [2018a b). and more fine-grat fininnr analysis for neural networks exploiting their specific struc- ture (Allen Zhu & Li 2019, Arora et al., 2019). More recently, there has been growing interest towards understanding the interpolation regime for overparameterized function fitting, where SGD is often the basic object of analysis (Belkin et al. 20199) Mei & Montanari 2019).
Finally, empirical questions the generalization of adaptive gradient methods were brought the forefront by Willson et al. (2017). who exhibit empirical and theoretical situations where adaptive methods generalize poorly. Building on this premise, Keskar & Socher (2017) suggest switching from Adam and SGD during training. (Smith & Topin (2019) develop doctrine of "supercon- vergence" which eschews adaptive methods. Reddi et al. (2018) point out some pathological set- tings where Adam fails to converge, and amends the algorithm accordingly. [Schneider et al. (2019) note some sociological problems leading to misleading research on optimizer selection, providing benchmarking suite for fairer hyperparameter searches, with mixed preliminary conclusions.
2
BACKGROUND
We begin by reviewing the stochastic optimization setting, and giving rigorous definitions of the adaptive gradient methods commonly used in practice.
2.1
OPTIMIZATION SETTING
We will focus on stochastic optimization tasks of the form
minimize F(w) def E.-plf(w; 2)],
where the expectation is over a random variable whose distribution D is initially unknown; in ma- chine learning, often represents pair (ar, y) of an example x and its corresponding label y, drawn
2

---

Under review as a conference paper at ICLR 2020
from unknown population. A stochastic optimization algorithm is given sample .1...... D from the underlying distribution, and produces a point t E R whose population loss ( as close possible to that of the minimizer w* arg min,. F(w). Often, iterative (first-order) opti- mization methods maintain a sequence of iterates w1,---a Wer and, at each step I, use the stochastic gradient
g: Vf 2t)
to form the next iterate Wa+1- The simplest stochastic optimization method is Stochastic Gradient Descent (SGD), whose update rule at step takes the form
W + Wi 7p 9t.
where > 0 is step size (or learning rute) parameter, whose scale and time-varying behavior are typically determined via hyperparameter search.
2.2 ADAPTIVE GRADIENT METHODS
Adaptive gradient methods comprise general family of iterative optimization algorithms which at- tempt to automatically adapt to anisotropic gradient and parameter sizes. Often, an adaptive method will incorporate different (adaptively computed) step size for each entry of the gradient vector. More specifically, the most common adaptive methods divide each parameter's gradient update by estimate of the scale of its historical gradients. concise way unify this family of adaptive methods is given by following update equation (starting from an arbitrary initializer Wg): W+++1 < we ar H, + 3.H,
( H
w(-1).
(1)
The above update expresses broad family of methods including SGD, momentum (i.e., Polyak's heavy-ball method), AdaGrad, RMSprop, and Adam. The particular instantiations of the parameters akk 3k are summarized below:
SGD
HB
AdaGrad
RMSprop
Adam
G2
1
G + De
32G + (1 D₂)
Gr
+
Dr
35
0gk
a
a
a
a
81 8
1
8x
0
0
0
Table 1: Parameter settings for common optimization algorithms in the unified framework of Equa-
Here, D, def diag(s: (2)) and G2 def H here, @2 denotess the entrywise square of vector or
tion
matrix.
We omit the parameters in the adaptive methods, the discussion in Section 3.1
3 PRACTICAL ASPECTS OF ADAPTIVE OPTIMIZATION
In this section, we compile some lesser-known practices in the usage of adaptive methods, which we have found to help consistently across large-scale experiments. We emphasize that this collection is restricted to simple ideas, which do not add extraneous hyperparameters or algorithmic alternatives.
3.1 THE HYPERPARAMETER
The general AdaGrad update, as originally proposed by Duchi et to allow for convenient inversions. Formally, the update looks like
2011). includes a parameter
1/2
W++1 = Wi - 1.H g6 H,
diag (g:
02)
The inclusion of e the the original proposal of the above updates seems to have been made for convenience of presenting the theoretical results. However, in practice, this parameter often turns
Adam handles the parameter slightly differently; Kingma & Ba 2014

---

Under review as a conference paper at ICLR 2020
145
1.4
104
107
10
0.6
107
107
5.0
Figure 1: Distribution of nonzero accumulator values --11= - 9- 1 in the embedding layer of the Transformer from Section 4.1 after t = 116200 training iterations. Left: Histogram of accumulator values, with counts linear scale. Right: Same plot left, except with logarithmic vertical scale, showing prevalence of extremely small values.
out to be a parameter that should be tuned depending on the problem instance. The default value the algorithm tend be quite high: e.g., in
of this parameter in the standard implementations Tensorflow (Abadi et al.) (2016) it is 0.1 which can be quite high. As large values would result in AdaGrad reducing SGD with an implicit 1/vē learning rate, and losing out all adaptive properties (RMSprop and Adam implementations also have an equivalent epsilon parameter). The effect can be seen Figure 4 which shows that along many coordinates the accumulated second moments of the gradient are very small, even in the middle of training.
At least one work (Agarwal et al., 2019) remarks that the ability to choose a large moment-based adaptive method might be a feature rather than shortcoming; the smooth interpola- tion with SGD may improve the stability more exotic optimizers. This does not appear to be the case for diagonal-matee -atrix adaptive methods, in the NLP setting investigated in this paper.
second-
Instead, we suggest removing this hyperparameter altogether and justify it in Section 4.2 and per- forming the AdaGrad update with the pseudoinverse instead of the full inverse. Then, the update is given by the following:
Wh+1 = w/ - H gt.
(2)
where At denotes the Moore-Penrose pseudoinverse of A and with the preconditioning matrices updated as before. The above means that if there is coordinate for which the gradient has been 0 thus far we make no movement in that coordinate. This fix which can similarly be applied to the full matrix version of AdaGrad, does not affect the regret guarantees of AdaGrad. We provide an analysis in the Appendix B verifying as sanity check that the standard AdaGrad regret bounds continue hold when is completely removed.
3.2 MOMENTUM
A key distinction between AdaGrad, RMSprop and Adam is as follows: AdaGrad does not include momentum, and there per-parameter learning rate which inverse of the accumulated gradient squares for that parameter. RMSprop as described in Hinton et al. (2012) uses exponential mov- ing averaging rather than straightforward accumulation that AdaGrad relies on, and Adam modifies RMSprop to add momentum for the gradients along with a bias correcctnnt factor. Note that imple- mentation RMSprop can vary based on the software library; e.g.. TensorFlow (Abadi et al., 2016) includes modification to include momentum, while Keras API (Chollet et al. (2015)) does not. We note that it is straightforward to extend AdaGrad to incorporate heavy-ball momentum, where we start with Go 0 (and from certain initialization wo) and iteratively update:
gt f 3g. 1 + (1 HT gri
W1+1 + W. nタt-
3.3 LEARNING RATE WARMUP
The original definition of the Adam optimizer (Kingma & Ba, 2014) includes a bias correction term, in which the moment estimates multiplied by the time-varying scalars -31 and
-35)) As
4

---

Under review as a conference paper at ICLR 2020
mentioned in the original paper, the bias correction can equivalently be written update to the learning rate. In the notation Table
1 32 39
0x -
As can be seen from Figure 2 for the typical values of B1 0.9 and B2 0.999, the effective mul- tiplier on the learning rate essentially resembles an external warmup applied on top of the learning rate. The general heuristic including warmup phase the beginning of training has gained sig- nificant popularity in state-of-ue saae-offthe--as aef-the-art art empirical works; see, for example, Goyal et al. (2017); Shazzer Stern (2018); Keskar et al. (2019); Radford et al. (2019) Devlin et al. (2018).
1.0
wifeccov muliglrr
0.4
500
1000
15000
2000
3000
Figure 2: The effective learning rate multiplier of Adam as function of steps for the commonly found default hyperparameters 81 0.9, 3/2 0.999 deep learning libraries.
Applying such a warm externally on Adam results in now 3 hyper-parametee (B1,. 32 and now the amount of warmup) conflating with each other, making hyper-parameter tuning difficult. Instead we suggest to complete disable this bias correction altogether and use an explicit warmup schedule in place of it. We use such schedule in all of our experiments for SGD as well as adaptive optimizers as we find that it helps consistently across language modelling experiments.
One motivation for warmup during the initial stages of training is that for adaptive updates, the squared norm of the preconditioned gradient during the initial stage is quite large compared to the scale of the parameters. For the initial steps the preconditioned gradient squared norm is proportional to the number of coordinates with non-zero gradients where the squared norm of the parameters are proportional to the number of nodes. Therefore adaptive methods are naturally forced to start with smaller learning rate. The warmup in such a case helps the learning rate to rise up while the norm of the gradients fall sharply as training proceeds.
3.4 LEARNING RATE DECAY SCHEDULE
Learning rate decay schedules are one of the hardest albeit a crucial hyperparameter of an optimizer. Stochastic gradient like algorithms, domain specific learning rate schedules have been derived over time with a of care and effort, examples include Resnet-50 on ImageNet-2012 where state of the art configuration of SGD++Momentum follows stair-case learning rate schedule (while other type of schedules maybe possible). Adaptive algorithms apriori come with potential promise of not requiring massive tuning of these schedules they come with in-built schedule with the caveat that AdaGrad variants like Adam or RMSprop does not enjoy a data-dependent decay like AdaGrad due to the presence of non-zero decay factor and requires external learning rate decay schedule. Even for experiments in Kingma Ba (2014) which introduces Adam optimizer has experiments to include 1/ 1/VT decay schedule for convergence. our experiments, we found this implicit decay of AdaGrad to be sufficient for achieving superior performance training machine translation model, while an external decay rate was necessary for training the Resnet-50 on 2022 high accuracy.
EXPERIMENTS
We study the empirical performance of various optimization methods for training large state-of-the- art deep models, focusing on two domains: natural language processing (machine translation) and image recognition.
5

---

Under review as a conference paper at ICLR 2020
H-
22
20
12
RMMSpppp
Aciam
0.0
0.0
0.0
0.5
3.2
Ir mi terations
Iritiing Eorssom
165
Figure 3: Training loss (i.e. log-perplexity) curves for several optimization methods the trans former-big model on the WMT14 en en-ffr dataset.
12
12
Itg claning
13
0.0
Figure 4: Convergence of AdaGrad with momentum in the translation setting of Section 4.1 varying only the initial accumulator value Note that using the default value 10 in the TensorFlow AdaGrad implementation might cause dramatic misconceptions about poor convergence.
4.1 MACHINE TRANSLATION WITH TRANSFORMER
We study the convergence of various optimization methods when training a Transformer model Vaswani al. (017) for machine translation. We used the larger Transformer-Bi architecture Chen et al. 2018): this architecture has layers in the encoder and decoder, with 1024 model dimensions, 8192 hidden dimensions, and 16 attention heads.
trained on the WMT' 14 English to French dataset (henceforth "en-tfr") that contains 36.3M sentence pairs. All experiments were carried out on 32 cores of a TPU-v3 Pod (Jouppi al., (207)) and makes use of the Lingvo (Shen) et al. 2019 sequence-tn-sequencc TensorFlow library.
We compared several optimization methods for training; the results are reported in Fig. 3 We see that properly tuned AdaGrad (with 0 and added momentum) outperforms Adam, while SGD with momentum, plain AdaGrad and RMSprop perform much worse on this task. These results illustrate that adaptivity and momentum are both extremely effective in training these models.
4.2 SENSITIVITY TO
In Section 3.1 we proposed an "s 0" variant of AdaGrad. Here we empirically motivate this modification, by investigating the effect of the parameter on the performance of AdaGrad. We train the Transformer model from above on the en->fr dataset using AdaGrad while varying the value e. The results are given in Fig. We see drastic improvement in convergence as we lower the value of e down to 10 (lower values do not improve convergence further and are thus omitted from the figure).
To see where these dramatic improvements come from, also visualize in ?? the histogram of the square gradient values for the embedding layer the model at step f = 116200, which indicates that large fraction of the cumulative gradient entries have extremely small magnitudes. The choice of e is thus important, and justify our prescription of removing the dependency all-together instead of tuning it separate hyper-parameter.

---

Under review as a conference paper at ICLR 2020
4.3 IMAGE CLASSIFICATION WITH RESNET-50
Next, we trained a ResNet-50 architecture (He et al. 2015) on the Imagenet-2012 (Deng et al. 2009) dataset. The task is to classify images as belonging to one of the 1000 classes. Our training setup consists of 512 cores of TPU v3 Pod and makes use of a relatively large batch size of 16386. As baseline, we considered SGD with momentum with highly-tuned staircase learning rate schedule, that achieves 75.3% test accuracy after 90 epochs. We compared several optimization methods on this task as seen in Fig. 5 the straightforward application of AdaGrad (with fixed and with heavy ball momentum) achieves only paltry 63.94% test accuracy. Noticing that AdaGrad implicit decay schedule does not decay sufficiently fast, an external decay rate was added starting at epoch 50 of the form (1 current spoch - y2 This change was sufficient for AdaGrad to reach a test accuracy of
50
74.76% drastic ~~ 10% jump. As demonstrated, learning rate schedule is a highly important
hyperparameter and requires tuning for each task. E.g., the baseline SGD is highly tuned and follows an elaborate stair case learning rate reach 75% test accuracy.
50
82
70
12
20
-2
HB
Ajaörad
10
10
ASaGraondecay
Adam
20
65
302
epoch
sport
Figure 5: Training curves for top-1 training and test accuracy of ResNet-50 ImageNet, trained with several optimization methods. Without the cyan curve, the plots suggest poor generalization of adaptive methods, despite their head start in optimization. However, the gap is closed upon a careful tuning of AdaGrad with momentum.
4.4 REVISITING THE EXPERIMENTS OF WILSON ET AL. (201))
We attempted to reproduce the experiments from Wilson et al. (217)). using the same codebases and identical hyperparameter settings. Although we were able replicate some of their findings on these smaller-scale experiments, others appear to be sensitive to hyperparameter tuning, and perhaps subtle changes the deep learning software and hardware stack that have occurred during the two years since the publication of that paper. In this section, we summarize these findings.
Image classification. On the classic benchmark task of CIFAR-10 classification with a VGG net- work (Simonyan & Zisserman 2014), we were able to replicate the (Wilson et al. 2017) results perfectly, using the same codebase We repeated the hyperparameter search reported in the paper, found the same optimal base learning rates for each optimizer, and found the same stratification in performance between non-adaptive methods, Adam & RMSprop, and AdaGrad.
Character- language modeling. Curiously, our replication of the language modeling experi- ment using the same popular repository was successful in reproducing the optimal hyperparameter settings, but resulted in an opposite conclusion. Here, SGD found the objective with the smallest training loss, but Adam exhibited the best generalization performance. We believe that software ver- sion discrepancies (our setup: CUDA 10.1, cuDNN 7. 7.5.1) may account for these small differences.
Generative parsing. We turn to the Penn Treebank (arrcus et al. 1994) constituency parsing code accompanying (Choe & Charniak] 2016). Using the same architectural and training protocol modifications as specified in (Wilson et al. (017), we were able to get the model to converge with
https://gitthbb.comm///////1//11

---

Under review as a conference paper at ICLR 2020
each optimizer. However, for two of the settings (SGD and RMSprop), the best reported learning rates exhibited non convergencc (the fainter curves in Figure 6 Similarly as the above experiment, the ranking of optimizers' training and generalization performance differs from that seen in the original report.
Finally, Wilson al. (017) include fourth set of experiments, generative parsing of Penn Tree- bank, using the coda accompanying (Cross & Huang 2016). Unfortunately, this DyNet (Neubig at. 2017) implementation, which was last updated in 2016, encountered fatal memory leak when trainin with our DyNet 2.1 setup.
All relevant plots given in F with color codes selected to match Figures 1 and in Wilson cct al. (2017). Together, these experiments are further evidence for a widespread reproducibility crisis in deep learning: despite the authors' exceptional transparency in disclosing their optimizer selection and evaluation protocol, these benchmarks have turned out to be brittle for unknown reasons. Along the lines the random-seed-tunin experiments of Henderson al. (2018), this suggests that there are further technical complications to the problems of credible optimizer evaluation addressed by Schnnider (2019) even on well-known supervised learning benchmarks.
3GD
27.5
AdaCrol
25.0
20
Adum
RMSSarop
22.5
20.0
200
50
132
3.4
3.400
55D
3.375
1.3
(ASTA No and harod
1,350
RNSorop
1.325
1.250
3.0
1.225
0.0
1.200
spech
24-
24-
1B
5GD
(1STH OPOIeR pare
AddOrol
AM5omp
nal
Admm
60
130
20
40
100
4poch
spoch
Figure 6: Attempted replication of the results of Wilson et al. (2117). Top: Image classification with a CNN. Results match perfectly, with non-adaptive methods generalizing the best. Middle: Character-level language modeling with 2-layer LSTM. The original reported hyperparameters are the best and all optimizers converge to reasonable solutions, but contradictory conclusions about generalization arise. Bottom: 3-layer LSTM for generative parsing. Training does not converge with all reported learning rates; conclusions about generalization are unclear.
httpss://iithbb.com///eeees////
8

---

Under review as a conference paper at ICLR 2020
5 THEORETICAL DISCUSSION
In this section we provide two simple examples of stochastic convex problems where it can be seen that when comes to generalization both AdaGrad and SGD can be significantly better than the other depending on the instance. Our purpose to provide both the examples is to stress our point that the issue of understanding the generalization performance of SGD vs. adaptive methods is more nuanced than what simple examples might suggest and hence such examples should be treated as qualitative indicators more for the purpose of providing intuition. Indeed which algorithm will perform better on given problem, depends on various properties of the precise instance.
5.1 EXAMPLE WHERE SGD ADAGRAD
We provide brief intuitive review of the construction provided by Wilson et al. (2017); ); for precise description, see Section 3.3 of that paper. Consider setting of overparameterized linear regression, where the true output (i.e. dependent variable)
{+1) } is the first coordinate of the feature vector (independent variable) æ. The next two coordinates of x are "dummy" coordinates set to then, the coordinates are arranged in blocks which only appear once per sample, taking the value of
The key idea is that in this setting, the solution space that AdaGrad explores is always in the subspace of the sign vector of X y. As result, AdaGrad treats the first three coordinates essentially indis- tinguishably putting equal mass on each. It can then be seen that for any new example the AdaGrad solution does not extract the true label information from the first three coordinates and hence gets the prediction wrong, leading to high generalization error; the other distinguishing features belong to the new unique block which are set to 0 for the AdaGrad solution, as it has not seen them.
5.1.1 EXAMPLE WHERE ADAGRAD > SGD
This example is motivated from the original AdaGrad paper (Duchi al. 2011), adapted to the overparameterizee setting. Consider a distribution Z supported over {0. 1 with equal 1/d mass over vectors with exactly 1 and 0 mass everywhere else. Let the label distribution be always y
1. Consider sampling dataset S of size e where < 1 (corresponding to the overparameterized setting) and consider the hinge loss
[((x) = -1- [1 v=(x+x+)]+ wx)]+
where (ze, yt) denotes the t-th (example, label) pair. Note that there is an optimal predictor given by the all-ones vector.
Running AdaGrad in such a setting, it can be seen that the first time vector that has not appeared yet is sampled, AdaGrad quickly adapts by setting the coordinate corresponding to the vector to and thereby making 0 error on the example. Therefore after one epoch of AdaGrad (ed steps), the training error reduces to 0 and the average test error becomes roughly (1 c). On the other hand, for SGD (with an optimal 1/ l/ decay scheme) after say cd/2 steps, the learning rate reduces to at most O(1/Vd) and therefore in the next cd/2 steps SGD reduces the error at most by factor of O(1 - 1 I leading to a total test error of at least (1 c/2) after total of ed steps. This is significantly smaller than the error achieved by AdaGrad at this stage. Further note that to get down
to the same test error as that achieved by AdaGrad, it can be seen that SGD requires at least 2? dd) times more steps than AdaGrad.
CONCLUSION
We have presented an empirical retrospective on adaptive gradient methods, along with several prac- tical algorithmic insights, concluding that the adaptive gradient methods do not in general carry generalization penalty. In no way do we purport to settle the debate about optimizer selection; at the very least, this would require massive hyperparameter search, which would need to be up- dated as trends evolve and new architectures and benchmarks emerge. Instead, our experiments aim to challenge some possible misconceptions, while pointing out the simple algorithmic suggestions they imply. In doing so, we hope to contribute to greater clarity on the messy frontier of optimizer selection and tuning.

---

Under review as a conference paper at ICLR 2020
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: system for large- scale machine learning. In 12th USENIX Symposium on Operating Systems Design and Imple- mentation, pp. 265-283, 2016.
Naman Agarwal, Brian Bullins, Xinyi Chen, Elad Hazan, Karan Singh, Cyril Zhang, and Yi Zhang. Efficient full- matrix adaptive regularization. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings the 36th International Conference Proceedings of Machine Learning Research, pp. 102-110, Long Beach, California, USA, 09-15 Jun 2019. PMLR. URL ptttt:::/ppocceednnge. ml r. press /ttssss/v////aaa v97/agarwal 9b. html
Machine Learning, volume 97 of
Zeyuan Allen-Zhe and Yuanzhi Li. Can sgd learn recurrent neural networks with provable general- ization? arXiv preprint 1902.0100 2019.
Rohan Anil, Vineet Gupta, Tomer Koren, and Yoram Singer. Memory-efficient adaptive optimiza- tion for large-scale learning. arXiv preprint arXiv: 1901. 1150, 2019.
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterizee two-layer neural networks. arXiv preprint arXiv:1901.08584, 2019.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- learning practice and the classical vari-nar trade-off. Proceedings of the National Academy of Sciences, 116(32):554491158) 15854, 2019.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1096, 2018.
Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Mike Schuster, Noam Shazeer, Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Zhifeng Chen, Yonghui Wu, and Maeduff Hughes. The best of both worlds: Combining recent advances in neural machine translation. In Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics, ACL 2018, pp. 76-86, 2018
Xinyi Chen, Naman Agarwal, Elad Hazan, Cyril Zhang, and Yi Zhang. Extreme tensoring for low-memory preconditioning. arXiv preprint arXiv: vrXiv:/902.000 1902.04620, 2019.
Do Kook Choe and Eugene Charniak. Parsing language modeling. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2331-2331 2336, 2016.
François Chollet et al. Keras.
2015.
James Cross and Liang Huang. Span-based constituency parsing with a structure-label system and provably optimal dynamic oracles. 2016.
J. Deng, W. Dong, R. Socher, L.J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. CVPR09, 2009.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805. 2018.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(1u))2121-2159, 2011.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv: wtt.:::700000277... 2017.
Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nathan Srebro. Characterizing implicit bias in terms of optimization geometry. arXiv preprint wrXiiv/802.08244. 2018
10

---

Under review as a conference paper at ICLR 2020
Suriya Gunasekar, Jason D Lee, Daniel Soudry, and Nati Srebro. Implicit bias of gradient descent on linear convolutional networks. In Advances in Neural Information Processing Systems, pp- 9461-9471, 2018b.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.01290, 2018.
Moritz Hardt, Benjamin Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. arXiv preprint 159.0:1500001240, 2015.
Elad Hazan, Introduction to online convex optimization. Foundations and Trends in Optimization, 2(3-4):157-325, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. arXiv preprint arxiv: 2015.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Thirty-Second AAAI Conference on Artificial Intel- ligence, 2018
Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent. Cited on, pp. 14, 2012.
Norman P Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al. In-datacenter performance analysis of a tensor processing unit. In Computer Architecture (ISCA), 2017 ACM/IEEE 44th Annual International Symposium on, pp. 1-12. IEEE, 2017.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. arXiv preprint 1710. 2017.
Nitish Shirish Keskar and Richard Socher. Improving generalization performance by switching from adam to sgd. arXiv preprint arXiv: 1rXiv:1712.0013 10t2.071.... 2017.
Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation, 2019.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization, arXiv preprint 1412.6900 2014.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions. In Advances in Neural Information Processing Systems, pp. 10215-10224, 2018.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprini arXiv: 1692, 2019.
Mitchell Mareus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Fer- guson, Karen Katz, and Britta Schasberger. The penn treebank: annotating predicate argument structure. In Proceedings of the workshop on Human Language Technology, pp. 114-119, Asso- ciation for Computational Linguistics, 1994.
Brendan McMahan and Matthew Streeter. Adaptive bound optimization for online convex opti- mization. arXiv preprint
2010.
Song Mei and Andrea Montanari. The generalization enror of random features regression: Precise asymptotics and double descent curve. arXiv preprint urXiv:1908.05333 2019.
Graham Neubig, Chris Dyer, Yoav Goldberg, Austin Matthews, Waleed Ammar, Antonios Anasta- sopoulos, Miguel Ballesteros, David Chiang, Daniel Clothiaux, Trevor Cohn, Kevin Duh, Manaal Faruqui, Cynthia Gan, Dan Garrette, Yangfeng Ji, Lingpeng Kong, Adhiguna Kuncoro, Gaurav Kumar, Chaitanya Malaviya, Paul Michel, Yusuke Oda, Matthew Richardson, Naomi Saphra, Swabha Swayamdipta, and Pengcheng Yin. Dynet: The dynamic neural network toolkit. arXiv preprint arXiv: 1000 009980 /70.00900 2017.
11

---

Under review as a conference paper at ICLR 2020
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. 2019.
Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of Adam and beyond. In International Conference Learning Representations, 2018.
Frank Schneider, Lukas Balles, and Philipp Hennig. Deepobs: deep learning optimizer benchmark suite. arXiv preprint MoXiv::9033.5499000 2019.
Noam Shazeer and Mitchell Stemn. Adafactor: Adaptive learning rates with sublinear memory cost. arXiv preprint 18X4v:884..444 2018.
Jonathan Shen, Patrick Nguyen, Yonghui Wu, Zhifeng Chen, et al. Lingvo: a modular and scalable framework for -equence-to-eequence modeling, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint
2014.
Leslie N Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using large learning rates. In Artificial Intelligence and Machine Learning for Multi- Domain Operations Applications, volume 11006, pp. 1100612. International Society for Optics and Photonics, 2019.
Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop: Divide the gradient by running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2):26- 31, 2012.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, pp. 5998-6008, 2017.
Ashia C Wilson, Rebecca Roelofs, Mitchell Stern, Nati Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. In Advances in Neural Information Processing Systems, pp. 4151-4161, 2017.
Zhilin Yang, Zihang Dai. Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V
Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint 17Xiv:1906.08237, 2019.
12

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1 DETAILS OF EXPERIMENTS
A.1.1 TRANSFORMER BIG ON WMT14 ENGLISH TO FRENCH
Models were trained with 384 batch size with a linear warm-up over 40k steps. We did grid search over the parameters. Ranges of these were: Learning rate 7 E [10-6, 10") Momentum B1 E (0.9, 0.99), and eecond-momen coefficient for Adam/RMSprop 32 E 0.99, 0.999.
1. AdaGrad: Following versions of AdaGrad had fixed zero. (Note indeed add of 10 10-30 such that /c is not infinity.)
(a) Vanilla: 7) = 0.01
(b) Heavy Ball Momentum: 31 = 0.9, 0.05
2. Adam: 31 = 0.9, 82 = 0.999, with maximal learning rate of 0.000075, and followed by
decay suggested in Vaswani al. (2017).
3. R.Sprop: B2 0.999, 17 0.000075
4. Momentum: 31 0.9, 17 0.08
A.2 RESNET-50 ON IMAGENET-2012
Models were trained with 16384 batch size, with L2 regularization and label smoothing coefficients set as le-4 and 0.1 respectively.
1. SGD+Momentum((IB) follows staircase learning rate schedule where learning rate is ramped up linearly from 0 to 6.4 over the first 5 epochs, followed by 10x drop learning rate at 30 epochs, 60 epochs and 80 epochs. Momentum is set to 0.9.
2. AdaGrad+Decayy We linearly increase the learning rate from 0 to 0.1 over the first 50 epochs, followed by external decay rate, that follows (1 spoch since 50 2 AdaGrad here refers to the epsilon fixed version with heavy-ball momentum coefficient set to 0.9.
3. Adam: Decay rate follows the same schedule as AdaGrad decay, however, uses maximal learning rate 0.001.
4. AdaGrad: without decay linearly increases the learning rate from 0.1 over the first 20 epochs. AdaGrad here refers to the epsilon fixed version with heavy-ball momentum.
B ANALYSIS OF ADAGRAD WITHOUT
In Section 3.1 provide method for setting the to O. The general update is performed as follows
Wr+1 = w; n High G = G. + 9:9T
where the preconditioning matrix H, is updated in two possible ways:
diagonal: H, def
(aag(G))')
full: H. def 2/2
The following theorem whose proof is very similar to the original analysis in Duchi et al. (201 shows that the above modification leads change in the regret guarantee of AdaGrad.
Theorem 1. The regret of the AdaGrad algorithm with updates implemented as bounded as
(full) Regret O max lw; w"lle tr(G(/3
T
(diagonal)
Regret < 0
max |/w; w0 11o0
Σ
Σ
gh.
t
1=1
t=1
13

---

Under review as a conference paper at ICLR 2020
We now provide quick proof sketch for the above highlighting the main parts of the proof that change from the standard version. As in the original proof we consider the case of linear loss functions at every step given by Jt. The first step is to note that the following relationship holds directly by the definition of the updates.
T gt (x. x²)
(1 xx Pi ||:211 x= (i, "(. H. H,
2rj
The analysis now follows in the standard way by summing the above over time and analyzing the first and second summation separately. The first term the same as the standard analysis and therefore leads to no change.
Further more for the second term the idea of the original proof is to show that
Eli H. s 2tr(HT))
The above statement follows in the same way the original proof with care for pseudoinverses. For instance in the diagonal version we can apply Lemma 4 from Duchi et al.] (201T) along each coordinate separately applying the lemma from the first time the coordinate sees non-7ero gradient and ignoring everything before is O.
14


Reviews:
[{"review_id": "HklEDFu3YB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper does an empirical study for the problem of \"whether acceleration harms the empirical learning performance\". Based on the study, the authors propose some empirical suggestions to fix the generalization gap for acceleration methods.\n\nOverall, this paper is a lack of insights and looks more like an experimental report. The significance of the paper is not enough. Please see questions below\n\nQ1. \"We re-examine this belief both theoretically and experimentally\". Where is the theoretical part? \n- I wish the authors can offer some useful Lemmas or Theorems, but Section 5 only offers some discussions. \n\nQ2. Since the paper wants to promote some empirical observations, please add STD to all curves, like,  Wilson et al. (2017). Some curves are too close (e.g., those Figure 6), it is hard to judge their statistic significance.\n\nQ3. \"we synthesize a user’s guide to adaptive optimizers\". \n- After reading the paper several times, I am still kind of confused. What is the \"user’s guide\"? Could the authors make some short summary?\n- Since the paper mainly does empirical study, this question is important.\n\nQ4. Why not SGD in Figure 3?\n\nQ5. \"As demonstrated, learning rate schedule is a highly important hyperparameter and requires tuning for each task\". \n- So, how to tune the learning rate?\n- Is the learning rate more important than the choice of the optimizer? If so, the problem with the acceleration this paper investigated has little meaning.\n\nQ6. Is it better to carry on hyperparameter optimization on factors that can contribute to the final performance of each algorithm? \n- In this way, we can see the extreme performance of each optimizer, the performance comparison condition on best possible hyperparameter can make more sense than what has been done in this paper.\n- It will be good if authors can check this paper \"Neural Optimizer Search with Reinforcement Learning\" and then re-design their methodology.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BylHymY19r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper revisits the question of whether adaptive methods deliver solutions with larger generalization errors than those of solutions found using SGD. The conclusion is that \"which algorithm will perform better on a given problem, depends on various properties of the precise instance.\" They reach this conclusion by rerunning the same experiments with the same settings as in the 2017 Wilson et al. paper which showed empirical evidence that adaptive methods exhibit worse generalization error than SGD, and showing that the results have changed, due to extraneous factors like the fact that they run on different hardware. The only advantage of adaptive methods seems to be that the amount of tuning is less intensive (e.g., no need to handtune staircase decaying learning rates). They provide two mathematical examples to show that SGD (with a particular decay rate) and Agagrad can beat the other in terms of generalization error, depending on the setting.\n\nThe takeaway of this paper seems to be that adaptive methods and non-adaptive methods both need careful parameter tuning. They also suggest using Adagrad without adding a multiple of the identity to the approximate Hessian, and point out that externally imposed decay rates can help adaptive methods. I think the paper is well-written, and provides convincing evidence that neither SGD or adaptive methods dominates the other. I lean towards accept.\n\nComments:\nFigure 5 doesn't show a significant gap between the test and training error for the adaptive methods, as claimed in its caption.\nThere is a missing figure reference at bottom of page 6", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BkxNHyW6YB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper revisited the a common belief that adaptive gradient methods hurts generalization performances. The authors re-examine this in more depth and provide a new set of experiments in larger-scale, state-of-the-art settings. The authors claimed that with proper tuning, the performance of adaptive optimizers can mitigate the gap  with non-adaptive methods.\n\n- It is great to have someone revisit and challenge the conventional ideas in the community. However, I did not find much insightful information in this paper. As the author mentioned, there are many recent works focusing on further improving the empirical generalization performances of adaptive gradient methods. The main aspects mentioned in the paper, \\epsilon tuning, learning rate warmup and decaying schedule, are not something new and many of which are mentioned or used in the recent advances. This makes the contribution of this paper look like combining all the tricks together. The authors might want to carefully comment the differences with the recent advances\n\n[1] Liu, Liyuan, et al. \"On the variance of the adaptive learning rate and beyond.\" arXiv preprint arXiv:1908.03265 (2019).\n[2] Loshchilov, Ilya, and Frank Hutter. \"Decoupled weight decay regularization.\" ICLR 2019.\n[3] Zaheer, Manzil, et al. \"Adaptive methods for nonconvex optimization.\" Advances in Neural Information Processing Systems. 2018.\n[4] Chen, Jinghui, and Quanquan Gu. \"Closing the generalization gap of adaptive gradient methods in training deep neural networks.\" arXiv preprint arXiv:1806.06763 (2018).\n[5] Luo, Liangchen, et al. \"Adaptive gradient methods with dynamic bound of learning rate.\" arXiv preprint arXiv:1902.09843 (2019).\n\n- In terms of tuning \\epsilon, the authors mentioned that default setting in Tensorflow is 0.1 for Adagrad which is too high. However, most papers regarding adaptive gradient method usually set \\epsilon as 10^-8. Pytorch set default value as 10^-10. In fact, Yogi paper mentioned above gives some different conclusions. In their experiments, they found that setting \\epsilon to be a bit larger like 10^-3 give better results compared with 10^-8. I wonder if the authors examine the reasons for different conclusions here?\n \n- at the end of page 6, missing reference for histogram\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper combines several recent optimizer tricks to provide empirical evidence that goes against the common belief that adaptive methods result in larger generalization errors. The contribution of this paper is rather small: no new strategies are introduced and no new theory is presented. The paper makes a good workshop paper, but does not meet the bar for publication at ICLR.

Decision: Reject