On Empirical Comparisons of Optimizers for Deep Learning
Authors: Dami Choi
Christopher J. Shallue
Zachary Nado
Jaehoon Lee
Chris J. Maddison
George E. Dahl
Venue:  (2020)
Keywords: Deep learning
optimization
adaptive gradient methods
Adam
hyperparameter tuning

Abstract: Selecting an optimizer is a central step in the contemporary deep learning pipeline. In this paper we demonstrate the sensitivity of optimizer comparisons to the metaparameter tuning protocol. Our findings suggest that the metaparameter search space may be the single most important factor explaining the rankings obtained by recent empirical comparisons in the literature. In fact, we show that these results can be contradicted when metaparameter search spaces are changed. As tuning effort grows without bound, more general update rules should never underperform the ones they can approximate (i.e., Adam should never perform worse than momentum), but the recent attempts to compare optimizers either assume these inclusion relationships are not relevant in practice or restrict the metaparameters they tune to break the inclusions. In our experiments, we find that the inclusion relationships between optimizers matter in practice and always predict optimizer comparisons. In particular, we find that the popular adative gradient methods never underperform momentum or gradient descent. We also report practical tips around tuning rarely-tuned metaparameters of adaptive gradient methods and raise concerns about fairly benchmarking optimizers for neural network training.

Under review as a conference paper at ICLR 2020
ON EMPIRICAL COMPARISONS OF OPTIMIZERS FOR DEEP LEARNING
Anonymous authors Paper under double-hlind review
ABSTRACT
Selecting an optimizer is a central step in the contemporary deep learning pipeline. In this paper, we demonstrate the sensitivity of optimizer comparisons the hy- perparameter tuning protocol. Our findings suggest that the hyperparameter search space may the single most important factor explaining the rankings obtained by recent empirical comparisons in the literature. In fact, we show that these results can be contradicted when hyperparameter search spaces are changed. As tuning effort grows without bound, more general optimizers should never underperform the ones they can approximate (i.e., Adam should never perform worse than mo- mentum), but recent attempts compare optimizers either assume these inclusion relationships are not practically relevant or restrict the hyperparameters in ways that break the inclusions. In our experiments, find that inclusion relationships between optimizers matter in practice and always predict optimizer comparisons. In particular, we find that the popular adaptive gradient methods never underper- form momentum or gradient descent. We also report practical tips around tuning often ignored hyperparameters of adaptive gradient methods and raise concerns about fairly benchmarking optimizers for neural network training.
INTRODUCTION
The optimization algorithm chosen by deep learning practitioner determines the training speed and the final predictive performance of their model. To date, there is no theory that adequately explains how to make this choice. Instead, our community relies on empirical studies (Wilson et al. 2017) and benchmarking (Schneider et al. 2019). Indeed, it is the de facto standard that papers introducing new optimizers report extensive comparisons across a large number of workloads. Therefore, to maximize scientific progress, must have confidence in our ability to make empirical comparisons between optimization algorithms.
Although there is no theory guiding when comparing optimizers, the popular first-order optimiz- ers form a natural inclusion hicrarchy. For example, ADAM (Kingma and Bal 2015) and RMSPROP Tieleman and Hinton 2012) can approximately simulate MOMENTUM (Polyak 1964) if the term in the denominator of their parameter updates is allowed to grow very large. However, these rela- tionships may not matter in practice. For example, the settings of ADAM's hyperparameters that allow it to match the performance of MOMENTUM may be too difficult to find (for instance, they may be infinite).
In this paper, we demonstrate two important and interrelated points about empirical comparisons of neural network optimizers. First, we show that inclusion relationships between optimizers actually matter in practice; in our experiments, more general optimizers never underperform special cases. Despite conventional wisdom (Wilson et al. 2017 Balles and Hennig 2017), we find that when carefully tuned, ADAM and other adaptive gradient methods never underperform MOMENTUM or SGD. Second, we demonstrate the sensitivity of optimizer comparisons to the hyperparameter tun- ing protocol. By comparing to previous experimental evaluations, we show how easy it is to change optimizer rankings on a given workload (model and dataset pair) by changing the hyperparameter tuning protocol, with optimizer rankings stabilizing according inclusion relationships more and more effort tuning. Our findings raise serious questions about the practical relevance of conclusions drawn from these sorts of empirical comparisons.
spend

---

Under review as a conference paper at ICLR 2020
The remainder of this paper is structured as follows. In Section 2 review related work, focusing on papers that make explicit claims about optimizer comparisons in deep learning and application papers that provide evidence about the tuning protocols of practitioners. We develop our definition of first-order optimizers in Section aloon with a notion of inclusion relationships between optimiz- ers. We present our experimental results in Section 4 Despite thorny methodological issues over how to avoid biases in comparisons due to search spaces that favor one optimizer over another, we believe that our experimental methodology is an acceptable compromise and has substantial practi- cal relevance. Among other results, we show that the inclusion hierarchy of update rules is almost entirely predictive of optimizer comparisons. In particular, NADAM (Dozat) 2016) achieves the best top-1 validation accuracy on ResNet-50 on ImageNet in our experiments. The 77.1% we obtain with NADAM, although not as good as the 77.6% obtained using learned data augmentation by Cubuk et al. (2018), is better than the best existing published results using any of the more standard pre-processing pipelines (76.5%, due to Goyal et al. (2017) using MOMENTUM).
2 BACKGROUND AND RELATED WORK
Our work was inspired by the recent studies of neural network optimizers by Wilson et al. (2017) and [Schneider al. (2019). Wilson et al.] (2017) constructed a simple classification problem in which adaptive gradient methods (e.g. ADAM) converge to provably worse solutions than standard gradient methods. However, crucially, their analysis ignored the e parameter in the denominator of some adaptive gradient methods. Wilson et al. (017)) also presented experiments in which ADAM produced worse validation accuracy than SGD across all deep learning workloads considered. How- ever, they only tuned over the learning rate and learning rate decay scheme in their experiments, leaving all other parameters of ADAM at fixed default values. Despite these findings, adaptive gra- dient methods continue be popular since the work of Wilson et al.. (2017). [Schneider et al.. (2019) presented a benchmark suite (DEEPOBS) for deep learning optimizers and reported that there was no single best optimizer across the workloads they considered. Yet (Schneider et al.. (2019) only tuned the learning rate of each optimizer and left all other hyperparameters at some fixed default values.
As we discuss in Section 4.3 the choices of hyperparameter tuning protocols in Wilson et al. (2017) and [Schneider et al.] (2019) may be the most important factor preventing their results from being relevant to practical choices about which optimizer to use. Hyperparameter tuning is a crucial step
of the deep learning pipeline (Bergstra and Bengio 2012 Snoek et al. 2012 Sutskever et al., 2013 Smith 2018), so it is critical for papers studying optimizers to match as closely as possible the tuning protocols of an ideal practitioner. Yet, tuning protocols often differ between works studying neural network optimizers and works concerned with training neural networks to solve specific problems.
Recent papers that study or introduce optimization algorithms tend to compare to ADAM and R.MPRROP without tuning their respective hyperparameters, presumably simplify their exper- iments. It is standard to leave at the common default value of 10 8 for ADAM and 10 for
RMSPROP (Tieleman and Hinton| 2012) Kingma and Ba 2015 Dozat 2016, Balles and Hennig. Loshchilov and Hutter 2017 7: Zou and Shen 2018:
2017
Ma and Yarats, [2018] Bernstein et al., Chen et al. 2019 Zou et al. 2019). Others do not even report the value of used (Balles
2018
and Hennig 2017: Zhang and Mitliagkas 2017 Keskar and Socher 2017|Chen et al. 2018 Zhou
al.
2018
Aitchison 2018 Reddi et al. 2019 Luo al.) 2019). There are exceptrons. Zaheer
ee al. 2018)
Liu et al. (2019) considered values orders of magnitude larger than the standard
both papers gave only a limited consideration to e, testing at most two values while tuning ADAM. De et al. (2018) is the only work we found that considered a broad range of values for 6. Both Zaheer et al. (018) and [e et al. (2018) found that non-default values of outperformed the default.
default. However, the experiments
While it is also extremely common in applications to use a default value of e, some notable pa- pers tuned and selected values up to eight orders of magnitude away from the common defaults. Szegedy et al. (2016) used e = 1 for RMSPROP: Liu et al. (2019) reported that their results were sensitive to and set 10 6 for ADAM: Tan et al. (2019) and Tan and Le (2019) set 10 for RMSPROP, the latter achieving -fate-of-the-ar ImageNet top-1 accuracy. In reinforcement learning,
Hessel et al. (017)) set 1.5 x 10-1
2

---

Under review as a conference paper at ICLR 2020
Despite being introduced solely prevent division by zerd ADAM'S can be interpreted in ways If ADAM is interpreted as an empirical,
that suggest the optimal choice is problem-dependee diagonal approximation to natural gradient descent (Kingma and Ba 2015), can be viewed multi-purpose damping term whose role is to improve the conditioning of the Fisher, in analogy to
the approximate second-order method considered by Beeker and Le Cun (1888). We can also view as setting a trust region radius (Martens and Grossej 2015] Adolphs et al., 2019) and controlling an interpolation between momentum and diagonal natural gradient descent, by either diminishing or increasing the effect of U/ on the update direction. Under either interpretation, the best value for c and likely benefit from tuning.
will be
WHAT IS AN OPTIMIZER?
Optimization algorithms are typically defined by their update rule, which is controlled by hyperpa- rameters that determine its behavior (e.g. the learning rate). Consider a differentiable loss function e R - R whose vector of first partial derivatives is given by VE(0) (more generally. V2( might be a stochastic estimate of the true gradient). In our context, l represents the loss function computed over entire dataset by neural network and A E Rd represents the vector of model parameters. The optimization problem is to find point that (at least locally) minimizes f. First-order iterative methods for this problem (Nesterov) 2018) construct a sequence 0g of iterates converging to local minimum 0, using queries to and VŁ. The sequence 0, is constructed by an update rule M, which determines the next iterate 0p+1 from the history H, of previous iterates along with their function and setting of hyperparameters á N - R". Given an initial parameter value fgy E Fd the sequence of points visited by an optimizer with update rule M is given by,
and gradient values, H₃ {0.. Ve(O.),
((f,)).=o> ((()))
0,+1 = -M(H, Φc).
(1) one of the sim-
The stochastic gradient descent algorithm (SGD; Robbins and Monro, 1951)
plest such methods used for training neural networks. SGD is initialized with On E Rdd and its hyperparameter is a learning rate schedule 7 + (0, x.). The SGD update rule is given by SGD(H,, n7) 0g n) VE(O.). The MOMENTUM method due to Polyak (964)) generalizes the SGD method by linearly combining the gradient direction with a constant multple of the previ- ous parameter update. Its hyperparameters are learning rate schedule 1) N - oo) and momentum parameter E [0, oc),
MOMENTUM (OM, 7t,
= 0. - 17 V((,)) - (0.--
There has been explosion of novel first-order methods in deep learning, all of which fall into this standard first-order scheme. In Table we list the first-order update rules considered in this paper.
The difference between optimizers is entirely captured by the choice of update rule M and hyper- parameters o. Since the roles of optimizer hyperparameters on neural network loss functions are not well-understood, most practitioners tune a subset of the hyperparameters over a validation set, while leaving some hyperparameters at fixed default values. The choice of which hyperparameters tune determines an effective family of update rules, and this family the critical object from a practitioners perspective. Thus, in analogy to (overloaded) function declara- tions in C++, we define an optimizer by an update rule "signature," the update rule name together with the free hyperparameter arguments. For example, in this definition MOMENTUM(- T). 7) is not the same optimizer as MOMENTUM(-- T), 0.9), because the latter has two free hyperparameters while the former only has one. ADAM with the default is "different" from ADAM with tuned
maximize performance
3.1 THE TAXONOMY OF FIRST-ORDER METHODS
The basic observation of this section is that some optimizers can approximately simulate others (i.c., optimizer A might be able to approximately simulate the trajectory of optimizer B for any partic- ular setting of B's hyperparameters) This is important knowledge because, as a hyperparameete tuning protocol approaches optimality, a more expressive optimizer never underperform any of its specializations. To capture the concept of one optimizer approximating another, we define the following inclusion relationship between optimizers.
'TensorFlow currently refers "a small constant for numerical stability"; hittps://ww/ http:/l...//ppppp//epaee//a/ee httpp//oo/ooggoo//////////////// merr
tensor ow. www.wwwwwwwww...oooo....oo
3

---

Under review as a conference paper at ICLR 2020
Table 1: Update rules considered in this work. SGD is due Robbins and Monro [195] )
MOMENTUM to Polyak 1964), NESTEROV to Nesterov (1983). RMSPROP to Tieleman and Hin- ton 2012), and NADAM Dozat (2016). All operations are taken component for vectors. In particular, for R x² is component-wise power function.
SGD(H,, m:)
ADAM H,, 04t, 81, $2,
0+11 02 Ti VE((.)
mo 0,
M - mm + (1 (()
MOMENTUM(HH ne,
et+1 = 324h + (1 - 82) V
co =
ut+1 77 + vE((,)
be+1 -
0.+1 O2 - 1/+0++1
M++1
NESTEROV 7), to
8t41 = 0 - at Ui+1 + bit1
MADAM(H;,) 01, $1, (2,
u1+1 7/0 + VE(O,)
8.41 = 02 Ti (YUL+1 + V(((.))
mo 0, 0 m4+1 = m.me + (1 V((()(
RMSPROP (H,, Th, Ts. P,
e++1 = 820r + (1 - ((O,))
to 1,mo = 0
d1+1 V - -B7+1
1241
U++1 tm + (1 p) V V((0,)²
1)t
m++1 -ym +
ve++ +c ve(+:)
31m++1 (1 31) V()
0c11 = 0 -0
b441
0.+1 O2 m++1
v +
Definition (Inclusion relationship). Let M,, N be update rules for use in a first-order optimization method. .M is a subset specialization of N, if for all á N + RR" there exists sequence xit N > IRm such that for all E [0, oo) and histories H,,
lim N(H,, v/ MMHH,, Φ1)
This is denoted M C N, with equality M N iff M N and M.
Evidently SGD C MOMENTUM, since SGD(H,, 7:) MOMENTUM(HH, Tr, 0). Many well- known optimizers fall naturally into this taxonomy. In particular, consider RMSPROP with momentum (Tieleman and Hinton, 2012). ADAM (Kingma and Ba (2015) and NADAM (Dozat) 2016 (see Table and show the following inclusions in the appendix.
SGD C MOMENTUM RMSPROP
SGD C MOMENTUM C ADAM
SGD C NESTEROV C NADAM
Note, some of these inclusions make use of the flexibility of hyperparameter schedules (dependence of r/$ on f). In particular, to approximate MOMENTUM with ADAM, one needs to choose learning rate schedule that accounts for ADAM's bias correction.
If two optimizers have an inclusion relationship, the more general optimizer can never be worse with respect to any metric of interest, provided the hyperparameters are sufficiently tuned to optimize that metric. Optimally-tuned MOMENTUM cannot underperform optimally-tuned SGD, because setting 7 = 0 in MOMENTUM recovers SGD. However, optimizers with more hyperparameters might be more expensive tune, creating) a more general optimizer. For example, MOMENTUM improves local convergence rates over SGD wicc-diifferentiable functions that smooth and strongly convex (Polyak 1964). and
should have theoretical or experimental reason for using (or
4

---

Under review as a conference paper at ICLR 2020
NESTEROV has globally optimal convergence rates within the class of smooth and strongly convex functions (Nesterov 1983 2018).
At first glance, the taxonomy of optimizer inclusions appears resolve many optimizer comparison questions. However, for a deep learning practitioner, there is no guarantee that the inclusion hierar- chy is at all meaningful in practice. For example, the hyperparameters that allow ADAM match or outperform MOMENTUM might not be easily accessible. They might exist only in the limit of very large values, be so difficult to find that only practitioners with huge computational budgets can hope to discover them. Indeed, empirical studies and conventional wisdom hold that the inclu- sion hierarchy does not predict optimizer performance for many practical workloads (Wilson al.] 2017 Balles and Hennig 2017]] [Schneider et al. 2019). Either these experimental investigations are too limited or the taxonomy of this section of limited practical interest and provides no guidance about which optimizer to use on real workload. In the following section we attempt to answer this question experimentally, and show that these inclusion relationships are meaningful in practice.
4 EXPERIMENTS
An empirical comparison of optimizers should aim to inform a careful practitioner. Accordingly, we model our protocol on practitioner that is allowed to vary all optimization hyperparameters for each optimizer (e.g. Ote, 31, 32, e for AdAM) addition to a parameterized learning rate decay schedule, in contrast to studies that fix subset of the optimization hyperparameters to their default values (c.g. Willson et al. 2017; Schneider et al. 2019). There is no standard method for selecting the values of these hyperparameters, but most practitioners tune at least subset of the optimization hyperparameters by running a set of trials to maximize performance over the validation set. In experiments,
tens to hundreds of individual trials per workload. Given the variety of workloads we consider, this trial budget covers a wide range of computational budgets.
Selecting the hyperparameter search space for each optimizer is key methodological choice for any empirical comparison of optimizers. Prior studies have attempted to treat each optimizer fairly by using the same search space for all optimizers (c.g. Wilson et al., 2017; Schneider et al. 2019). hyperparameters should take similar
However, this requires the assumption that similarly-nama values between optimizers, which is not always true. For example, MOMENTUM and NESTEROV both have similar-looking momentum and learning rate hyperparameters, but NESTEROV tolerates larger values of its momentum hyperparameter (Sutskever al., 2013), so any fixed search space will likely be more favorable for one of the two. The situation worsens with less closely related opti- mizers, and designing a search space that is equally appropriate for optimizers with incommensurate hyperparameters is almost impossible. Despite coming with its own set of challenges, it is most in- formative to compare optimizers assuming the practitioner is allowed to tune hyperparameters for different optimizers independently by way of optimizer-specific search spaces.
In our experiments, we chose the search space for each optimizer by running an initial set of exper- iments over a relatively large search space. In typical case, we ran single set of initial trials per optimizer to select the final search space. However, in some cases we chose the initial search space poorly, so we ran another set of experiments select the final search space. The effort required to choose each search space cannot simply be quantified by the number of initial trials; the provenance of each search space is difficult to trace exactly. In some cases, our search spaces were informed by published results or prior experience with particular models and optimizers. We note that this is true of all search spaces in the literature: they are hard-won treasures that tend to refined over many experiments and across many workloads, representing the sum total of our community's experience. We validated our search spaces by checking that that the optimal hyperparameter values were away from the search space boundaries for all optimizers in all experiments (see Figure 5 Appendix D. We provide our final search spaces for all experiments in Appendix D The fact that our final er- rates compare favorably to prior published results including reaching ffte-oof-hh-art for our particular configuration of ResNet-50 on ImageNet (see Section 4.2) supports our claim that our methodology is highly competitive with expert tuning procedures.
5

---

Under review as a conference paper at ICLR 2020
Table 2: Summary of workloads used in experiments.
Evaluation metric
Target error
Batch size
Task
Model
Dataset
Budget
Simple CNN
Fashion MNIST
6.6%
256
10k steps
Image
Classification
ResNet-32
CIFAR-10
7%
256
50k steps
classification
error
CNN
CIFAR-100
256
350 epochs
VGG-16
CIFAR-10
128
250 epochs
ResNet-50
ImageNet
24%
1024
150k steps
Classification error
Language
LSTM
War and Peace
50
200 epochs
modeling
Cross entropy
Transformer
LMIB
3.45
256
750k steps
4.1 OVERVIEW OF WORKLOADS AND EXPERIMENTAL DETAILS
We investigated the relative performance of optimizers across variety of image classification and language modeling tasks. For image classification, we trained simple convolutional neural network (Simple CNN) on Fashion MNIST (Xiao et al.) 2017): ResNet-32 (He et al., 2016a) CIFAR-10 10
KKrizheskyy| 2009); a CNN on CIFAR- 100; VGG-16 (Simonyan and Zisserman 2014) on CIFAR- ImageNet (Russakovsky et al. 2015). For language modeling,
trained 2-layer LSTM model (Hochreiter and Schmidhuber] 1997) on Tolstoy's War and Peace; and Trans- former (Vaswani et al. 2017) on LMIB (Chelba et 2014). We used a linear learning rate decay schedule parameterized the same way as Shallue et al. (2019) for all workloads. We used a fixed batch size and fixed budget of training steps for each workload independent of the optimizer. Table 2oole2Ssmme summarizes these workloads and Appendix B provides the full details.
10; and ResNet-50
Given a ppperrubb-ssaapee search space, our tuning protocol sought to model a practitioner with fixed budget of trials trying to achieve the best outcome using tens of feasible trials (10, 50, or 100 depending on the workload) P feasible trial is any trial that achieves finite training loss. We used quasi-random uniform search (Bousquet et al.) 2017). and continued the search until we obtained fixed number of feasible trials. From those trials we considered two statistics. The first, in order to characterize the best outcome, is metric of interest (e.g. test accuracy) corresponding to the trial achieving the optimum of some other metric (e.g. validation accuracy). The second, in order to characterize the speed of training, is the number of steps required to reach fixed validation target conditional on at least one trial in the search having reached that target. We chose the target for each workload based on initial experiments and known values from the literature (see Table 2}. We estimated means and uncertainties using the bootstrap procedure described in Appendix C
4.2 INCLUSION RELATIONSHIPS MATTER PRACTICE
Figure ssoows the final predictive performance of six optimizers on four different workloads after tuning hyperparameters to minimize validation error. Regardless of whether we compare final val- idation error or test error, the inclusion relationships hold in all cases a more general optimizer never underperforms any of its specializations within the error bars. Similar results hold for training error (see Figure [ in Appendix D.. Training speed is also an important consideration, and Fig- ure demonstrates that the inclusion relationships also hold within error bars when we compare the number of steps required to reach a target validation error. Moreover, these results confirming the relevance of optimizer inclusion relationships do not depend the exact step budgets error tar- gets we chose (see Figure in Appendix although large changes to these values would require new experiments.
2 AAthooghh we used budget of tens of independent tuning trials throughout this section, in retrospect the best validation error across tuning trials converged quite quickly for our final search spaces, producing good results with fewer than 20 trials in many cases. See Figures 6 8 in Appendix B

---

Under review as a conference paper at ICLR 2020
Simple CNN Fashion MNIST
ResNet-32 CIFAR-10
ResNet-50 ImageNet
Transformer on LM1B
Valldaation Cross Cruss Entrropy
trror
603
Volidation Errer
0.075
Velidation Einer
3,290
2.4
0.000
0.000
0.22
0.00
0.03
0.000
3.30
2.4
0.003
1.000
122
3.2
0.000
0220
3.
Tost Cross Estropy
Test Erter
Test Enterrrr
0.673
5.000
9.29
1.000
3230
1.000
1.355
+
SGD
+ Momentum
Nesterev
RMSProp
+ Adam
+
NAcam
The relative performance of optimizers consistent with the inclusion relationships, re- gardless of whether we compare final validation enror (top) or test error (bottom). For all workloads, we tuned the hyperparameters of each optimizer separately, and selected the trial that achieved the lowest final validation error.
Figure 1:
Simple CNN on Fashion MNIST
ResNet-32 on CIFAR-10
ResNet-50 on ImageNet
Transformer on LM18
Stapsto Target
to ldstion Target
Stppto Valldation Target
Sinps Voldatioon Tanget
3.4
3.0
3.0
3.5
2.0
0.6
SGD
Momentum
Nesterov
RMSProp
Adam
NAdam
Figure 2: The relative training speed of optimizers is consistent with the inclusion relationships. We measured (idealized) training speed as the number of training steps required to reach target validation error (see Table for the error targets).
Of course, just because a more general optimizer is no worse than any of its specializations doesn't t mean the choice of optimizer makes a large difference on all workloads. For some workloads in Fig- ures [ and 2 all optimizers perform about the same, while other workloads have clear ranking or even dramatic differences. For example, the choice of optimizer seems to make little difference for ResNet-32 on CIFAR-10; all optimizers achieve similar predictive performance and training speed. the other hand, Transformer on LMIB exhibits a clear ranking in terms of predictive perfor- mance and training speed. For this workload, ADAM needs roughly half the steps that MOMENTUM requires to reach our target error, and, although not shown in Figure 2 roughly six times fewer steps get the same result SGD. These differences are clearly significant enough matter to a practitioner, and highlight the practical importance of choosing the right optimizer for some workloads.
The most general optimizers we considered were RMSPROP, ADAM, and NADAM, which do not include each other as special cases, and whose relative performance is not predicted by inclusion relationships. Across the workloads we considered, none of these optimizers emerged as the clear winner, although ADAM and NADAM generally seemed to have an edge over RMSPROP. For all of these optimizers, we sometimes had to set the parameter orders of magnitude larger than the default value in order to get good results. In particular, we achieved a validation accuracy of 77.1 % for ResNet-50 on ImageNet using NADAM with 9475, result that exceeds the 76.5% achieved by Goyal et al. (0117) using MOMENTUM. Across just these workloads, the range the optimal values of the parameter spanned 10 orders of magnitude. Faced with this reality, practitioner might reasonably doubt their ability to find a value near the optimum. However, we found that we could reasonably expect to find suitable value with only tens of trials. When tuning for ADAM or NADAM over a large range, we found it more efficient to search over (c, ag/e] instead of (c, an); Appendix D for more details.

---

Under review as a conference paper at ICLR 2020
VGG on CIFAR-10
Using Code from Wilson al., 2017
Using Our Code
0.10
+
Final Tisal Error
Test
0.08
+
+
+ +
+
+
0.07
Grid search over LR
Tuning LR + (x:e
Tuning LR schedule
Tuning LR schedule
(7.8,.8..)) + L2 reg.
SGD
Momentum
RMSProp
Adam
Figure 3: Tuning more hyperparameters removes the differences in test error between optimizers observed by Wilson et al. (017)) Tuning subset of optimizer hyperparameters and the initial learning rate is sufficient to equalize performance between all optimizers (left). More extensive hyperparameter tuning in setup, including the learning rate schedule, improves results for all optimizers and still does not produce any differences between optimizer performances (right).
4.3 RECONCILING DISAGREEMENTS WITH PREVIOUS WORK
In order to confirm that differences in hyperparameter tuning protocols explain the differences be- tween our conclusions and those Wilson al. (2017) and |Schneider al. (2019), we reproduced representative subset of their results and then inverted, or at least collapsed, the ranking over optimizers just by expanding the hyperparameter search space.
The left pane of Figure shows our experiments on VGG on CIFAR-10 using code released by Wilson et al. (2017). When we match their protocol and perform their grid search over the initial learning rate and no other tuning, we reproduce their original result showing worse test error for RMSPROP and ADAM. However, when we tune the momentum parameter and e with random search, all four optimizers reach nearly identical test error rates f With our learning rate schedule search space, merely tuning the learning rate schedule was enough to make all optimizers reach the same test error within error bars. When additionally tuned the optimization hyperparameters and weight decay in our setup we also get similar results for all optimizers, removing any evidence the inclusion relationships might be violated in practice.
Figure shows our results with different tuning protocols for a CNN on CIFAR-100 and an LSTM language model trained War and Peace match the experiments in Schheider al. (219). As reported by [Schniddrr et al. (2019), if we only tune the learning rate without tuning the decay schedule or other optimizer hyperparameters. ADAM does worse than MOMENTUM for the CNN and SGD performs slightly better than ADAM and MOMENTUM on the War and Peace dataset, although [Schneider al. (2019) found a larger advantage for SGD, However, once we tune the all the optimizer hyperparameters, ADAM does better than MOMENTUM which does better than SGD, as predicted by the inclusion relationships.
We conclude that the reason both Schneider et al. (2019) and Wilson et al. (2017) observed ranking that, at first glance, contradicts the inclusion relationships is because they were not tuning enough of the hyperparameters. If we recast their results in terminology where ADAM with default e is a different optimizer than ADAM with e tuned then there is no contradiction with our results and it becomes clear immediately that they do not consider the most interesting form of ADAM for practitioners.
Wilson et al. (017) selected trials to minimize the training loss and then report test set results. As Figure shows, removing this somewhat non-standard choice and tuning validation set and reporting test set results does change anything.
8

---

Under review as a conference paper at ICLR 2020
CNN on CIFAR-100
LSTM on War and Peace
0.55
0.42
Final Test Error
0.50
0.41
0.40
0.39
0.35
0.38
Tuning
Tuning schedule
Tuning const. LR (%,4. e
Tuning schedule Pr.
Tuning
Tuning LR schedule + (%9.9))))
SGD
Momentum
+
Adam
Figure 4: Tuning more hyperparameters changes optimizer rankings from Schneider et al. (2019) to rankings that consistent with the inclusion relationships. The leftmost columns for each workload reproduce the rankings from [Schneider et al. (2019), while the remaining columns tune over increasingly general search spaces. All columns use our random search tuning protocol.
5 CONCLUSIONS
Inspired by the recent efforts of Wilson et al. (2017) and [Schneider et al. (2019), we set out to provide detailed empirical characterization of the optimizer selection process deep learning. Our central finding is that inclusion relationships between optimizers are meaningful in practice. When tuning all available hyperparameters under realistic protocol scales common in deep learning, we find that more general optimizers never underperform their special cases. In particular, we found that RMSPROP, ADAM, and NADAM never underperformed SGD, NESTEROV, MOMENTUM under our most exhaustive tuning protocol. We did not find consistent trends when comparing optimizers that could approximate each other. We also found workloads for which there was a statistically significant separation in the optimizer ranking.
Our experiments have some important limitations and we should be careful not to overgeneralize from our results. The first major caveat is that we did not measure the effects of varying the batch size. Recent empirical work (Shallue et al., 2019) Zhang al., 2019) has shown that increasing the batch size can increase the gaps between training times for different optimizers, with the gap from SGD to MOMENTUM (Shallue al.) 2019) and from MOMENTUM ADAM (Zhang et al.) 2019) increasing with the batch size. Nevertheless, would be predictive at any batch size under a tuning protocol similar to the one we used. The second important caveat of results is that they inevitably depend the tuning protocol and workloads that we considered. Although we made every attempt to conduct realistic experiments, should only expect our detailed findings hold for similar workloads under similar protocols, namely uniform quasi-random tuning for tens to hundreds of trials, over hypercube search spaces, and with our specific learning rate schedule parameterization. Nevertheless, these caveats reinforce our central point: all empirical comparisons of neural network optimizers depend heavily on the hyperparameter tuning protocol, perhaps far more than we are used with comparisons between model architectures.
strongly suspect that the inclusion relations
If were to extract "best practices" from our findings, then we suggest the following. If we can afford tens or more runs of our code, we should tune all of the hyperparameters of the popular adaptive gradient methods. Just because two hyperparameters have similar role in two different update rules doesn't 't mean they should take similar values- optimization hyperparameters tend to be coupled and the optimal value for one may depend on how the others are set. Our results also the onus is empirical studies
confirm that the optimal value of Adam's is is problem-dependent, that fix e = 10-8 to defend that choice. Finally, we should be skeptical of empirical comparisons of optimizers in papers, especially if optimizer underperforms any of its specializations. When we do inevitably compare optimizers, we should report search spaces and highlight decisions about what hyperparameters were tuned when interpreting results.

---

Under review as a conference paper at ICLR 2020
REFERENCES
Leonard Adolphs, Jonas Kohler, and Aurelien Lucchi. Ellipsoidal trust region methods and the marginal value of Hessian information for neural network training. arXiv:/905.09201, 2019.
arXiv preprint
Laurence Aitchison. A unified theory of adaptive stochastic gradient descent as Bayesian filtering. arXiv preprint arXiv: 1rXiv:8877007500 2018.
Lukas Balles and Philipp Hennig. Dissecting Adam: The sign, magnitude and variance of stochastic gradients. arXiv e-prints, art.
May 2017.
S Becker and Y Le Cun. Improving the convergence of the backpropagation learning with second order methods. Morgan Koufmann, San Mateo, CA, 1988.
James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. Journal of Machine Learning Research, (3(Feb):281-305, 2012.
Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Anima Anandkumar. signSGI: Compressed optimisation for non-convex problems. arXiv preprint arXiv: arXiv:/802.00 /e0.. 2018.
Olivier Bousquet, Sylvain Gelly, Karol Kurach, Olivier Teytaud, and Damien Vincent. Critical hyper-parameters: no random, no cry. arXiv preprint arXiv: / 170.033200 2017.
Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Kochn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. In Conference of the International Speech Communication Association, 2014.
Jing Chen, Liu Zhao, Xue Qiao, and Yang Fu. NAMSG: An efficient method for training neural networks. arXiv preprint
2019.
Xiangyi Chen, Sijia Liu, Ruoyu Sun, and Mingyi Hong. On the convergence of class of Adam-type algorithms for non-convex optimization. arXiv preprint arXiv:1808.022: 02988888.221,,1 2018.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc Le. Autoaugment: Learning augmentation policies from data. In Conference on Computer Vision and Patlern Recog- nition, 2018
Soham De, Anirbit Mukherjee, and Enayat Ullah. Convergence guarantees for RMSProp and ADAM non-convex optimization and an empirical comparison Nesterov acceleration. arXiv preprint arXiv:1807.06766, 2018.
Timothy Dozat. Incorporating Nesterov momentum into Adam. In ICLR Workshops, 2016.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: training ImageNet in 1 hour. arXiv preprint arXiv: 1706.0267, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Conference on Computer Vision and Pattern Recognition, pages 770-778. IEEE, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European Conference on Computer Vision, pages 630-645. Springer, 2016b.
M Hessel, Modayil, and H van Hasselt. Rainbow: Combining improvements in deep reinforcement learning. 2017. arXiv preprint arXiv:1710.02299 2017.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8): 1735-1780, 1997.
Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generaliza- tion gap in large batch training of neural networks. In Advances in Neural Information Processing Systems, pages 1731-1741, 2017.
10

---

Under review as a conference paper at ICLR 2020
Sergey Loffe and Christian Szegedy. Batch normalization: accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pages 448- 456, 2015.
Nitish Shirish Keskar and Richard Socher. Improving generalization performance by switching from Adam SGD. arXiv preprint arXtv:1712.07622 2017.
Diederik Kingma and Jimmy Ba. Adam: a method for stochastic optimization. In ICLR, 2015.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical re-
port, University of Toronto, 2009. URL http://www cs ..s.toont...ddu-kkii..
learning- wewturg-eeeeteeeeeeeeeeeeeeeeeee -. pdE
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. arXiv preprint arXiv::908.00 2019.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: robustly optimized BERT pretrain- ing approach. arXiv e-prints, art. 1rXiv:1907.11692, Jul 2019.
Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in Adam. arXiv preprint arXiv:1711.05101, 2017.
Liangehen Luo, Yuanhao Xiong, Yan Liu, and Xu Sun. Adaptive gradient methods with dynamic bound of learning rate. arXiv preprint 10Xiv:1902.09843 2019.
Jerry Ma and Denis Yarats. Quasi-hyperbolic momentum and Adam for deep learning, arXiv preprint wrXiv:1810.06800) 2018.
James Martens and Roger Grosse. Optimizing neural networks with Krotecker-factored approxi- mate curvature. arXiv preprint arXiv:1503.0567/ page 58, 2015.
Yurii Nesterov. Lectures on convex optimization, volume 137. Springer, 2018.
Yurii E Nesterov. A method for solving the convex programming problem with convergence rate O(1/22).. In Dokl. akad. nauk Sssr, volume 269, pages 543-547, 1983.
Boris T Polyak. Some methods of speeding up the convergence of iteration methods. USSR Com-
putational Mathematics and Mathematical Physics, 4(5):1-17, 1964.
Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of Adam and beyond. In ICLR, 2019.
Herbert Robbins and Sutton Monro. A stochastic approximation method. The Annals of Mathemat- ical Statistics, 22(3):400-400 1951.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, (5(3):211-2522 2015.
Frank Schneider, Lukas Balles, and Philipp Hennig. DeepOBS: a deep learning optimizer bench- mark suite. arXiv preprint arXiv: 1903905549999 2019.
Christopher J. Shallue, Jaehoon Lee, Joseph Antognini, Jascha Sohl-Dickstein, Roy Frostig, and George E. Dahl. Measuring the effects of data parallelism on neural network training. Journal of Machine Learning Research, 20( 12):1-49, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1533 2014.
Leslie N. Smith. A disciplined approach to neural network
Part learning rate,
batch size, momentum, and weight decay. arXiv e-prints, art. 103.vv1803.09820, Mar 2018.
11

---

Under review as a conference paper at ICLR 2020
Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. Practical Bayesian optimization of machine learning algorithms. In Proceedings of the 25th International Conference on Neural Information Processing Systems Volume 2, NIPS' 12, pages 2951-2959 USA, 2012. Curran Associates Inc.
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: the all convolutional net. arXiv preprini 14/2,666 2014.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initializa- tion and momentum deep learning. ICML, 2013.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818-2826, 2016.
Mingxing Tan and Quoe V Le. Efficientnet: Rethinking model scaling for convolutional neural networks. arxiv preprint arXiv: 1196, 2019.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2820-2828, 2019.
Tijmen Tieleman and Geoffrey Hinton. Lecture 6,5 6.5-RMSProp: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning. 4(2):26- 31, 2012.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Advances Neural Infor- mation Processing Systems, pages 5998-6008 2017.
Ashia C Wilson, Rebecca Roelofs, Mitchell Stern, Nati Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. In Advances in Neural Information Processing Systems 30, pages 4148 4158. Curran Associates, Inc., 2017.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: novel image dataset for bench- marking machine learning algorithms. arXiv preprint wrxXiv:100.007744 2017.
Manzil Zaheer, Sashank Reddi, Devendra Sachan, Satyen Kale, and Sanjiv Kumar. Adaptive meth- ods for nonconvex optimization. In Advances in Neural Information Processing Systems 31, pages 9793-9803. Curran Associates, Inc., 2018
Guodong Zhang, Lala Li, Zachary Nado, James Martens, Sushant Sachdeva, George E. Dahl, Christopher J. Shallue, and Roger Grosse. Which algorithmic choices matter at which batch sizes? insights from a noisy quadratic model. arXiv e-prints, art. 19Xi:1907.04466 Jul 2019.
Jian Zhang and loannis Mitliagkas. Yellowfin and the art of momentum tuning. arXiv preprint 1rXiv:1706.03341,,30 2017.
Zhiming Zhou, Qingru Zhang, Guansong Lu, Hongwei Wang, Weinan Zhang, and Yong Yu. Adashift: Decorrelation and convergence of adaptive learning rate methods. arXiv preprint arXiv: 2018.
Fangyu Zou and Li Shen. On the convergence of weighted AdaGrad with momentum for training deep neural networks. arXiv preprint 1808. 1808.03408 2018.
Fangyu Zou, Li Shen, Zequn Jie, Weizhong Zhang, and Wei Liu. A sufficient condition for conver- gences of Adam and RMSProp. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 11127-1 1135, 2019.
12

---

Under review as a conference paper at ICLR 2020
A
OPTIMIZER INCLUSIONS
summarizes the update rules for the optimizers we consider in this work. We assume update rules as implemented in TensorFlow r1.15. RMSPROP includes momentum. Here we prove the their inclusion relationships, see Definition [
MOMENTUM can exactly implement SGD
MOMENTUM 4: Tt 0) = SGD( m/). so SGD C MOMENTUM.
NESTEROV exactly implement SGD
NESTEROV 7t. 0) = SGD( m.), so SGD C NESTEROV.
RMSPROP with momentum exactly implement MOMENTUM
Consider RMSPROP(4: 7 1, = 0), so that
m4+1 yym + By VE(O))
0++1 O₂ m4+1-
This is equivalent to MOMENTUM, since
(RMSPROP)
(MOMENTUM)
m4+1
Tp 2+4
Thus RMSPROO((,. nu. y, 1,0 = MOMMENTUM(1, Mi, 7), so MOMENTUM C RMSPROP.
RMSTEROV can exactly implement NESTEROV
Consider RMSTEROV Th, 7, 1,e 0), so that
m4+1 7m + n7e Ve))))
o.41 O2 lym + ma VE(O,)
This is equivalent to MOMENTUM, since
(RMSPROP)
(NESTEROV)
M++1
Thus RMSTEROV Tr, 7, 7,1,0) = MOMENTUM(4e Te, Y), SO MOMENTUM C RMSTTERV.
ADAM can approximate MOMENTUM for large
Consider ADAM(4, a, =
= 7 - >²), 31 = 7: 32 = 0,e), so that
m++1 = 7m,4 + 7) V((O:).
17t
m++1
4+11 (1 7)
[E(O0))/e
If large, so that /L(0,)l/e < 1, then
M -mm -1 V((O.),
m1++1
0.1 0, 1 1-7"
This is equivalent to MOMENTUM, since
m = (1 7) De
(ADAM)
(MOMENTUM)
Thus lim AD, - -,,,0, MOMENTUM De, 7), so MOMENTUM ADAM
NADAM can approximate NESTEROV for large
Consider NADAM(I,, 04 = cn (1 - 79). 7, B2 0, c), so that
m4+1 =ym + (1 v((e))
7m + (1 - VE(())
1)
0+11 = 02 - VE(0))//++1
13

---

Under review as a conference paper at ICLR 2020
If é is large, so that VV(0,))/c < 1, then
M++1 7 (1 -7)
ym++1 + ve(0,)
0,+1 02 7
This is equivalent to NESTEROV, since
(NADAM)
=(1 -1--) (f (NBSTEROV)
m4
Thus lime-aoe ( - 7,,,7,,e) NESTEROV (e,)))))) 7), so NESTEROV C NADAM.
B WORKLOAD DETAILS
This section details the datasets and models summarized in Table 2
B.1 DATASET DESCRIPTIONS
For Fashion MNIST, CIFAR-10, ImageNet, and LMIB, our setup identical Shallue et al. (2019) except for the image pre-processing details described below. For War and Peace, our setup was identical the "Tolstoi" dataset of Schneider et al. (2019).
CIFAR-10/100: We pre-processed images by subtracting the average value across all pixels and channels and dividing by the standard deviation For experiments with the ResNet-32 and CNN models, we followed the standard data augmentation scheme used in He et al. (2016a): 4 pixels padded on each side with single random crop from padded image or horizontal reflection. We did not use random cropping for experiments with VGG for consistency with Wilson et al. (2017).
ImageNet: We augmented images at training time by resizing each image, taking a random crop of 224 x 224 pixels, randomly horizontally reflecting the cropped images, and randomly distorting the image colors. At evaluation time, we performed single central crop 224 x 224 pixels. both training and evaluation. we then subtracted the global mean RGB value from each pixel using the values computed by Simonyan and Zisserman 2014
B.2 MODEL DESCRIPTIONS
Simple CNN is identical to the base model described in Shallue et al. (011)). It consists of convolutional layers with max pooling followed by 1 fully connected layer. The convolutional layers use x 5 filters with stride 1, "same" padding, and ReLU activation function. Max pooling uses a 2 2 window with stride 2. Convolutional layers have 32 and 64 filters each and the fully connected layer has 1024 units. It does not use batch normalization.
CNN is the "AII-CNN-C" CN model from Springenberg al. (2014), used in SShhheidder al. 2019)
The model consists of convolutional layer blocks with max pooling. The convolutional layers use 5 x filters with stride 1, "same" padding, and ReLU activation function. Max pooling uses a x window with stride 2. Convolutional layer blocks have 96, 192 and 192 filters each. As in Schneider al. (2019), we used L2 regularization of X 10 14
ResNet is described in He et al. (2016a). We used the improved residual block described in He et al. 2016b). We used batch normalization (loffe and Szegedy 2015) with exponential moving average (EMA) decay of 0.997 for ResNet-32, and ghost batch normalization (Hoffer et al. 2017) with ghost batch size of 32 and EMA decay of 0.9 for ResNet-50.
VGG is based on "model C" from Simonyan and Zisserman (2014). It consists of 13 convolutional layers followed by fully connected hidden layers. We followed the modification used by Wilson et al. (017) with batch normalization layers.
WWe used the TensorFlow op
SSee https: / //// st gi thub. com bsm/ks/te m 18390118837777 descr in for the RGB values used.
14

---

Under review as a conference paper at ICLR 2020
LSTM is two hidden-layer LSTM model (Hochreiter and Schmidhuber 1997) identical to the model used in Schneider et al. (2019). uses 128 embedding dimensions and 128 hidden units.
Transformer is the "base" model described in (Vaswani et al., 2017). We used it autoregressive language model by applying the decoder directly to the sequence of word embeddings for each sentence. Unlike the default implementation, weight matrices for the input embedding layer and the pre-softmax linear transformation, as we observed these choices led to better performing models.
removed dropout regularization and used separate
C ESTIMATING TRIAL OUTCOMES VIA BOOTSTRAP
Our tuning protocol corresponds to running trials with quasi-random hyperparameter values sampled uniformly from the search space until K feasible trials are obtained, with K depending on the workload. We then select the best trial, based on statistic of interest, over those K trials.
We used the following bootstrap procedure to estimate means and uncertainties of our tuning proto- col. We ran N > K trials, with N depending the workload. Then, for each bootstrap sample, we resampled the dataset of N trials with replacement and computed our statistic on the first K trials of the resampled dataset. We collected 100 such bootstrap samples each time, and from those computed the means, 5thh percentiles, and 95th percentiles of the bootstrap distribution. We used this procedure generate the means and error bars for each plot.
Simple CNN on Fashion MNIST used (K, .) = (100, 500); ResNet-32 on CIFAR-100 used (K, (100, 500): ResNet-50 on ImageNet used N) (50, 250): Transformer on LMIB used (K, .N = (50, 100); VGG on CIFAR- 10 with our code used (K, .N) (50, 250) for tuning the learning rate schedule and (K, N) (100, 500) for tuning the learning rate schedule, {7, $1,32.00 $2,c), and L2 regularization; CNN on CIFAR-10 used (K, N) (100,500); LSTM on War and Peace used (K, N) = (10, 50) for tuning just the learning rate and N) = (100, 500) for tuning the learning rate schedule and Br, B2,e).
The sole exceptions to this bootstrap procedure are the two left panels of Figure 3 for which we used similar procedure to Wilson al. (2017) to ensure comparability. For each optimizer, selected the trial that minimized validation error in our final search space and ran the same hyperparameter values 5 times, reporting the mean, minimum, and maximum test error over those 5 runs in Figure This is slightly different to Wilson et al. 2017). who chose the trial that minimized training error and reported validation error. When tuning the learning rate and {y, <}, we used 24 trials per optimizer in the initial search space (which we used to select the final search space), and 16 trials per optimizer in the final search space.
D HYPERPARAMETER SEARCH SPACES
When tuning hyperparameters over large range, found that our search could sometimes be made more efficient if we parametrized the search space in way that decorrelated the axes of the space. For example, with MOMENTUM and NESTEROV learning rate "/o and the momentum parameter 7 smaller values of Y/o require larger values of y for good performance, and vice versa. Indeed, (Shallue et al. (2019) suggested that these optimizers are governed by the "effective learning rate" Jeeff no/ (1 7), and inspired by this, we found that searching over (No, Tleir) instead of (Nu, usually led to a more efficient hyperparameter search. Similarly, with ADAM and NADAM we observed relationship between the initial learning rate og and the parameter, larger values of ag require larger values of for good performance, and vice versa. This is not surprising given the analysis in Appendix [ that showed that, for large e, oa/e analogous to the effective learning rate of ADAM and NADAM. We found that searching over (E, ao/e) was usually more efficient than searching over (E, We used these techniques in subset of our experiments.
observed clear relationship between the initial
Below we report the search spaces used for our experiments. We include both the initial search spaces used to refine the search spaces, and the final spaces used to generate the plots. When only one search space was used, we denote the initial space final. no, ag, 1 1, 1 31, 1 8g, e, and combinations thereof are always tuned on a log scale. The number of samples from each search space is specified in Appendix C
15

---

Under review as a conference paper at ICLR 2020
D.1 CNN ON FASHION MNIST
We used linear learning rate decay for all experiments. We tuned the number of decay steps within [0.5.1.0 times the number of training steps and the learning rate decay factor within (10-3, 1002, 10- We did not use L regularization or weight decay.
%o
initial
[10-2, 10a)
final
[10-3,10']
Table SGD
Yo
- ?
initial
[10 10
[10 4, 1] [10-4.1]
final
[10 10
Table 4: MOMENTUM
?
initial
[10 10a
[10 4, 1]
final
10-1, 10
10-1,11
Table 5: NESTEROV
7Ju [10-4, 100 [10-5,1] 1]
1-2 y
p
c
initial
[10-2, 1] [10-2,1]
[10-4 1]
[10 10
final
[10-3,1 11
[10-10 10 51
Table 6: RMSPROP
31 [10-3, 3,5 10-1
1 - 32
initial
[10-4,10 10-1111 2 [10-5,101
[10- 10
[10-9 9 10 10-55
final
[10-3, 10-3.1]
[10-4, 4,
[10 10 51
Table 7: ADAM
0g/e
- 31
B2
6
initial
110-2,10* 10
[10-3 3
[10 4 1]
[10 10 10
final
[10-1,10"]
110-3, 1]
[10
[10 6 10-21
Table 8: NADAM
16

---

Under review as a conference paper at ICLR 2020
D.2 RESNET-32 ON CIFAR-10
We used linear learning rate decay for all experiments. We tuned the number of decay steps within [0.5, 1.0] times the number of training steps and the learning rate decay factor f within the values shown in the tables below. Aa denotes the Le regularization coefficient.
7/0
X (10-5, 10 10-3 10 10
final
10-22 10
(10-4, 1003 10 10
Table 9: SGD
7/u
- 7 110-3,
X (10-5, 10-1 10-3 10-3)
final
[10-1 10
{10-1 10-3 10-2 10 10-1
Table 10: MOMENTUM
1-7
AL
f
initial
[10 10a]
[10-4, 10"]
10
{10-3 3 10 2 10111 3
final
[10-4, 1004,11%
110+,11 .,1]
{10-5, 5, 10 1010 10
10022) 2}
{10- 10 10
10-11
Table 11: NESTEROV
Ylu
1-7
1 1-
ALa
initial
[10-4, 10
[10 10-2.11
[10-4.] 1]
[10-5, 10
10-4
[10-3.10* 10 10-1)
{10-5,10* 10
(10-4,10 10-3
final
[10-4, 104]
[10-9,1]
[10-4,1] 1]
[10-9, 100
10-3, 10-$,10-3 10
10-2,10-1
Table 12: RMSPROP
Okg
1 3
32
A
{10-3, (10-3.10-2 10
initial
[10-4, 10-1)
[10-3,5 5 10-2]
[10-4, 10-1
[10-9, 10-3]
10-1
10-11
{10-5, 10
(10-4,10-b
final
[10-3,10']
[10-3,1]
110-1,10-1
[10-5, 10'
10-3,10-2]
10-2,10-1)
Table 13: ADAM
an/c
1 - 81
B2
X (10-5,101 10
f
{10-1, 10
initial
[10-2,101% 10
[10-9 3 1]
[10-4,1] 1]
[10- 10 10
10 10-3, 10-
10-2,10-1)
(10-5,10-1
(10-1,10-3
final
2,10
[10 1]
[10-4, 1]
[1, 10
10-3.10-2
10-2,10-1}
Table 14: NADAM
17

---

Under review as a conference paper at ICLR 2020
D.3 RESNET-50 ON IMAGENET
We used linear learning rate decay for all experiments. We tuned the number of decay steps within [0.5, 1.0] times the number of training steps and the learning rate decay factor f within the val- ues shown in the tables below. Awd denotes the weight decay coefficient and 7 denotes the label smoothing coefficient.
770
Awd
initial
[10-5
10']
[10 10 ?]
{0, 10 ? 10-1)
{10 4, 10 - 10
final
[0
[10-4, 10-9)
10-1
(10-1, 10-3, 10 10
Table 15: SGD
% [10-3, 1] [10-2,1]
1 -
Awd
initial
10-3, 1]
110-5, 10 10-2)
{0, 10 10-1)
{10-1, 10-3 10-2.
final
[10-,1] 1]
110-1 10-3)
10-2
(10-1,10-3, 10- 10-1)
Table 16: MOMENTUM
1 --1
Awd
initial
[10-3, 1] [10-2,1]
[10-3,] 3 1]
10 31
{0, 10 2, 10 10
10
final
[10 [10-3,1]
110 10 33
0
{10
10 3 10 10 1
Table 17: NESTEROV
Mo/VE
1-y
1-
Awd
initial
[10-2, 10
0.1
[10-4.1 1]
[10 10 10
[10 10 10-21
{0. 10 (0-2.10011
10-3
(10-4,10-3
final
1102,11 1]
0.1
[10-2, 2, I]
[10-8, 10 10-3 3]
[10-4, 10
10 10-3,10-11
Table 18: RMSPROP
ag/E
1 B1
Awd
T
f
initial
[1, 10a1
[10-3, 3, 1]
[1. 10
[10 10
{0. 10 1001)
10-3
11044,10
final
[0º]
[10-2,
[10- 10a)
10-4
10
10-2,10-1)
Table 19: ADAM
ao/e
1- Bi
Awd
T
f
initial
[10-1, 10-1,.00 10
[10-3,1 3. 1]
[10-2, 1020
[10 10 10-21
{0, 10 -2 10
10-1) -1
10
final
[1
110-311 1]
[103, 10
10-4
10
10
Table 20: NADAM
18

---

Under review as a conference paper at ICLR 2020
D.4 TRANSFORMER ON LMIB
We used linear learning rate decay for all experiments. We tuned the number of decay steps within [0.5. 1.0] times the number of training steps and the learning rate decay factor within {10-², 10-3, ,,
Y/o
final
[10-1,
10-1
Table 21: SGD
?
final
[10-1,00 10-1
[10 10-4,11 1]
Table 22: MOMENTUM
final
[10 4, 10
[10 4 1]
Table 23: NESTEROV
To
1 1-9
1
initial
[10-4, 10a]
[10-2,1] 1]
[10-2,1 1]
[10 12 10
final
[10-9, 10 21
[10-2.1]
[10-3
[10-7,10-1 10101
[10 -
Table 24: RMSPROP
1 - B1
1 1 B2
initial
[10 5 10 2]
[10-3,6 x 10
[10 [10- 10 1,044,10'1
[10 [10-9, 10 10
final
[10-4, 4,10".1.
[10-3.11
[10-5 5 10
[10 [10-7,10*
Table 25: ADAM
Ox
Bg
final
[10 10-2)
[10 3.1]
[10 110-111011
10 9
10 51
Table 26: NADAM
19

---

Under review as a conference paper at ICLR 2020
D.5 VGG ON CIFAR-10 10 USING CODE FROM WILSON ET AL. (2017)
D.5.1 GRID SEARCH OVER LEARNING RATE
We tuned over the same grid of initial learning rate values for each optimizer as Wilson et al. (017))
As in Wilson et al. (2017), we decayed the initial learning rate by factor of 0.5 every 25 epochs and used fixed L2 regularization coefficient of 0.0005.
D.5.2 TUNING LEARNING RATE & {}
We used our quasi-rame quasi-random tuning protocol to tune over the initial learning rate, MOMENTUM's % R.SPROP's e, and ADAM'S in Wilson al. (0117). we decayed the initial learning rate by factor of 0.5 every 25 epochs and used fixed L2 regularization coefficient of 0.0005.
initial
[10-
final
110-1,101]
Table 27: SGD
To
?
initial
[10-3,
[10 3 1]
final
[10-1,10 10'
[10 1, .1]
Table 28: MOMENTUM
00/ Ve
initial
[10 10 10
[10 10
final
[10-2, 10
[10 10a
Table 29: RMSPROP
ap/c
initial
[10 10 10
[10 10
final
[106, 10
[10- 10a
Table 30: ADAM
20

---

Under review as a conference paper at ICLR 2020
D.6 VGG ON CIFAR- 10 USING OUR CODE
We used linear learning rate decay for all experiments. We tuned the number of decay steps within [0.5. 1.0] times the number of training steps and the learning rate decay factor within (10-², 10 10 1201,11011 }.
D.6.1 TUNING LEARNING RATE SCHEDULE
We fixed all optimizer hyperparameters excluding the learning rate to match those specified Wil- et al. 2017). As in Wilson al. 2017), used fixed L2 regularization coefficient of 0.0005.
Yo (SGD)
Yo (MOMENTUM)
To (RMSPROP)
@p (ADAM)
initial
[10-,109
110-3 104
1009.10-1
[10-5, 10-1]
final
1.0
[10
1100410-- 10
[10-9,10-1]
Table 31: Learning rate search ranges.
D.6.2 TUNING LEARNING RATE SCHEDULE & {,, 11, e, A
11b
AL
initial
[10-3, 100
[10 5 10
final
[10-?.1]
[10 10 10
Table 32: SGD
1lo
1 - y
XLz
initial
[10-3, 110
[10-3 3
[10-5,10-2 10-21
final
[10-2.1]
[10-1,1]
[10-3 10-1)
Table 33: MOMENTUM
an/Ve
1-9 7
1-p p
AL
initial
110 10
[10-3.1]
[10-3.1]
[10 10 10
110 00-5,00 10 31
final
[10-,1]
[10-1.1
[10-3,10-2
110 106
110 10-1
Table 34: RMSPROP
on/c
1 &1
82
\
initial
[10-2,10) 10
10-3.1 1]
[10-1, 10-11
110-10 1010
(10-5, 10
final
[10-2,10*]
100-1.1
[10-A, 10
[10 20
110-3, 10
Table 35: ADAM
21

---

Under review as a conference paper at ICLR 2020
D.7 CNN ON CIFAR-100
D.7.1
TUNING CONSTANT LEARNING RATE
We fixed all optimizer hyperparameters excluding the learning rate to match those specified in Schneider 2019).
Tj (SGD)
(MOMENTUM)
a (ADAM)
initial
[10-2.1
[10-1.1]
110 D, 10-2
final
[10-1.1]
[10-3 3, 10
110-1,10-1 10-31
Table 36: Learning rate search ranges.
D.7.2 TUNING LEARNING RATE SCHEDULE
We used linear learning rate decay, and tuned the number of decay steps within [0.5, 1.0] times the number of training steps and the learning rate decay factor within {10
10 10-0,,0111 10-1).
?o (SGD)
To (MOMENTUM)
ao (ADAM)
initial
[10-3, 3, 1]
[10 4. 1]
[10 5, 10 ?]
final
[10-1.1]
[10-3,10-] 10
10 10 31
Table 37: Learning rate search ranges.
D.7.3 TUNING CONSTANT LEARNING RATE & S1 32,c]
For SGD, we reused the results from Appendix D.7.I since there were no additional hyperparame- ters to tune.
1 -
final
[10-1,1 1]
[10-3, 3 1
Table 38: MOMENTUM
a/e
1- 8
1- B2
initial
110-2,10 10
[10-3,1]
[10-4 10 1
[10-10 10 10-10
final
10-1,10-1
[10-2.1]
[10-4, 1,111110
10%]
Table 39: ADAM
D.7.4 TUNING LEARNING RATE SCHEDULE & (7,81,82.44
We used linear learning rate decay, and tuned the number of decay steps within [0.5, 1.0] times the number of training steps and the learning rate decay factor within (10-4,10-3 1000,0002,11001111111111110111,, 10-1J. 20 For SGD, we reused the results from Appendix D.7.2 since there were no additional hyperparameters to tune.
initial
[10-4, 1]
[10-2 2, 1]
final
[10 3 10 1
[10-3 1011
Table 40: MOMENTUM
22

---

Under review as a conference paper at ICLR 2020
00/c
1 31
1 1- 82
initial
11011100)
[10-3.1 1]
1101,110 10-1]
[10 109)
final
100
[10-31]
10-5,10-2
110 109)
Table 41: ADAM
23

---

Under review as a conference paper at ICLR 2020
D.8 LSTM ON WAR AND PEACE
D.8.1 TUNING CONSTANT LEARNING RATE
final
[10 10
Table 42: SGD
7
1 7
final
[10 -00-,110 1]
0.99
Table 43: MOMENTUM
a/E
31
1 B2
final
[10-5 10-21
0.9
0.999
10-8
Table 44: ADAM
D.8.2 TUNING LEARNING RATE SCHEDULE & {y, B2,
We used linear learning rate decay, and tuned the number of decay steps within [0.5, 1.0] times the number of training steps and the learning rate decay factor within the values shown in the tables below.
110
initial final
[10 3, 10
{10- 10 10 2. 1
[1, 10
{10-4,10-3
10 10 '1
Table 45: SGD
1/u
1-y
initial final
[10-1, 1]
[10-3 1]
110-4,102 10-11110-110-1110-)1
10-1)
[10-1, 10
[10-2.1]
(10-2,10-3,1000 10,,00-2,000
10-3,
Table 46: MOMENTUM
0n/c
1 - 8
1 - 82
f
initial
[10-2,101 10*]
[10- 1] [10-2,1]
[10-4 4 10
[10 10 10t0]
{10- 4
10
10-2 10-2,10-1)
final
[1, 102]
0.999
10
3
Table 47: ADAM
24

---

Under review as a conference paper at ICLR 2020
E
ADDITIONAL PLOTS
CNN on CIFAR-100 with Momentum
Final Validation Error
0.54
0.52
0.50
0.48
0.46
0.44
0.42
0.40
10
10
10
10
10
10a2
10
Learning rate
- 7
Figure 5: Example plot of We consider this search space to be appropriate because the optimal values are away from the search space boundaries.
final validation error projected onto the axes of the hyperparameter space.
SGD
Momentum
Nesterov
4.0
4.0
4.0
3.8
3.8
3.8
3.7
3.7
3.7
Valeation
3.6
3.6
3.6
3.5
3.5
3.5
3.37
3.3
RMSProp
Adam
NAdam
3.9
3.9
3.9
Valietien Cress
3.8
3.8
3.8
3.7
3.6
3.6
3.6
3.4
3.4
3.4
3.3
2
2
2'
2
2
2'
2
2
Number
Number Independent
Number Independent
Figure 6: Validation performance of the best trial mostly converges with few 24 hyperparameter tuning trials for Transformer on LMIB, Shaded regions indicate 5th and 95th percentiles estimated with bootstrap sampling (see Appendix C). The search spaces can found Appendix D.4
25

---

Under review as a conference paper at ICLR 2020
SGD
Momentum
Nesterov
0.27
0.27
0.27
Final Validation Error
0.25
0.25
0.25
0.24
0.24
0.24
0.23
0.23
0.23
2
2
2
2
RMSProp
Adam
NAdam
0.27
0.27
0.27
Final Validation Emorr
0.26
0.25
0.25
0.25
0.24
0.24
0.24
0.23
0.23
0.23
2
2
2' 2
2
2
Number Independent Trials
Number Indeperdent Trials
Number of Indapendent Trials
Figure 7: Validation performance of the best trial mostly converges with as few as 24 hyperparameter tuning trials for ResNet-50 in ImageNet. Shaded regions indicate 5th and 95th percentile estimated with bootstrap sampling (see Appendix C
The search spaces can be found in D.3
SGD
Momentum
Adam
0.70
0.70
0.70
0.65
0.65
0.65
Test Eree
Final Tist
0.55
0.55
0.55
0.50
0.50
0.50
0.45
0.40
0.40
0.40
0.35
0.35
0.35
2
21
2'
2
2
27
2.
2
2
2
21
Number Independent Triats
Number of Independent Trials
Number of Independent Trials
Figure 8: Test performance of the best trial mostly converges with few 2 hyperparameter tuning trials for a 2-layer LSTM on War and Peace. Shaded regions indicate 5th and 95th percentile estimated with bootstrap sampling (see Appendix C
The search spaces can be found in D.8.2
Simple CNN on Fashion MNIST
ResNet-32 on CIFAR-10
ResNet-50 on ImageNet
Transformer LM18
60).
0.66
1.8
3:00
0.30
Train Dross Eetrogs
1.4
3.50
Train Loge
Train
0.03
0.000
1.0
346
0.15
06
300
0.030
0.000
0.000
0.-
320
SGD
Momentum
Nesterev
RMSProp
+
Adam
+ NAdam
Figure 9: The relative performance optimizers is consistent with the inclusion relationships when select for lowest training loss. Note that SGD, ADAM, and NADAM for ResNet-50 ImageNet used label smoothing in their final search spaces (see Section D.3) which makes their loss values incommensurate with the other optimizers. This is because their final search spaces optimized to minimize validation error-if we had optimized their search spaces to minimize training error instead, we would have used label smoothing, and expect their training loss values would be consistent with the inclusion relationships.
26

---

Under review as a conference paper at ICLR 2020
Transformer on LM1B
x10
3.70
Validation Cross Entropy
3.65
Steps to Threshold
3.50
3.45
3.40
3.35
4.5
3.50
Step Budget
1e5
Validation Cross Entropy Target
ResNet-32 on CIFAR-10
5.0 x10
0.10
Steps Threshold
4.5
Validation Error
0.09
4.0
3.0
0.07
2.5
2.5
3.0
3.5
4.0
4.5
5.0
0.080
0.075
0.070
0.065
Step Budget
1e4
Validation Error Target
SGD
Momentum
Nesterov
RMSProp
Adam
NAdam
Figure 10: Our results confirming the relevance of optimizer inclusion relationships do not depend on the exact step budgets or error targets we chose.
27


Reviews:
[{"review_id": "ryllFmFoYH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents experimental data supporting the claim that the under aggressive hyper-parameter tuning different optimizers are essentially ranked by inclusion --- if the hyper-parameters of method A can simulate any setting of the hyper-parameters of method B then under aggressive hyper-parameter tuning A will dominate B.  One way to achieve this rather trivially is to do a hyper-parameter search for B and then set the hyper-parameters of A so that A is simulating B.  But the point here is that direct and feasible tuning of A with dominate B even in the case where A has more hyper-parameters and where hyper-parameter optimization of A would seem to be more difficult.  An important conclusion is that without loss of generality one can always use Adam even in vision where SGD is currently the dominant optimizer used in practice.  Another important conclusion is that quasi-random hyper-parameter optimization is quite effective.\nI find the claims to be intuitively plausible and I find the empirical results compelling.\n\nJust a couple minor complaints.  First,\"Hyper-parameter\" please not \"meta-parameter\".  I find the attempt to overturn standard usage inappropriate.  Second, I found most of section 3 uninformative.  I don't think algorithm 1, or the definition of a first order optimizer adds anything to the paper.  Inclusion can be easily defined in complete generality for any parameterized algorithm.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}, {"review_id": "rJgB9OBCtS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n\nFirst, I would like to note that the claim that SGD with momentum is a special case of Adam with large epsilon is technically wrong because Adam also includes the bias-corrected momentum estimates which SGD with momentum does not consider. It might seem like a small difference, however it is a form of learning rate schedule which most users of Adam are not aware of. In practice, however, Adam with large epsilon can approximate SGD with momentum. Just don't claim the equivalent since it is not there. \n\nI have some difficulties understanding the contribution of the paper. For example \n\"When tuning all available metaparameters under a realistic protocol at scales common in deep learning,\nwe find that more general update rules never underperform their special cases.\"\nIn practice you do adjust hyperparameter search spaces to fit your conclusions, e.g., \"We found that searching over (epsilon, alpha0/epsilon) was more efficient than searching over (epsilon, alpha).\" Again, this alone invalidates your experimental setup since you biased it in order to fit your conclusion: \"was more efficient\" was found after running some prior experiments. \nAnother situation where your experimental setup is unfairly tuned is when you used different hyperparameter ranges for similar hyperparameter, e.g. see D.2 for ResNet-32 on CIFAR-10 where 6 orders of magnitute difference was used for the initial learning of Momentum and 3 orders of magnitude difference for the initial learning rate of Adam. Similarly, there is a difference of 10x for ImageNet experiments. \n\nThe paper suggests that 16 experiments is enough to produce good results. First, one should not forget the  special arrangements (see above) done for hyperparameter search space. Second, for any person working in black-box optimization it is clear that 16 experiments is next to nothing. It should give you something good in 1D, possibly in 2D if your search range is narrow. This is absolutely nothing in larger dimensions (providing that your benchmark in not super trivial and your hyperparameter search space is not absolutely boring when you already narrowed it around the optimum). After 16 evaluations you get pretty bad settings for most algorithms. \n\nUpdate: \n\nThe paper uses a naive hyperparameter optimizer and runs it for a very small budget. The latter likely affects the conclusion of the paper that different training algorithms perform similarly. The authors seem to accept it by mentioning that this is the case for their tuning protocol/budget. \n\nIf we would like to compare different training algorithms, we should optimize them on a set of problems using 2-3 state-of-the-art hyperparameter optimizers. Then, we should study how the best seen solutions so far and their robustness  change as a function of the computation budget (the maximum budget should be large enough). Then, one would see that the results are not that different for small budgets (a boring result) and somewhat different for larger budgets. Showing only the boring part seems more misleading than useful. \n\nUpdate#2:\nAs I mentioned in my review, Adam with large epsilon is not equivalent to momentum SGD but only approximates the latter. This is because the original Adam has a bias correction term and even if the same *global* learning rate schedule is used both for Adam with large epsilon and momentum SGD, they are not equivalent. In order to obtain the exact equivalence, one would need to either\n1) drop the bias correction term of Adam and thus modify the algorithm in order to satisfy the claimed equivalence\nor \n2) set a particular learning rate *for each batch pass* of Adam to simulate the effect of the bias correction term, this leads to a large number of hyperparameters - as many as the number of batch passes, this is intractable (the setup of the authors does not optimize such batch-wise hyperparameters, they are defined by a global scheduler as a function of batch/epoch index).  \nIf you avoid these modifications, then you can't claim the equivalence but only an approximation. If you don't have the equivalence of the two approaches and so momentum SGD is not a particular case of Adam, then the following sentence from the abstract is false: \" As tuning effort grows without bound, more general optimizers should never underperform the ones they can approximate (i.e., Adam should never perform worse than momentum)\". Again, strictly speaking, it is false that \"Adam should never perform worse than momentum\" because momentum SGD is not a particular case of the original Adam *unless* you drop the bias correction term or simulate it with tons of hyperparameters, one learning rate value per batch pass. Any global learning rate schedule *used for both* algorithms will not solve the issue because the bias correction term will remain. If you don't modify the learning rate schedule of Adam but only of momentum SGD, then you basically adjust your SGD by moving some part of Adam in it to claim the equivalence of the two, such actions can make pretty much every second algorithm equivalent to another. \n\nMy main concern is described in the first Update. It is trivial that a more general optimizer is capable to perform at least as good as its particular case. What is not trivial is to clarify the interplay of computational budgets spent on hyperparameter tuning vs number of hyperparameters vs performance over time.  ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BkgPIQ2yqr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper provides an empirical comparison of a set of first-order optimization methods for deep learning models. Those optimizers include stochastic gradient descent, momentum  method, RMSProp, Adam, Nesterov, and Nadam, which arguably covers all popular variants used in the literature. Although it is not the first empirical study on this topic, its conclusion differs slightly. The conclusion is a rather intuitive one: With proper parameter search, the 'richer', more powerful optimizers tend to work better, regardless of the downstream tasks. \n\nPros: \n- Intuitive results with a well designed workloads and experiments. For practitioners that want to start their own hyperparameter search, the workloads and setups are likely to be useful. \n\nCons:\n- I am not entirely convinced that the inclusion relationship is indeed a major cause or indicator of different optimizers' performance. There is no theoretical justification; Empirically, if one takes two optimizers equally rich and tunes one of them more intensively, one should expect a better performance, too.\n\nSuggestions:\n\n- I think at least the basic definitions of different optimizers should be given in the main text. Otherwise, readers without detailed knowledge of all these optimizers cannot follow the paper. For example, the paper starts talking about the taxonomy of the optimizers with their corresponding hyperparameters in Section 3.2 before giving any functional form of the optimizers. \n\n- I would suggest the authors to follow the convention and use the term \"hyperparameter\" rather than \"metaparameter\". The readers of this paper are not primarily Bayesian, there is really no need to divert from the convention. Besides, the term \"Bayesian hyperparameter tuning\" is widely used even. \n\n- I wonder to which extent the network structures impact the choice of the hyperparameter (e.g., CNN vs. RNN). ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper examines classifiers and challenges a (somewhat widely held) assumption that adaptive gradient methods underperform simpler methods.

This paper sparked a *large* amount of discussion, more than any other paper in my area. It was also somewhat controversial.

After reading the discussion and paper itself, on one hand I think this makes a valuable contribution to the community. It points out a (near-) inclusion relationship between many adaptive gradient methods and standard SGD-style methods, and points out that rather obviously if a particular method is included by a more general method, the more general method will never be worse and often will be better if hyperparameters are set appropriately.

However, there were several concerns raised with the paper. For example, reviewer 1 pointed out that in order for Adam to include Momentum-based SGD, it must follow a specialized learning rate schedule that is not used with Adam in practice. This is pointed out in the paper, but I think it could be even more clear. For example, in the intro "For example, ADAM (Kingma and Ba, 2015) and RMSPROP (Tieleman and Hinton, 2012) can approximately simulate MOMENTUM (Polyak, 1964) if the ε term in the denominator of their parameter updates is allowed to grow very large." does not make any mention of the specialized learning rate schedule.

Second, Reviewer 1 was concerned with the fact that the paper does not clearly qualify that the conclusion that more complicated optimization schedules do better depends on extensive hyperparameter search. This fact somewhat weakens one of the main points of the paper.

I feel that this paper is very much on the borderline, but cannot strongly recommend acceptance. I hope that the authors take the above notes, as well as the reviewers' other comments into account seriously and try to reflect them in a revised version of the paper.
Decision: Reject