Descending through a Crowded Valley — Benchmarking Deep Learning Optimizers
Authors: Robin Marc Schmidt
Frank Schneider
Philipp Hennig
Venue:  (2021)
Keywords: Deep learning
optimizers
benchmark

Abstract: Choosing the optimizer is considered to be among the most crucial design decisions in deep learning, and it is not an easy one. The growing literature now lists hundreds of optimization methods. In the absence of clear theoretical guidance and conclusive empirical evidence, the decision is often made based on anecdotes. In this work, we aim to replace these anecdotes, if not with a conclusive ranking, then at least with evidence-backed heuristics. To do so, we perform an extensive, standardized benchmark of more than a dozen particularly popular deep learning optimizers while giving a concise overview of the wide range of possible choices. Analyzing almost $35,000$ individual runs, we contribute the following three points: (i) Optimizer performance varies greatly across tasks. (ii) We observe that evaluating multiple optimizers with default parameters works approximately as well as tuning the hyperparameters of a single, fixed optimizer. (iii) While we can not discern an optimization method clearly dominating across all tested tasks, we identify a significantly reduced subset of specific algorithms and parameter choices that generally lead to competitive results in our experiments. This subset includes popular favorites and some lesser-known contenders. We have open-sourced all our experimental results, making them directly available as challenging and well-tuned baselines. This allows for more meaningful comparisons when evaluating novel optimization methods without requiring any further computational efforts.

Under review as a conference paper at ICLR 2021
DESCENDING THROUGH A CROWDED VALLEY BENCHMARKING DEEP LEARNING OPTIMIZERS
Anonymous authors Paper under double-blind review
ABSTRACT
Choosing the optimizer is considered to be among the most crucial design deci- sions in deep learning, and not an easy one. The growing literature now lists hundreds of optimization methods. In the absence of clear theoretical guidance and conclusive empirical evidence, the decision is often made based on anecdotes. In this work, we aim to replace these anecdotes, if not with conclusive ranking, then at least with evidence- baaced heuristics. To do so, we perform an extensive, standardized benchmark of more than dozen particularly popular deep learning optimizers while giving a concise overview of the wide range of possible choices. Analyzing almost 35,000 individual runs, contribute the following three points: (i) Optimizer performance varies greatly across tasks. (ii) We observe that eval- uating multiple optimizers with default parameters works approximately as well as tuning the hyperparameters of a single, fixed optimizer. (iii) While we can not discern an optimization method clearly dominating across all tested tasks, we identify significantly reduced subset of specific algorithms and parameter choices that generally lead to competitive results in our experiments. This subset includes popular favorites and some lesser-known contenders. We have open-sourced all our experimental results, making them directly available as challenging and well-tuned baselines. This allows for more meaningful comparisons when evaluating novel optimization methods without requiring any further computational efforts.
INTRODUCTION
Large-scale stochastic optimization drives wide variety of machine learning tasks. Because choosing the right optimization algorithm and effectively tuning hyperparameters heavily influences the training speed and final performance of the learned model, doing so is an important, every-day challenge to practitioners. Hence, stochastic optimization methods have been a focal point of research (cf. Figure 1), engendering ever- wver-growing list of algorithms, many of them specifically targeted towards deep learning. The hypothetical machine learning practitioner who is able to keep up with the literature now has the choice among hundreds of methods (cf. Table 2 in the appendix) each with their own set of tunable hyperparameters when deciding how to train their model.
There is limited theoretical analysis that would clearly favor of these choices Some authors have offered empirical comparisons on comparably small sets of popular methods (e.g. Wilson al., 2017; Choi al., 2019; Sivaprasad al., 2020); but for most algorithms, the only formal empirical evaluation is offered by the original work introducing the method. Many practitioners and researchers, meanwhile, rely on personal and anecdotal experience, and informal discussion on social media or with colleagues. The result is an often unclear, perennially changing "state of the art" occasionally driven by hype. The key obstacle for an objective benchmark the combinatorial cost of such an endeavor posed by comparing large number of methods on large number of problems, with the high resource and time cost of tuning each method's 's parameters and repeating each (stochastic) experiment repeatedly for fidelity.
the others.
Offering our best attempt to construct such comparison, we conduct large-scale benchmark of optimizers to further the debate about deep learning optimizers, and to help understand how the choice of optimization method and hyperparameters influences the training performance. Specifically,
hhttp:://gt thub. com AnmnSummm ttert/ Submi s8153443

---

Under review as a conference paper at ICLR 2021
mentions
2006
2008
2010
2012
2014
2016
2018
2020
AMSBound
Adadelta
Lookahead
Nadam
RMSProp
AMSGrad
Adagrad
Momentum
Other
SGD
AdaBound
Adam
NAG
RAdam
Figure 1: Number of times ArXiv titles and abstracts mention each optimizer per year. All non- selected optimizers from Table the appendix are grouped into Other.
we examine whether recently proposed methods show an improved performance compared more established methods such as SGD or ADAM. Additionally, we are interested in assessing whether optimization methods with well-working default hyperparameters exist that are able to keep up with tuned optimization methods. To this end, we evaluate more than dozen optimization algorithms, largely selected for their perceived popularity, on a range of representative deep learning problems (see Figure 4) drawing conclusions from tens of thousands of individual training runs.
Right up front, we want to state clearly that is impossible include all optimizers (cf. Table 2 the appendix), and to satisfy any and all expectations readers may have on tuning and initialization proce- dures, or the choice of benchmark problems-no least because everyone has different expectations in this regard. In our personal opinion, what is needed is an empirical comparison by third party not involved in the original works. As a model reader of our work, we assume a careful practitioner who does not have access to near-limitless resources, to broad range of personal experiences. As such, the core contributions (in order of appearance, not importance) of our work are:
concise summary of optimization algorithms and schedules partly automated, mostly man- ual literature review provides a compact but extensive list of recent advances in stochastic optimization. We identify more than a hundred optimization algorithms (cf. Table in the appendix) and more than 20 families of hyperparameter schedules (cf. Table 3 in the appendix) published at least as pre-prints.
An extensive optimizer benchmark on deep learning tasks We conduct a large-scale optimizer benchmark, specifically focusing optimization problems arising in deep learning. We evaluate 14 optimizers on eight deep learning problems using four different schedules, tuning over dozens of hyperparameter settings, our knowledge, this is the most comprehensive empirical evaluation of deep learning optimizers to date (cf. Section 1.1 on related work).
An analysis thousands of optimization runs Our empirical experiments indicate that an opti- mizer's performance highly depends on the test problem (see Figure 4). But some high-level trends emerge, too: (1) Evaluating multiple optimizers with default hyperparameters works approximately well tuning the hyperparameters for fixed optimizer. (2) Using an additional untuned learning rate schedule helps on average, but its effect varies greatly depending on the optimizer and the test problem. (3) While there is no optimizer that clearly dominates across all tested workloads, some of the algorithms we tested exhibited highly variable performance. Others demonstrated decent performance consistently. We deliberately refrain from recommending single one among them, because we could not find clear winner with statistical confidence.
open-source baseline for future optimizer benchmarks Our results are accessible online in an open and easily accessible form (see footnote Page 1). These results thus be used as competitive and well-tuned baselines for future benchmarks new algorithms, drastically reducing the amount of computational budget required for meaningful optimizer comparison. Our baselines can easily be expanded, and we encourage others to contribute to this collection.
The high-level result of our benchmark is, perhaps expectedly, not clear winner. Instead, comparison shows that, while some optimizers are frequently decent, they also generally perform similarly, switching their relative positions in the ranking which can partially be explained by the
2

---

Under review as a conference paper at ICLR 2021
No Free Lunch Theorem (Wolpert & Macready, 1997). A key insight of our comparison is that practitioner with a new deep learning task can expect to do about equally well by taking almost any method from our benchmark and tuning it. they would by investing the same computational resources into running a set of optimizers with their default settings and picking the winner.
Possibly the most important takeaway from our comparison is that "there are now enough optimizers." Methods research in stochastic optimization should focus on significant (conceptual, functional, performance) smprovements-suchh
methods specifically suited for certain problem types, inner- loop parameter tuning or structurally novel methods. We make this claim not to discourage research but, quite on the contrary, offer motivation for more meaningful, non-incremental research.
1.1 RELATED WORK
Following the rapid increase in publications optimizers, benchmarking these methods for the application in deep learning has only recently attracted significant interest. Schneider et al. (2019) introduced a benchmarking framework called DEEPOBS, which includes wide range of realistic deep learning test problems together with standardized procedures for evaluating optimizers, Metz al. (2020) presented TASKSET, another collection of optimization problems focusing on smaller but many more test problems. For the empirical analysis presented here, we use DEEPOBS as it provides optimization problems closer to real- deep learning tasks. In contrast to our evaluation of existing methods, TASKSET and its analysis focuses on meta-learning new algorithms or hyperparameters.
Both Choi et al. (2019) and Sivaprasad et al. (2020) analyzed specific aspects of benchmarking process. Sivaprasad et al. (2020) used DEEPOBS to illustrate that the relative performance of an optimizer depends significantly on the used hyperparameter tuning budget. The analysis by Choi et al. (2019) supports this point, stating that "the hyperparameter search space may be the single most important factor explaining the rankings." " They further stress a hierarchy among optimizers, demonstrating that, given sufficient hyperparameter tuning, outperformed by special cases. In their study, however, they manually chose a hyperparameter search space per optimizer and test problem basing it either on prior published results, prior experiences, or pre-tuning trials. Here we instead aim to identify well- well-performing optimizers in the case of a less extensive tuning budget and especially when there is no prior knowledge about well-working hyperparameter values for each specific test problem. We further elaborate on the influence of our chosen hyperparameter search strategy in Section 4 discussing the limitations of our empirical study.
general optimizers can never be
Our work is also related to empirical generalization studies of adaptive methods, such as that of Wilson al. (2017) which sparked an extensive discussion whether adaptive methods (e.g. ADAM) tend to generalize worse than standard first-order methods (i.e. SGD).
2 BENCHMARKING PROCESS
Any benchmarking effort requires tricky decisions the experimental setup that influence the result. Evaluating on a specific task or picking a certain tuning budget, for example, may favor or disadvantage certain algorithms (Sivaprasad al., 2020). It is impossible to avoid these decisions to cover all possible choices. Aiming for generality, we evaluate the performance on eight diverse real-world deep learning problems from different disciplines (Section 2.1). From a collection of more than hundred deep learning optimizers (Table 2 in the appendix) we select 14 of the most popular and most promising choices (cf. Figure 1) for this benchmark (Section 2.2). For each problem and optimizer we evaluate all possible combinations of three different tuning budgets (Section 2.3) and four selected learning rate schedules (Section 2.4), thus covering the following combinatorial
space:
Problem
Optimizer
Tuning
Schedule
P1
AMSBound
constant
one-shot
P2
AMSGrad
cosine decay
x
small budget
x
cosine warm restarts (trapezoidal
large budget
P8
SGD
3
14
Combining those options results in 1,344 possible configurations and roughly 35,000 individual runs.
3

---

Under review as a conference paper at ICLR 2021
Table Summary of test problems used in our experiments. The exact model configurations can be found in the work of Schneider et al. (2019).
Data set
Model
Task
Metric
Batch size
Budget epochs
PI
Artificial data set
Noisy quadratic
Minimization
Loss
128
100
P2
MNIST
VAE
Generative
Loss
64
50
P3
MANSST
Simple CNN: 2c2d
Classification
Accuracy
128
100
P4
CIFAR-I0
Simple CNN: 3c3d
Classification
Accuracy
128
100
P5
Fashion-MNIST
VAE
Generative Classification
Loss
64
100
P6
CIFAR-100
All-CNN-C
Accuracy
256
350
P7
SVHN
Wide ResNet 16-4
Classification
Accuracy
128
160
P8
War and Peace
RNN
NLP
Accuracy
50
200
2.1
TEST PROBLEMS
We consider the eight optimization tasks summarized in Table 1, available as the "small" (P1-P4) and "large" (P5-P8) problem sets, respectively. together forming the default collection of DEEPOBS, A detailed description of these problems. including architectures, training parameters, etc. can be found in the work of Schneider al. (2019).2 DEEPOBS test problems provide several performance metrics, including the training and test loss, the validation accuracy, etc. While these are all relevant, any comparative evaluation optimizers requires picking only a few, if not just one particular performance metric. For analysis (Section 3), we focus on the final test accuracy (or the final test loss, if no accuracy is defined for this problem). This metric captures, for example, the optimizer's ability generalize and is thus highly relevant for practical use. Our publicly released results include all metrics for completeness. An example training loss performance shown in Figure 16 in the appendix. Accordingly, the tuning (Section 2.3) done with respect to the validation metric. We discuss possible limitations resulting from these choices in Section 4.
2.2 OPTIMIZER SELECTION
In Table 2 in the appendix we collect over hundred optimizers introduced for, suggested for, used in deep learning. This list was manually and incrementally collected by multiple researchers trying to keep up with the field over recent years. It is thus necessarily incomplete, although may well represent one of the most exhaustive of such collections. Even this incomplete list, though, contains many entries for meaningful benchmark with the degrees of freedom collected above. This is serious problem for research: Even an author of a new optimizer, let alone a practitioner, could not possibly be expected compare their work with every possible competing method.
We thus selected subset of 14 optimizers, which we consider to be currently the most popular choices in the community (see Table in the appendix). These do not necessarily reflect the "best" algorithms, but are either commonly used by practitioners and researchers, or have recently generated enough attention garner interest. Our selection is focused on first-order optimization methods, both due to their prevalence for non-convex continuous optimization problems in deep learning as well as to simplify the comparison. Whether there a significant difference between these optimizers or they are inherently redundant is one of the questions this work investigates.
With our list, we tried to focus on optimization algorithms over techniques, although we acknowledge, the line being very blurry. Techniques such as averaging weights (Izmailov et al., 2018, e.g.) or ensemble methods (Garipov et al., 2018, e.g.) have been shown be simple but effective at improving the optimization performance. Those methods, however, be applied to all methods in our lists, similar to regularization techniques, learning rate schedules, or tuning methods and we have, therefore, decided omit them from Table 2.
All experiments were performed using version 2.2. 0-bbta of DEEPOBS and TensorFlow version 1.15 Abadi al. (2015).
4

---

Under review as a conference paper at ICLR 2021
2.3 TUNING
Budget Optimization methods for deep learning regularly expose hyperparametters to the user. The user sets them either by relying on the default suggestion; using experience from previous experiments; or using additional tuning to find the best-performing setting. All optimizers in our benchmark have tunable hyperparameters, and we consider three different tuning budgets.
The first budget consists of just single run. This one-shot budget uses the default values proposed by the original authors, where available (Table 4 in the appendix lists the default parameters). If an optimizer performs well in this setting, this has great practical value, as it drastically reduces the computational resourees required for training. The other budgets consist of 25 and 50 tuning runs for what we call the small and large budger settings, respectively.
We only use single seed for tuning, then repeat the best setting 10 times using different seeds. This allows us report standard deviations in addition to means, assessing stability. Progressing in this way has the "feature" that our tuning process can sometimes pick "lucky" seeds, which do not perform as well when averaging over multiple runs. This is arguably a good reflection of reality. Stable optimizers should be preferred in practice, which is thus reflected in our benchmark. See Appendix C for further analysis. By contrast, using all 10 random seeds for tuning as well would drastically increase cost, not only for this benchmark, rendering practically infeasible, but also as an approach for the practical user. Appendix D explores this aspect further: If anything, re- re-tuning would further broaden the distribution of results.
Tuning method We tune parameters by random search, for both the small and the large budget. Random search is a common choice in practice due to its efficiency advantage over grid search (Bergstra & Bengio, 2012) and its ease of implementation and parallelization compared to Bayesian optimization (see also Section 4). A minor complication of random search is that the sampling distribution affects the optimizer's 's performance. One can think of the sampling distribution as a prior over good parameter settings, and bad priors consequently ruin performance. We followed the mathematical bounds and intuition provided by the optimizers' authors for relevant hyperparameters. The resulting sampling distributions can be found in Table 4 in the appendix. In case there is no prior knowledge provided in the cited work we chose similar distributions for similar hyperparameters across different optimizers. Even though a hyperparameter might have a similar naming throughout different optimization algorithms (e.g. learning rate a), its appropriate search space can differ across optimizers. Without grounded heuristics on how the hyperparameters differ between optimizers, the most straightforward approach for any user is to use the same search space.
What should be considered hyperparameter? There's a fuzzy boundary between (tunable) hyperparameters and (fixed) design parameters. A recently contentious example the & adaptive learning rate methods like ADAM. It was originally introduced safeguard against division by zero, but has recently been re-interpreted as a wooblemddepeneen hyperparameter choice (see Choi et al. (2019) for a discussion). Under this view, can actually consider several separate optimizers called ADAM: From an easy-to-tune but potentially limited ADAM.. only tuning the learning rate, to the tricky-to- toky-tt-tunn tune but pow-rflerful ADAMa, 9t ,9,.e, which subsumes SGD as corner case in its hyperparameter space. In our benchmark, we include ADAMa, 62 as a popular choice. While they share the same update rule, we consider them to be different optimizers.
2.4 SCHEDULES
The literature on learning rate schedules now nearly as extensive as that on optimizers (ef. Table 3 in the appendix). In theory, schedules can be applied to all hyperparameters of an optimization algorithm but to keep our configuration space feasible, we only apply schedules to the learning rate, by far the most popular practical choice (Goodfellow et al., 2016; Zhang et al., 2020). We choose four different learning rate schedules, trying to cover all major types of schedules (see Appendix E):
A constant learning rate schedule;
cosine decay (Loshchilov & Hutter, 2017) as an example of smooth decay;
A cosine with warm restarts schedule (Loshchilov & Hutter, 2017) as cyclical schedule;
A trapezoidal schedule (Xing et al., 2018) from the warm-up schedules (Goyal et al., 2017).
5

---

Under review as a conference paper at ICLR 2021
3 RESULTS
How well do optimizers work out-of-the-box? By comparing each optimizer's one-shot results against the tuned versions of all 14 optimizers, we can construct a 14 x 14 matrix of performance gains. Figure 2 illustrates this on five test problems showing improvements by positive sign and green cell. Detailed plots for all problems are in Figures 9 and 10 the appendix. For example, the bottom left cell of the largest matrix in Figure 2 shows that AMSBOUND (1) tuned using small budget performs 2.5% better than SGD ( (4) with default parameters on this specific problem.
CIF 3c3d
HMNIST
CIFAR-10 AI-GNNN
AMSBound:
4.55
0.2
403
0.1
0.1
-05
AMSGrad:
38.5
26.0
26.0
16.
10:0
100
38.6
41:1
14
1
38.6
30.1
50.0
AdeBound:
8.
4.1
07
0.1
0.5
4.3
405
2.5
0.7
0.3
0.5
-22
4.6
-01
Adadella:
4.2
43.5
442
42.
463
425
421
40.5
4411
46.1
46.0
424
422
43.5
Adagrad:
1.0
2.
2.0
Adam:
1.
1.6
18
One-shot
LAIMom.::
5.7
5.4
13
11
LAIRAdami:
SVHN Wide ResNet 16-4
Tolstal Char RNN
Mom.
3
8
RAdam: 12
3
RMSProp: 13
3.5
2.6
11
SGD: 14
2.5
2.0
24
13
24
2.
2.4
2.6
1.
20
Small budget
Perforance improvement
Figure 2: The absolute test set performance improvement after switching from any untuned optimizer (g-axis, one-shot) to any tuned optimizer (æ-axis, small budget) as an average over 10 random seeds for the constant schedule. We discuss the unintuitive occurrence of negative diagonal entries in Appendix F. The colormap is capped at +3 to improve presentation, although larger values occur.
A green row in Figure 2 indicates that an optimizer's s default setting is performing badly, since it can be beaten by any well-tuned competitor. We can observe badly modly-performing default settings for MOMENTUM, NAG and SGD, advocating the intuition that non-adaptive optimization methods require more tuning, but also for AMSGRAD and ADADELTA. This just statement about the default parameters suggested by the authors or the popular frameworks, well-working default parameters might well exist for those methods. Conversely, a white & red row signals a well- performing default setting, since even tuned optimizers cannot significantly outperform this algorithm. ADAM, NADAM and RADAM, as well as AMSBOUND and ADABOUND all have white or red rows several (but not all!) test problems, supporting the rule of thumb that adaptive methods have well-working default parameters. Conversely, green (or red) columns highlight optimizers that, when tuned, perform better (or worse) than all untuned optimization methods. We do observe such columns consistently across tasks. This supports the conclusion that an optimizer's performance heavily grobemm-depended and that there is no single best optimizer across workloads.
Figures 9 to 12 in the appendix and our conclusions from them suggest an interesting alternative approach for machine learning practitioners: Instead of picking single optimizer and tuning its hyperparameters, trying out multiple default setting optimizers and picking the best one should yield competitive results with less computational and tuning choice efforts. The similarity of those two approaches might be due to the fact that optimizers have implicit learning rate schedules and trying out different optimizers is similar to trying out different (well-tested) schedules (Agarwal et al., 2020).
How much do tuning and schedules help? We consider the final performance achieved by varying budgets and schedules to quantify the usefulness of tuning and applying parameter- free schedules (Figure 3). While there is no clear trend for any individual setting (gray lines), in the median we observe that increasing the budget improves performance, albeit with diminishing returns. For

---

Under review as a conference paper at ICLR 2021
example, using the large budget without any schedule leads median relative improvement of the performance of roughly 3.4 % compared to the default parameters (without schedule).
Similarly, applying parameter-fre free (i.e. untuned) schedule improves median performance. For example, the large tuning budget coupled with a trapezoidal learning rate schedule leads to median relative improvement of roughly 5.3 % compared to the default parameters. However, while these trends hold in the median, their individual effect varies wildly among optimizers and test problems, is apparent from the noisy structure of the individual lines shown Figure 3.
Relative improvement
6.0%
4.0%
2.0%
0.0%
-2.0%
Tuning:
oneshot
oneshot
oneshot
oneshot
small
small
small budget
small budget
large
large
large budget
large
budget
budget
budget
budget
budget
Schedule:
const.
cosine
cosine
trapez.
const.
cosine wr
cosine
trapez.
const.
cosine
cosine ne
trapez.
Figure 3: Lines in gray (-, smoothed by cubic splines for visual guidance only) show the relative
improvement for certain tuning and schedule (compared the one-shor tuning without schedule)
for all 14 optimizers on all eight test problems. The median over all lines is plotted in orange (-) with the shaded area (I) indicating the area between the 25th and 75th percentile.
Which optimizers work well after tuning? Figure 4 compares the optimizers' performance across the test problems. There is no single optimizer that dominates its competitors across all tasks. Never- theless, some optimizers generally perform well, while others vary wildly in their behavior. Further
Quadratic Deep
MNIST VAE
F-MNIST 2c2d
CIFAR-10 3c3d
F-MNIST VAE
CIFAR-100 AII-CNN-C
SVHN Wide ResNet 16-4
Tolstoi Char RNN
27.5
86%
23.0
62.5%
6.0%
969%
62.0%
100
91.5%
61.5%
110
24.0
5.0%
91.0%
80%
61.0%
120
29.5
24.5
45%
944%
90.9%
78%
60.5%
130
30.0
40%
25.0
30.5
90.0%
76%
-9%%
60.0%
140
88.9%
74%
A5.5
59.5%
AMSBound
LA(Mom.)
RMSProp
AMSGrad
Adadelta
Adam
LAIRAdam)
NAG
RAdai
SGD
Figure 4: Mean test set performance over 10 random seeds of all tested optimizers on all eight optimization problems using the large budget for tuning and no learning rate schedule. One standard deviation for the tuned ADAM optimizer is shown with a red error bar (I; error bars for other methods omitted for legibilityy. The performance of untuned ADAM reference. The upper bound of each axis represents the best performance achieved in the benchmark, while the lower bound is chosen in relation the performance of ADAM with default parameters.
and ADABOUND
are marked for
supporting the hypothesis of previous sections, note that taking the best out of small of

---

Under review as a conference paper at ICLR 2021
tuned optimizers - for example, ADAM and ADABOUND frequently results in competitive overall performance, even compared to well-tuned optimizers. Combining these runs with a tuned version of ADAM (or variants thereof) generally yields competitive results in our benchmark. Nevertheless, achieving (or getting close to) the absolute best performance still requires testing multiple optimizers. Which optimizer wins in the end, though, is problem-dependem optimizers that achieve top scores on one problem can perform rather badly on other tasks. We note in passing that the individual optimizer rankings can change when considering e.g, smaller budget or an additional learning rate schedule (see Figures 13 to 15 in the appendix). However, the overall trends described here are consistent.
4 LIMITATIONS
Any empirical benchmark has constraints and limitations. Here highlight some of them and characterize the context within which our results should be considered.
Generalization of the results By using the test problems from DEEPOBS, which span models and data sets of varying complexity, size, and different domains, we aim for generalization. Our results are, despite our best efforts, reflective of not just these setups, but also to the chosen training parameters, the software framework, and further unavoidable choices. The design of our comparisons aims to be close to what an informed practitioners would encounter in practice. It goes without saying that even a carefully curated range of test problems cannot cover all challenges of machine learning or even just deep learning. In particular, our conclusions may not generalize to other types of workloads such as GANs, reinforcement learning, crucial. Similarly, our benchmark does not cover more large-scale problems such as ImageNet (Deng al., 2009) or transformer models (Vaswani et al., 2017) for machine translation. Studying, whether there are systematic differences between these types of optimization problems presents an interesting avenue for further research.
applications where e.g. memory usage is
We don't consider this study the definitive work on benchmark deep learning optimizers, but rather an important step in the right direction. While our comparison includes many "dimensions" of deep learning optimization, e.g. by considering different problems, tuning budgets, and learning rate schedules, there are many more. To keep the benchmark feasible, we chose to use the fixed L2-regularization and batch size that DEEPOBS suggests for each problem. We also did not include optimization techniques such as weight averaging or ensemble methods as they can be combined with all evaluated optimizers. Future works could study how these techniques interact with different optimization methods. However, to keep our benchmark feasible, we have selected what we believe to be the most important aspects affecting an optimizer comparison. We hope, that our study lays the groundwork so that other works can build on it and analyze these questions.
Influence of the hyperparameter search strategy As noted by, egg., Choi et al. (2019) and Sivaprasad et al. (2020), the hyperparameter tuning method, its budget, and its search domain, can significantly affect performance. By reporting results from three different hyperparameter optimization budgets (including the tuning- free one-shot setting) we try to quantify the effect of tuning. We argue that our random search process presents a realistic setting for many but certainly not all deep learning practitioners. One may criticize our approach as simplistic, but note that more elaborate schemes, in particular Bayesian optimization, would multiply the number of design decisions (kernels, search utilities, priors, and scales) and thus significantly complicate the analysis.
The individual hyperparameter sampling distributions significantly affect the relative rankings of the optimizers. A badly chosen search space can make tuning next to impossible. Note, though, that this problem is inherited by practitioners, It is arguably an implicit flaw of an optimizer to not come with well-identified search spaces for its hyperparameters and should thus be reflected in benchmark.
CONCLUSION
Faced with an avalanche of research to develop new stochastic optimization methods, practitioners are left with the near-imposee impossilee task of not just picking method from this wer-growing list, but also to guess or tune hyperparameters for them, even to continuously tune them during optimization. Despite efforts by the community, there is currently method that clearly dominates the competition.
8

---

Under review as a conference paper at ICLR 2021
We have provided an extensive empirical benchmark of optimization methods for deep learning. It reveals structure in the crowded field of optimization for deep learning: First, although many methods perform competitively, subset of methods tends to problems. Secondly, tuning helps about as much as trying other optimizers. Our open data set allows many, more technical observations, e.g., that the stability to re-runs is often overlooked challenge.
up near the top across the spectrum of
Perhaps the most important takeaway from our study is hidden in plain sight: the field is in danger of being drowned by noise. Different optimizers exhibit surprisingly similar performance distribution compared to single method that is re-tuned or simply re-run with different random seeds. It is thus questionable how much insight the development of new methods yields, least if they are conceptually and functionally close to the existing population. We hope that benchmarks like ours can help the community rise beyond inventing yet another optimizer and to focus on key challenges, such as automatic, inner-loop tuning for truly robust and efficient optimization. We are releasing our data to allow future authors to ensure that their method contributes to such ends.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-Scale Machine Learning Heterogeneous Systems, 2015. URL http: /ttessttnnloo..oc.oooo org/. Software available from tensorflow.org.
Naman Agarwal, Rohan Anil, Elad Hazan, Tomer Koren, and Cyril Zhang. Disentangling Adaptive Gradient Methods from Learning Rates. arXiv preprint: 2002. 11 1103, 2020.
James Bergstra and Yoshua Bengio. Random Search for Hyper- Optimization. The Journal of Machine Learning Research, JMIR,, 13, 2012.
Dami Choi, I.ristopher J. Shallue, Zachary Nado, Jaehoon Lee, Chris J. Maddison, and George E. Dahl. On Empirical Comparisons of Optimizers for Deep Learning. arXiv preprint: 1 0. 05446, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: Large-Scale Hierarchical Image Database. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Institute of Electrical and Electronics Engineers (IEEE), 2009.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs. In Advances in Neural Information Processing Systems 31, NeurlPS, 2018.
lan Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, Large Minibatch SGD: Training ImageNet Hour. arXiv preprint: 776.. 02677, 2017.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Wilson Andrew G. Averag- ing weights leads wider optima and better generalization. In Uncertainty Artificial Intelligence Proceedings of the 34th Conference, UAI 2018, 2018.
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic Gradient Descent with Warm Restarts. In 5th International Conference Learning Representations, ICLR, 2017.
Luke Metz, Niru Maheswaranathan, Ruoxi Sun, C. Daniel Freeman, Ben Poole, and Jascha Sohl- Dickstein. Using thousand optimization tasks to learn hyperparameter search strategies. arXiv preprint: 2002 11887, 7, 2020.
Frank Schneider, Lukas Balles, and Philipp Hennig. DeepOBS: A Deep Learning Optimizer Bench- mark Suite. In 7th International Conference on Learning Representations, ICLR, 2019.
Prabhu Teja Sivaprasad, Florian Mai, Thijs Vogels, Martin Jaggi, and Francois Fleuret. Optimizer Benchmarking Needs to Account for Hyperparameter Tuning, In 37th International Conference Machine Learning, ICML, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. In Advances in Neural Information Processing Systems 30, NIPS, 2017.
Ashia C Wilson, Rebecca Roelofs, Mitchell Stern, Nati Srebro, and Benjamin Recht. The Marginal Value of Adaptive Gradient Methods in Machine Learning. In Advances in Neural Information Processing Systems 30, NIPS, 2017.
10

---

Under review as a conference paper at ICLR 2021
David H. Wolpert and William G. Macready. No free lunch theorems for optimization. IEEE Trans. Evol. Comput., 1(1):67-83) 82, 1997.
Chen Xing, Devansh Arpit, Christos Tsirigotis, and Yoshua Bengio. A Walk with SGD, arXiv preprint: 1802. 08770, 2018.
Aston Zhang, Zachary C. Lipton, Mu Li, and Alexander J. Smola. Dive into Deep Learning. 2020. https://dee .ai.

---

Under review as a conference paper at ICLR 2021
A LIST OF OPTIMIZERS AND SCHEDULES CONSIDERED
Table 2: List of optimizers we considered for our benchmark. Note, that this is still far from being a complete list of all existing optimization methods applicable to deep learning, but only a subset, comprising of some of the most popular choices.
Name
Ref.
Name
Ref.
ACClip
(Zhang al., 2020)
LARS
(You 2017)
ADAHESSIAN
(Yao al., 2020)
LookAhead
(Thang al., 2019)
ALI-G
(Berrada et al., 2020)
M-SVAG
(Balles & Hennig. 2018)
AMSBound
(Lao et al., 2019)
MTAdam
(Malkiel Walf, 2020)
AMSGrad
(Reddi
Nesterov
(Nesterov, 1983)
ARSG
(Chen al. 2019b)
Polyak
(Polyak, 1964)
AcceleGrad
(Levy al., 2018)
MVRC-I
(Chen & Zbou, 2020)
AdaAlter
(Xie al., 2019)
MVRC-2
(Chen & Zhou, 2020)
AdaBatch
(Devarukondaed et. 2017)
NAMSB
(Chen al., 2019b)
AdaBayes
(Aitchisom, 2020)
NAMSG
(Chen al., 2019b)
AdaBayes-SS
(Aitchison, 2020)
ND-Adam
(Zhang et 2017)
AdaBlock
(Yun al., 2019)
Nadam
2016)
AdaBound
(Lao al., 2019)
Noisy Adam
(Zhnng 2018)
AdaComp
(Chen et al., 2018)
Noisy K-FAC
(Zhngg al., 2018)
AdaFTRL
(Orabona & Pál. 2015)
NosAdam
(Huang et 2019)
AdaFix
(Bae et al., 2019)
Novograd
(Ginsburg et al., 2019)
AdaFom
(Chen et al., 2019s)
PAGE
(Li al., 20206)
AdaLoss
(Teixeira al., 2019)
PAL
(Mutschler Zoll, 2020)
AdaMax
(Kingma & Ba, 2015)
Padam
(Chen 2020)
AdaMod
(Ding al. 2019)
PolyAdam
(Orvicto al., 2019)
AdaScale
(Johnson et al., 2020)
PowerSGD
(Vogels ct al., 2019)
AdaSGD
(Wang & Wiens, 2020)
PowerSGIMM
(Vogels et al., 2019)
(Zhou al., 2019)
(Mahsereci, 2018)
AdaSgrt
(Hu
PStorn
(Xu, 2020)
AdaX
al., 2020a)
QHAdam
(Ma & Yarats, 2019)
AdaX-W
(Li et al,, 2020a)
QHM
(Ma & Yarats, 2019)
Adadelta
(Zeiler, 2012)
RAdam
(Liu et al., 2020)
Adagrad
(Duchi al., 2011)
RMSProp
(Tieleman Hinton, 2012)
Adam
(Kingma & Ba, 2015)
RMSterov
(Choi 2019)
AdamAL
(Tao al.,
Ranger
(Wright, 2020b)
AdanNC
(Reddi al., 2018)
Rangerl.n
(Grankin, 2020)
AdamP
(Heo al., 2020)
SAMSGrad
(Tong al., 2019)
A
(Zhou et al., 2020)
SAdam
(Wang et al., 2020b)
AdamW
(Loshchilov Hutter, 2019)
SC Adagrad
(Mukkamala Hein, 2017)
AdamX
(Tran Phong, 2019)
SC-RMSProp
(Mukkamala
2017)
Adathm
(Sun al., 2019)
SDProp
(Ida 2017)
ArmijoL.
(Vaswani al.,
SGD
(Robhins & Monro, 1951)
AvaGrad
(Savarese et al., 2019)
SGD-BB
(Tan et al., 2016)
BAdam
(Salas et al., 2018)
SGD-G2
(Ayadi & Turinici, 2020)
BGAdan
(Bai Zhung, 2019)
SGDM
(Liu & Lao, 2020)
BRMSProp
(Aitchison, 2020)
(Heo 2020)
2020)
SGDR
(Losbchilov Hutter, 2017)
C-ADAM
(Tutunov al., 2020)
SHAAdagrad
(Huang 2020)
CProp
(Preechakul & Kijsirikul, 2019)
SKQN
(Yang al., 2020)
Cool Momentum
(Borysenko & Byshkin, 2020)
S4QN
(Yang et al., 2020)
Compositional ADAM
(Tatanov al., 2020)
SNGM
(Zhao et al., 2020)
Curveball
(Menriques al., 2019)
(Wang 2020k)
Dadam
(Nazari al., 2019)
(Koskar
2017)
DeepMemory
(Wright. 2020a)
SWNTS
(Chen 2019c)
DifTGrra
(Duhey et al., 2020)
SADAM
(Tong et al., 2019)
Eve
(Loughi, 2017)
Shampoe
(Anil al., 2020; Gupta et al., 2018)
Gadam
(Zhang Gouza, 2018)
SignAdam++
(Wang et al., 2019t)
GOLS-I
(Kafka
2019)
SignSGD
(Bemssten 2018)
Gravilon
(Kelterborn al.,
SoftAdam
(Fotteeman 2019)
HAdam
(Jiang al.
S-SGD
(Sung 2020)
HyperAuaam
(Wang et al., 2019b)
TAdam
(Ilbouda et al., 2020)
K-BPGS
(Goldfarb et al., 2020)
VAdam
(Khan et al., 2018)
K-BFGSIL)
(GoldBarb et al., 2020)
VR-SGD
(Shang et al., 2020)
KFAC
(Martens Grosse, 2015)
WNGrrd
et 2018)
KFLR
(Botev al.,
YellowFin
(Zhang Mitliagkas, 2019)
KFRA
(Botev al., 2017)
Yogi
(Zabeer 2018)
1.Adam
(Rolinck & Martius, 2018)
vSGD-b
(Schaull et al., 2013)
LAMonentum
(Rolinck Martius, 2018)
vSGD-f
(Schaul & LoCun, 2013)
LAMB
(ou et al., 2020)
vSGD-g
(Schaul et al., 2013)
LaProp
(Zayin al., 2020)
(Schaul 2013)
12

---

Under review as a conference paper at ICLR 2021
Table 3: Overview commonly used parameter schedules. Note, while we list the schedules parameters, it isn't t clearly defined what aspects of a schedule are (tunable) parameters and what is a-priori fixed. In this column, dg denotes the initial learning rate, d% and Cup the lower and upper bound, Al indicates epoch count at which to switch decay styles, & denotes a decaying factor.
Name
Ref.
Illustration
Parameters
Constant
Step Docay
constant factor
Airon k
multi-step
ao. Af...... kin..
Smooth Decay
linear decay
(Goodfellow et al., 2016)
ap. (At, ap)
polynomial decay
k. (0%)
exponential decay
k. (Ct.)
inverse time decay
egg. (Botton, 2012)
ao. k. (ow)
cosine decay
(Loshchilov Hutter, 2017)
ap. (otn)
decay
(Bello al., 2017)
do. (Ow)
MM
Cyclical
triangular
(Smith, 2017)
Otos Oup
triangular
(Smith, 2017)
ases Oup
dlcaay
triangular exponential decay
(Smith, 2017)
Oleo Oup,
cosine
(Loshchilov Hutter, 2017)
Cup, At. (cw)
cosine
(Loshchilov & Hutter, 2017)
aep At, k. (aw)
warm restarts
decay
Warmup
constant warmup
e.g. (Ho et al., 2016)
abs no, At
gradual warmup
(Goyal et al., 2017)
ao, At, (mws
gradual warmup multi step decay
(Goyal et al., 2017)
ao, At, Atsan ki (cpp)
gradual warmup step number decny
(Vaswani et al,, 2017)
ap. (aw)
slanted triangular
(Howard Ruder, 2018)
At, (aw)
long trapezoid
(Xing 2018)
ap, Afg. Atarm (aw)
Super Convergenee
lcycle
(Smith & Topin, 2017)
Oupe At, Atennt (mr)
13

---

Under review as a conference paper at ICLR 2021
B LIST OF OPTIMIZERS SELECTED
Table 4: Selected optimizers for our benchmarking process with their respective color, hyperparame- ters, default values, tuning distributions and scheduled hyperparameters. Here, [U((-,) denotes the log-uniform distribution while ((: denotes the discrete uniform distribution.
Optimizer
Ref.
Parameters
Default
Tuning Distribution
Scheduled
AMSBound
(Loo et al., 2019)
10-3
[u((10=11)
0.1
CM((1 9,0.5)
91
0.9
CV4(0.5, 0.999)
82
0.999
(24(0.8 0.999)
LU(10 4, 4,10 +)
7
10-
AMSGrad
(Reddi et al., 2018)
a
10-2
CM((1-1.1)
91
0.9
(0,(0,5 0.999)
32
0.999
Q 0.999) X
10
AdaBound
(Luo et al., 2019)
10-3
2U(10-1.1)
a1
0.1
CM(10-3 (.00)
91
CU,0,5 0.999)
82
0.999
CM
10
CM(10 1, !)
Adadeltu
(Zeiler, 2012)
10-3
CU(10-1,1)
10
0.95
CM(I0 4,1)
Adugrad
(Duchi et al., 2011)
10
CU(10-4.1)
10
Adam
(Kingma & Ba, 2015)
10
CM(10-1,
91
0.9
CU(055, 0.999)
92
0.999
LV((088 0.999)
Lookabead
(Zhung et al., 2019)
0.5
Cu(10-1.1) 1)
Momentum
a
10
2U(11 (1(10-11)
abbr. LA(Mom.)
1111 20}
0.99
CM(10 4.1)
Lookahead
(Zhang et al., 2019)
0.5
cu(10-4.1)
RAdam abbr. LA(RAdam)
af
10
Lu(le 4.1)
91
0.9
CU
82
0.999
10-7
(11,20)
Momentum
(Polyak, 1964)
10 2
CM(10 4.1)
0.99
CM(10
NAG
(Nesterov, 1983)
10-2
CL(10-1111)
0.99
CU(11 4.1)
NAdam
(Doaat, 2016)
10-3
CM(10-4.1)
81
0.9
(1,
92
0.999
10-7 7
X
RAdam
(Liu al., 2020)
10-3
CU(10-1.1)
31
0.9
(1,00 0.999)
32
0.999
(V((088, 0.999)
10
X
RMSProp
(Tieleman & Hinton, 2012)
10
CU(10 4,1)
10-10
X
1
0.9
(1(10-1.1)
SGD
(Robbins & Monro, 1951)
10
CM(10 4.1)
14

---

Under review as a conference paper at ICLR 2021
C ROBUSTNESS TO RANDOM SEEDS
Data subsampling, random weight initialization, dropout and other aspects deep learning introduce stochasticity to the training process. As such, judging the performance of optimizer on a single run may be misleading due to random fluctuations. In our benchmark we use 10 different seeds of the final setting for each budget in order to judge the stability of the optimizer and the results. However, to keep the magnitude of this benchmark feasible, we only use single seed while tuning, analogously to how a single user would progress. This means that our tuning process can sometimes choose hyperparameter settings which might not even converge for seeds other than the one used for tuning.
Figure 5 illustrates this behavior on an example problem where we used 10 seeds throughout tuning process using grid search. The figure shows that in the beginning performance increases when increasing the learning rate, followed by an area were it sometimes works but other times diverges. Picking hyperparameters from this "danger zone" can lead to unstable results. In this case, where only consider the learning rate, is clear that decreasing the learning rate bit to get away from this "danger zone" would lead to a more stable, but equally well-performing algorithm. In more complicated cases, however, we unable to simple heuristic such as this. This might be the case, for example, when tuning multiple hyperparameters or when the effect of the hyperparameter on the performance is less straight forward. Thus, this is problem not created by improperly using the tuning method, but by an unstable optimization method.
mean repetition seed tuning seed
1.0
Test accuracy
0.8
0.6
0.4
0.2
10-5
10
10-3
10
10-1
10a
10a
10a
Learning rate
Figure 5: Performance of SGD simple multilayer perceptron. For each learning rate, markers in orange (x) show the initial seed which would be used for tuning, blue markers (*) illustrate nine additional seeds with otherwise unchanged settings. The mean over all seeds is plotted as a blue line (-), showing one standard deviation as a shaded area (i).
In our benchmark, we observe in total divergent seeds for the small budget and 56 the large budget, roughly 1% of the runs in each budget. Most of them occur when using SGD (23 and 18 cases for the small and large budget respectively). MOMENTUM (13 and 17 cases for the small and large budget respectively) which might indicate that adaptive methods are less prone to this kind of behavior. For the small budget tuning, none of these cases oceur when using constant schedule (4 for the large budget), and most of them occur when using the cosine with warm restarts schedule (27 and 25 cases for the small and large budget respectively). However, as our data diverging seeds is very limited, it conclusive enough to draw solid conclusions.
NAG (7 and 12 cases for the small and large budget respectively),
15

---

Under review as a conference paper at ICLR 2021
D RE-TUNING EXPERIMENTS
In order to test the stability of our benchmark and especially the tuning method, we selected two optimizers in benchmark and re-tuned them on all test problems second time. We used completely independent random seeds for both tuning and the 10 repetitions with the final setting. Figure and Figure 7 show the distribution of all 10 random seeds for both the original tuning as well as the re- ve-tuning runs for RMSPROP and ADADELTA. It is evident, that re- re-tuning results in a shift of this distribution, since small (stochastic) changes during tuning can result in different chosen hyperparameter setting.
These differences also highlight how crucial is look at multiple test problems. Individually, small changes, such as re-doing the tuning with different seeds can lead to optimization methods changing rankings. However, they tend to average out when looking at an unbiased list multiple problems. These results also further supports the statement made in Section 3 that there is no optimization method clearly domination the competition, as small performance margins might vanish when re-tunnng.
Quadratic Deep
SVHN Wide ResNet 16-4
VAE
2c2d
3c3d
VAE
All-CNN-C
Char RNN
27.5
90
2330
62.5%
60%
-9.20%
84%
62.0%
55595
91..5%
61.5%
110
24.0
91.0%
80%
120
24.5
45
94%
90.599
78%
60.5%
130
30.0
40%
25,0
30.5
9.%
60.0%
140
35%
25.5
59.5%
74%
AMSBound
Adadella
Adarn
LAIRAdam)
NAG
RAdam
RMSProp (2)
Adagrad
LA/Mom.)
Mom.
RMSPrap
AdaBound
Figure 6: Mean test set performance of all 10 seeds of RMSPROP (-) on all eight optimization problems using the small budget for tuning and learning rate schedule. The mean shown with thicker line. We repeated the full tuning process on all eight test problems using different random seeds, which is shown in dashed lines blue (-) The mean performance of all other optimizers is shown in transparent gray lines.
16

---

Under review as a conference paper at ICLR 2021
Quadratic Deep
MNIST VAE
F-MNIST 2c2d
CIFAR-10 3c3d
F-MNIST VAE
CIFAR-100 AII-CNN-C
SVHN Wide ResNet 16-4
Tolstoi Char RNN
27.5
86%
23.0
62.5%
92.00
96%
100
84%
62.0%
28.5
91.39
55%
61.5%
110
50%
93:00
80%
61.0%
120
29.5
24.5
455
39
400 %
25.0
30.5
90.0%
76%
93%
60.0%
89.5%
74%
25.5
59.5%
AMSBound
Adadelta
Adagrad
LA(Mom.)
Mom.
Nadam
RMSPrap
Adadelta
Adam
AdaBound
Figure 7: Mean test set performance of all 10 seeds of ADADELTA (-) on all eight optimization problems using the small budge! for tuning and learning rale schedule. The mean is shown with thicker line. We repeated the full tuning process on all eight test problems using different random seeds, which is shown in dashed lines blue -). The mean performance of all other optimizers is shown in transparent gray lines.
17

---

Under review as a conference paper at ICLR 2021
E
LIST OF SCHEDULES SELECTED
The schedules selected for our benchmark are illustrated in Figure 8. All learning rate schedules are multiplied by the initial learning rate found via tuning or picked as the default choice.
1.0
0.8
Scheduled factor
0.6
0.4
0.2
0.0
20
40
60
80
100
120
140
Epochs
constant
cosine decay
cosine warm restarts
trapezoidal
Figure 8: Illustration of the selected learning rate schedules for a training duration of 150 epochs.
We use cosine decay (Loshchilov & Hutter, 2017) that starts at and decays the form of half period of cosine to 0. As example of cyclical learning rate schedule, test cosine with warm restarts schedule with a cycle length At 10 which increases by a factor of 2 after each cycle without any discount factor. Depending that training stops shortly after one of those warm restarts. Since performance typically declines shortly after increasing the learning rate, we don't report the final performance for this schedule, but instead the performance achieved after the last complete period (just before the next restart). This approach is suggested by the original work of Loshchilov Hutter (2017). However, we still the final performance while tuning.
the number of epochs train model, it is possible
representation schedule including warm-up the trapezoidal schedule from Xing et al. (2018). For our benchmark we set warm-up and cool-down period of 1/10 the training time.
18

---

Under review as a conference paper at ICLR 2021
F IMPROVEMENT AFTER TUNING
When looking at Figure 2, one might realize that few diagonal entries contain negative values. Since diagonal entries reflect the intra-optimizer performance change when tuning the respective task, this might feel quite counterintuitive at first. In theory, this can occur if the respective tuning distributions chosen poorly, the tuning randomness simply got "unlucky". or we observe significantly worse results for our additional seeds (see Figure 5).
If we compare Figures and 10 Figures 11 and 12 we can see most negative diagonal entries vanish or at least diminish in magnitude. For the latter two figures we allow for more tuning runs and only consider the seed that has been used for this tuning process. The fact that the effect of negative diagonal entries reduces is an indication that they mostly result from the two latter reasons mentioned.
Quadratic Deep
MNIST VAE
AMSBound AMSGrad
AdaBound
Adadelta
Adagrad
21
Adam
LA(Mom.)
LA(RAdam)
Mom.
NAG
VOA
VOA
Nadam
RAdam
RMSProp
One-shot
SGD
17A
F-MNIST 2c2d
CIFAR
333d
AMSBound
32
A4
AMSGrad
AdaBound
Adadelta
Adagrad
Adam
LA(Mom.)
LA(RAdam)
Mom.
5
NAG
3
Nadam
RAdam
RMSProp
8
3
SGD
AMSBound
AMSGrad
AdaBound
Adadelta
Adagrad
Adam
LA(Mom.)
LA(RAdam)
Mom.
NAG
Nadam
RAdam
RMSProp
SGD
AMSBound
AMSGrad
AdaBound
Adadelta
Adagrad
Adam
LA(Mom.)
LA(RAdam)
Mom.
NAG
Nadam
RAdam
RMSProp
SGD
Large budget
Figure 9: The absolute test set performance improvement after switching from any untuned optimizer (y-axis, one-shot) to any tuned optimizer (æ-axis, small budget) as an average over random seeds for the constant schedule. This is detailed version of Figure 2 in the main text showing the first four problems.
19

---

Under review as a conference paper at ICLR 2021
F-MNIST VAE
CIFAR-100 AII-CNN-C
AMSBound
AMS Grad
AdaBound
VVSA
Adadelta
Adagrad
Adam
LA(Mom.)
LA(RAdam)
02
254
Mom.
NAG
Nadam
RAdam
RMSProp
One-shot
SGD
SVHN Wide ResNet 16-4
Tolstoi Char RNN
AMSBound
AMSGrad
AdaBound
RFF
60
12
25
25
Adadelta
Adagrad
Adam
LA(Mom.)
LA(RAdam)
227
Mom.
NAG
Nadam
2
2
2
RAdam
2
RMSProp
SGD
8
AMSBound
AMSGrad
AdaBound
Adadelta
Adagrad
Adam
LA(Mom.)
LA(RAdam)
Mom.
NAG
Nadam
RAdam
RMSProp
SGD
AMSBound
AMSGrad
AdaBound
Adadelta
Adagrad
Adam
LA(Mom.)
(A(RAda))
Mom.
NAG
Nadam
RAdam
RMSProp
SGD
Small budget
Figure The absolute test set performance improvement after switching from any untuned optimizer (g-axis, one-shot) to any tuned optimizer (æ-axis, small budget) as average over 10 random seeds for the constant schedule. This a detailed version of Figure the main text showing the last four problems.
20

---

Under review as a conference paper at ICLR 2021
Quadratic Deep
MNIST VAE
AMSBound
MMSGrao
AdaBound
Adadelta
PES
Adagrad
Adam
LA(Mom.)
112
LA(RAdam
120
Mom.
NAG
Nadam
RAdam
PCS
CAN
RMSProp
One-shot
SGD
F-MNIST 2c2d
CIFAR-
333d
AMSBound
AMSGrad
AdaBound
42
Adadelta
Adagrad
Adam
LA(Mom.)
LA(RAdam)
Mom.
NAG
5
Nadam
3
RAdam
RMSProp
SGD
AMSBound
AMSGrad
AdaBound
Adadelta
Adagrad
Adam
LA(Mom.)
LA(RAdam)
Mom.
NAG
Nadam
RAdam
RMSProp
SGD
AMSBound
AMSGrad
AdaBound
Adadelta
Adagrad
Adam
LA(Mom.)
LA(RAdam)
Mom.
NAG
Nadam
RAdam
RMSProp
SGD
Large budget
Figure The absolute test set performance improvement after switching from any untuned optimizer (j-axis, one-shot) to any tuned optimizer (æ-axis, large budget) for the constant schedule. This is structurally the same plot as Figure but comparing to the large budget and only considering the seed that has been used for tuning.
21

---

Under review as a conference paper at ICLR 2021
F-MNIST VAE
CIFAR-100 AII-CNN-C
AMSBound
AMS Grad
AdaBound
Adadelta
113
CO4
123
Adagrad
Adam
LA(Mom.)
1132
LA(RAdam)
252
Mom.
NAG
Nadam
RAdam
RMSProp
One-shot
SGD
SVHN Wide ResNet 16-4
Tolstoi Char RNN
AMSBound
AMSGrad
AdaBound
OF
68
88
Adadelta
Adagrad
Adam
LA(Mom.)
LA(RAdam)
207
Mom.
NAG
3
Nadam
3
RAdam
8
2
RMSProp
SGD
8
AMSBound
AMSGrad
AdaBound
Adadelta
Adagrad
Adam
LA(Mom.)
LA(RAdam)
Mom.
NAG
Nadam
RAdam
RMSProp
SGD
AMSBound
AMSGrad
AdaBound
Adadelta
Adagrad
Adam
LA(Mom.)
LA(RAdam)
Mom.
NAG
Nadam
RAdam
RMSProp
SGD
Large budget
Figure 12: The absolute test set performance improvement after switching from any untuned optimizer (y-axis, one-shot) to any tuned optimizer (æ-axis, large budget) for the constant schedule. This is structurally the same plot as Figure 10 but comparing to the large budget and only considering the seed that has been used for tuning.
22

---

Under review as a conference paper at ICLR 2021
G OPTIMIZER PERFORMANCE ACROSS TEST PROBLEMS
Similarly to Figure 4, we show the corresponding plots for the small budget with no learning rate schedule Figure and the large budger with the cosine and trapezoidal learning rate schedule in Figures 14 and 15. Additionally, in Figure 16 we show the same setting as Figure 4 but showing the training loss instead of the test loss/accuracy.
Quadratic Deep
MNIST VAE
MNIST
CIFAR-10 3c3d
F-MNIST VAE
CIFAR-10D AII-CNN-C
SVHN Wide ResNet 16-4
Toistol Char RNN
2c2d
27.5
86%
23.00
62.5%
6.0%
28
96%
62.0%
5.5
2855
2%
61.5%
110
29.0
24.0
5.0%
801
61.0%
120
29.5
24.5
45%
94%
90.9%
78%
60.5%
130
30.0
30.5
90.0%
76%
-93%
60.0%
140
-35%
31.0
89.5%
74%
25.5
AMSBound
LA(Mem.)
RMSProp
AMSGrad
Adadelta
Adam
LA(RAdam)
NAG
RAdai
SGD
Figure 13: Mean test set performance over 10 random seeds of all tested optimizers on all eight optimization problems using the small budger for tuning and no learning rate schedule. One standard deviation for the tuned ADAM optimizer is shown with a red error bar (I). The performance of the untuned versions of ADAM bound each axis represents the best performance achieved in the benchmark, while the lower bound chosen in relation the performance of ADAM with default parameters.
and ADABOUND
are marked for reference. Note, the upper
The high-level trends mentioned in Section 3 also hold for the smaller tuning budget in Figure 13. Namely, taking the wimning optimizer for several untuned algorithms (here marked for ADAM and AdABound) will result in decent performance in most test problems with much less effort. Adding a tuned version ADAM (or variants thereof) to this selection would result in a very competitive performance. The absolute top-performance however, is achieved by changing optimizers across different test problems.
Note, although the large budget is true superset of the small budget it is given that will always perform better. Our tuning procedure guarantees that the validation performance on the seed that has been used for tuning as least as good on the large budget than on the small budget. But due to averaging over multiple seeds and reporting test performance instead of validation performance, this hierarchy no longer guaranteed. We discuss the possible effects of averaging over multiple seeds further in Appendix C.
The same high-level trends also emerge when considering the cosine or trapezoidal learning rate schedule in Figures 14 and 15. We can also see that the top performance in general increase when adding a schedule (cf. Figure and Figure 15).
Comparing Figure 4 and Figure 16 we can assess the generalization performance of the optimization method not only to an unseen test set, but also to different performance metric (accuracy instead of loss), Again, the overall picture of varying performance across different test problems remains consistent when considering the training loss performance. Similarily to the figures showing test set performance cannot identify clear winner, although ADAM ands its variants, such RADAM perform near the top consistently. Note that while Figure 16 shows the training loss, the optimizers have still be tuned achieve the best validation performance (i.e. accuracy if available, else the loss).
23

---

Under review as a conference paper at ICLR 2021
Quadratic
CIFAR-10
F-MNIST
Tolstoi
90
62.5%
28.0
92.0
9699
2.4%
62.0%
100
23-5
28.5
51.5%
22%
61.5%
110
95
29.0
24.0
5.0%
91.0%
61.0%
29.5
944
83%
60.5%
44%
25.0
30.5
90.0%
966
93%
60.0%
355
31.0
89.5%
25.5
59.5%
AMSBound
AdaBcund
Adagrad
LA(Mom.)
Mem.
Nadam
RMSProp
AMSGrad
Adadelta
Adam
LA(RAdam)
NAG
RAdam
SGD
Figure 14: Mean test performance over 10 random seeds of all tested optimizers all eight optimization problems using the large budget for tuning and the cosine learning rate schedule. One standard deviation for the tuned ADAM optimizer is shown with red error bar (I). The performance of the untuned versions of ADAM (V) and BOBNO (A) are marked for reference (this time with the cosine learning rate schedule). Note, the upper bound of each axis represents the best performance
achieved in the benchmark, while the lower bound is chosen in relation to the performance of ADAM with default parameters (and no schedule).
Quadratic Deep
CIFAR-10 3c3d
F-MNIST
90
81536
62.5%
6.0%
24.0
02.0%
9999
44%
62.0%
100
28.3
91.5%
55%
8286
61.5%
110
9305
29.0
24
91.0%
61.00
29.5
949
90.5%
78%
60.5%
30.0
40%
25.0
30.5
90.0%
76%
93%
60.0%
-35%
31.0
89.5%
74%
25.5
59.5%
AMSBound
AdaBound
Adagrad
LAIMom.)
Mom.
Nadem
RMSProp
AMSGrad
Adadelta
Adam
LAIRAdam)
NAG
RAdam
SGD
Figure 15: Mean test performance over 10 random seeds of all tested optimizers all eight optimization problems using the large budget for tuning and the trapezoidal learning rate schedule. One standard deviation for the tuned ADAM optimizer is shown with red eтror bar (I). The performance of the untuned versions of ADAM (V) and ADABOUND (A) are marked for reference (this time with the trapezoidal learning rate schedule). Note, the upper bound of each axis represents the best performance achieved in the benchmark, while the lower bound is chosen in relation to the performance of ADAM with default parameters (and no schedule).
24

---

Under review as a conference paper at ICLR 2021
Quadratic Deep
MNIST VAE
F-MNIST 2c2d
CIFAR-10 3c3d
F-MNIST VAE
CIFAR-100 AII-CNN-C
SVHN Wide ResNet 16-4
Tolstol Char RNN
26.5
22.5
0.005
0..5
1.14
100
23
0.015
0.
23.5
1.18
38.0
0.020
24.0
120
0.9
2.5
0.3
1.22
29.0
0.025
24.5
1.0
1.24
3.0
0.030
-0.4
1.1
25.0
1.26
140
0.035
3.5
AMSBound
AdaBound
Adagrad
LA[Mom.J
Mom.
Nadam
RMSProp
Adam
LAIRAdam)
Figure 16: Mean training loss performance over 10 random seeds of all tested optimizers on all eight optimization problems using the large budget for tuning and no learning rate schedule. One standard deviation for the tuned ADAM optimizer is shown with red error bar (I). The performance of the untuned versions of ADAM (V) and ADA BOUND (4) are marked for reference. Note, the upper bound of each axis represents the best performance achieved in the benchmark, while the lower bound is chosen in relation to the performance of ADAM with default parameters (and no schedule).
H TABULAR VERSION
Table 5: Tabular version of Figure 4. Mean test performance and standard deviation 10 random seeds of all tested optimizers on all eight optimization problems using the large budger for tuning and learning rate schedule. For comprehensability, mean and standard deviation are rounded.
Quadratic Deep
F-MNIST
F-NNIST
CIFAR-100
AMSHound
91.02 1.9
28.12 0.1
92.09 +: 0.2
83.22 100
29.11 0.1
55.54 $354-:000
95.52 + 022
5933 + 0.4
AMSGGmd
86
52.55
Adaaouund
95.39 9.1
28.10 0.1
92.11 +
83.28 100
23.39
55.78
95.39 02
59.48
Adadetaa
120.7 0.2
27.91 0.0
82.55 09
2322 0.0.
47.99 0:08
03
59.33
Adagrad
88.39 =:
27.94 4:
92.04 0.2
83.54
+ 1.0
23.18 4.1
27.89 2
95.12 1112:000
62,02 0.1
Adamn
88.35 4: 5.7
21.77 0.0
0.03
83332
2307 0.0.1
$492
0:
0.1.1
LA(Mom.)
87.15 0:
55.20 0.9
88.56 1:
82.34 1: 1.0
25.38 1.03
13:37 1
95.67 1:67::001
6021 03
LA Adam)
87,88 00 5.8
34.28 9.4
0.2
0.7
2:.84
55.84 1.4
95.74:1:23
61.72:0011
Momentum
87.03
35.95 11.
90.15 0:3
83.23 0.
24.33
$5.86
95.55 95
6191 0.1
NAG
87.08
36.21
0.3
83.40 0.66
23.86:20.11
57
05
62.00 00
88.19.5577
27.72
04
1.022:200
95
1000
RAdam
88.18
27.82 0.1
91.66 0.26
0:06
23.12.20.111
44.55110
94
01
RMSPiop
87.40
27.83
88.84 0.6
81.51
23.18 0.1
94.41 44.1:00
SGD
37.98 6.5
36.15 10.
91.89 +
92.0 + 1.3
2354 +
52.89 +.1
95.11 1.11::00
61.28 01
25

---

Under review as a conference paper at ICLR 2021
APPENDIX REFERENCES
Laurence Aitchison. Bayesian filtering unifies adaptive and non-adaptive neural network optimization methods. In Advances in Neural Information Processing Systems 33, NeurIPS, 2020.
Rohan Anil, Vineet Gupta, Tomer Koren, Kevin Regan, and Yoram Singer. Second Order Optimization Made Practical. arXiv preprint: 2002. 09018 8, 2020.
Imen Ayadi and Gabriel Turinici. Stochastic Runge- Kutta methods and adaptive SGD-G2 stochastic gradient descent. arXiv preprint: 2002. 09304, 2020.
Kunge-Kutta
Kiwook Bae, Heechang Ryu, and Hayong Shin. Does Adam optimizer keep close to the optimal point?. arXiv preprint: 1911. 1911.00289, 2019.
Jiyang Bai and Jiawei Zhang. BGADAM: Boosting based Genettic ADAM for Convolu- tional Neural Network Optimization. arXiv preprint: 1908. 08015, 2019.
Lukas Balles and Philipp Hennig. Dissecting Adam: The Sign, Magnitude and Variance Stochastic Gradients. In 35th International Conference on Machine Learning, ICML, 2018
Irwan Bello, Barret Zoph, Vijay Vasudevan, and Quoc V. Le. Neural Optimizer Search with Reinforcement Learning. In 34th International Conference on Machine Learning, ICML, 2017.
Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar. SIGNSGD: Compressed Optimisation for Non-Convex Problems. In 35th International Con- ference on Machine Learning, ICML, 2018.
Leonard Berrada, Andrew Zisserman, and M. Pawan Kumar. Training Neural Networks for and by Interpolation. In 37th International Conference on Machine Learning, ICML, 2020.
Oleksandr Borysenko and Maksym Byshkin. CoolMomentum: Method for Stochastic Optimization by Langevin Dynamics with Simulated Annealing, arXiv preprint: 2005. I 4 4655, 2020.
Aleksandar Botev, Hippolyt Ritter, and David Barber. Practical Gauss-Newton Optimisation for Deep Learning. 34th International Conference on Machine Learning, ICML, 2017.
Léon Bottou. Stochastic gradient descent tricks. In Neural networks: Tricks of the trade. Springer, 2012.
Chia-Yo Yu Chen, Jungwook Choi, Daniel Brand, Ankur Agrawal, Wei Zhang, and Kailash Gopalakrish- nan. AdaComp: Adaptive Residual Gradient Compression for Data Iata-Parallel Distributed Training. In 32nd AAAI Conference Artificial Intelligence, AAAI, 2018.
Jinghui Chen, Dongruo Zhou, Yiqi Tang. Ziyan Yang, Yuan Cao, and Quanquan Gu. Closing the generalization gap of adaptive gradient methods in training deep neural networks. In 29th International Joint Conference on Artificial Intelligence, IJCAI, 2020.
Xiangyi Chen, Sijia Liu, Ruoyu Sun, and Mingyi Hong. On the Convergence of Class of Adam- Type Algorithms for Non-Convex Optimization. In 7th International Conference on Learning Representations, ICLR, 2019a.
Yushu Chen, Hao Jing, Wenlai Zhao, Zhiqiang Liu, Ouyi Li, Liang Qiao, Wei Xue, Haohuan Fu, and Guangwen Yang. An Adaptive Remote Stochastic Gradient Method for Training Neural Networks. arXiv preprint: 1905. 01422, 2019b.
Yushu Chen, Hao Jing, Wenlai Zhao, Zhiqiang Liu, Liang Qiao, Wei Xue, Haohuan Fu, and Guangwen Yang. NAMSG: An Efficient Method For Training Neural Networks. arXiv preprint: 1905. 01 2019c.
Ziyi Chen and Yi Zhou. Momentum with Variance Reduction for Nonconvex Composition Optimiza- tion. arXiv preprint: 2005. 07755. 2020.
Dami Choi, Christopher J. Shallue, Zachary Nado, Jachoon Lee, Chris J. Maddison, and George E. Dahl. On Empirical Comparisons of Optimizers for Deep Learning, arXiv preprint: 1910. 05446, 2019.
26

---

Under review as a conference paper at ICLR 2021
Aditya Devarakonda, Maxim Naumov, and Michael Garland. AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks. arXiv preprint: 1712. 02029, 2017.
Jianbang Ding, Xuancheng Ren, Ruixuan Luo, and Xu Sun. An Adaptive and Momental Bound Method for Stochastic Learning. arXiv preprint: 1910. 12249, 2019.
Timothy Dozat. Incorporating Nesterov Momentum into Adam. In 4th International Conference on Learning Representations, ICLR, 2016.
Shiv Ram Dubey, Soumendu Chakraborty, Swalpa Kumar Roy, Snehasis Mukherjee, Satish Kumar Singh, and Bidyut Baran Chaudhuri. diffGrad: An Optimization Method for Convolutional Neural Networks. IEEE Transactions on Neural Networks and Learning Systems, 2020.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive Subgradient Methods for Online Learning and Stochastic Optimization. Journal of Machine Learning Research, JMLR, 12, 2011.
Abraham J. Fetterman, Christina H. Kim, and Joshua Albrecht. SoftAdam: Unifying SGD and Adam for better stochastic gradient descent, 2019.
Boris Ginsburg. Patrice Castonguay, Oleksii Hrinchuk, Oleksii Kuchaiev, Vitaly Lavrukhin, Ryan Leary, Jason Li, Huyen Nguyen, and Jonathan M. Cohen. Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks. arXiv preprint: 1 1905. 11286, 2019.
Donald Goldfarb, Yi Ren, and Achraf Bahamou. Practical Quasi-Newton Methods for Training Deep Neural Networks. In Advances Neural Information Processing Systems 33, NeurIPS, 2020.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour. arXiv preprint: 1 1706. 02677, 2017.
Mikhail Grankin. Rangerl https:: /gttttbtttttu..go wttp.///ggaaiii//oeerroooo 2020.
Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned Stochastic Tensor Optimization. In 35th International Conference on Machine Learning. ICML, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In IEFE Computer Society Conference Computer Vision and Pattern Recognition, 2016.
João F. Henriques, Sébastien Ehrhardt, Samuel Albanie, and Andrea Vedaldi. Small Steps and Giant Leaps: Minimal Newton Solvers for Deep Learning. In IEEE/CVF International Conference on Computer Vision, ICCV, 2019.
Byeongho Heo, Sanghyuk Chun, Seong Joon Oh, Dongyoon Han, Sangdoo Yun, Youngjung Uh, and Joog-Woo Ha. Slowing Down the Weight Norm Increase in Momentum- Optimizers. arXiv preprint: 2006 0821782277 7, 2020.
Jeremy Howard and Sebastian Ruder. Universal Language Model Fine-tuning for Text Classification. In 56th Annual Meeting of the Association for Computational Linguistics, 2018
Yifan Hu, Siqi Zhang, Xin Chen, and Niao He. Biased Stochastic First-Order Methods for Conditional Stochastic Optimization and Applications in Meta Learning. Advances in Neural Information Processing Systems 33, NeurIPS, 2020.
Yuzheng Hu, Licong Lin, and Shange Tang. Second-order Information in First-order Optimization Methods, arXiv preprint: 1912. 09926, 2019.
Haiwen Huang, Chang Wang, and Bin Dong. Nostalgic Adam: Weighting More of the Past Gradients When Designing the Adaptive Learning Rate. In 28th International Joint Conference on Artificial Intelligence, IJCAI, 2019.
27

---

Under review as a conference paper at ICLR 2021
Xunpeng Huang, Hao Zhou, Runxin Xu, Zhe Wang, and Lei Li. Adaptive Gradient Methods Can Be Provably Faster than SGD after Finite Epochs. arXiv preprint: 2006. 07037, 2020.
Yasutoshi Ida, Yasuhiro Fujiwara, and Sotetsu Iwamura. Adaptive Learning Rate via Covariance Matrix Based Preconditioning for Deep Neural Networks. In 26th International Joint Conference on Artificial Intelligence, IJCAI, 2017.
Wendyam Eric Lionel Ilboudo, Taisuke Kobayashi, and Kenji Sugimoto. TAdam: A Robust Stochastic Gradient Optimizer. arXiv preprint: 2003. 001 79, 2020.
Zhanhong Jiang, Aditya Balu, Sin Yong Tan, Young M Lee, Chinmay Hegde, and Soumik Sarkar. On Higher-order Moments in Adam. arXiv preprint: 1910. 06878, 2019.
Tyler B. Johnson, Pulkit Agrawal, Haijie Gu, and Carlos Guestrin. AdaScale SGD: A User-Friendly Algorithm for Distributed Training, 2020.
Dominic Kafka and Daniel Wilke. Gradient-only line searches: An Alternative Probabilistic Line Searches. arXiv preprint: 1 1903. 09383, 2019.
Chad Kelterborn, Marcin Mazur, and Bogdan V. Petrenko. Gravilon: Applications of a New Gradient Descent Method to Machine Learning. arXiv preprint: 2008. 11370, 2020. 2020.
Nitish Shirish Keskar and Richard Socher. Improving Generalization Performance by Switching from Adam to SGD. arXiv preprint: 1712. 07628, 2017.
Mohammad Emtiyaz Khan, Didrik Nielsen, Voot Tangkaratt, Wu Lin, Yarin Gal, and Akash Srivastava. Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam. In 35th International Conference Machine Learning, ICML, 2018.
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In 3rd Interna- tional Conference on Learning Representations, ICLR, 2015.
Kfir Yehuda Levy, Alp Yurtsever, and Volkan Cevher. Online Adaptive Methods, Universality and Acceleration. In Advances in Neural Information Processing Systems 31, NeurlPS, 2018.
Wenjie Li, Zhaoyang Zhang, Xinjiang Wang, and Ping Luo. AdaX: Adaptive Gradient Descent with Exponential Long Term Memory. arXiv preprint: 2004. 09740, 2020a.
Zhize Li, Hongyan Bao, Xiangliang Zhang, and Peter Richtárik. PAGE: A Simple and Optimal Probabilistic Gradient Estimator for Nonconvex Optimization. arXiv preprint: 2008. 10898, 2020b.
Liang Liu and Xiaopeng Luo. A New Accelerated Stochastic Gradient Method with Momentum. arXiv preprint: 2006. 00423, 2020.
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. In 8th International Conference Learning Representations, ICLR, 2020.
Pietro Longhi. Wall crossing invariants from spectral networks. Annales Henri Poincaré, 19(3), 2017.
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic Gradient Descent with Warm Restarts, In 5th International Conference on Learning Representations, ICIR, 2017.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR, 2019.
Liangchen Luo, Yuanhao Xiong, Yan Liu, and Xu Sun. Adaptive Gradient Methods with Dynamic Bound of Learning Rate. In 7th International Conference on Learning Representations, ICLR, 2019.
Jerry Ma and Denis Yarats. Quasi-hyyerbolic momentum and Adam for deep learning. In 7th International Conference Learning Representations, ICLR, 2019.
28

---

Under review as a conference paper at ICLR 2021
Maren Mahsereci. Probabilistic Approaches to Stochastic Optimization. Ph.D. Thesis, University of Tuebingen, 2018.
Itzik Malkiel and Lior Wolf. MTAdam: Automatic Balancing of Multiple Training Loss Terms. arXiv preprint: 2006 1 4 4683, 2020.
James Martens and Roger Grosse. Optimizing Neural Networks with Kronecker-Pactored Approxi- mate Curvature. In 32nd International Conference on Machine Learning, ICML, 2015.
Mahesh Chandra Mukkamala and Matthias Hein. Variants of RMSProp and Adagrad with Logarithmic Regret Bounds. In 34th International Conference on Machine Learning. ICML, 2017.
Maximus Mutschler and Andreas Zell. Parabolic Approximation Line Search for DNNs. In Advances in Neural Information Processing Systems 33, NeurlPS, 2020.
Parvin Nazari, Davoud Ataee Tarzanagh, and George Michailidis. DADAM: A Consensus-based Distributed Adaptive Gradient Method for Online Optimization. arXiv preprint: 1901. 09109, 2019.
Yurii Nesterov. A method for solving the convex programming problem with convergence rate O(1/k2)). Soviet Mathematics Doklady, 27, 1983.
Francesco Orabona and Dávid Pál. Scale-Free Algorithms for Online Linear Optimization. In Algorithmic Learning Theory- 26th International Conference, ALT, 2015.
Antonio Orvieto, Jonas Köhler, and Aurélien Lucchi. The Role of Memory in Stochastic Optimization. In 35th Conference on Uncertainty in Artificial Intelligence, UAI, 2019.
B. T. Polyak. Some methods of speeding up the convergence of iteration methods. USSR Computa- tional Mathematics and Mathematical Physics, 4(5), 1964.
Konpat Preechakul and Boonserm Kijsirikul, CProp: Adaptive Learning Rate Scaling from Past Gradient Conformity. arXiv preprint: 1 1912. 11491, 2019.
Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. On the Convergence of Adam and Beyond. In 6th International Conference on Learning Representations, ICLR, 2018.
Herbert Robbins and Sutton Monro. A Stochastic Approximation Method. The Annals of Mathemati- cal Statistics, 22(3), 1951.
Michal Rolinek and Georg Martius. L4: Practical loss-based stepsize adaptation for deep learning. In Advances in Neural Information Processing Systems 31, NeurlPS, 2018.
Arnold Salas, Samuel Kessler, Stefan Zohren, and Stephen Roberts. Practical Bayesian Learning of Neural Networks via Adaptive Subgradient Methods. arXiv preprint: 1811. 021.036779 2018.
Pedro Savarese, David McAllester, Sudarshan Babu, and Michael Maire. Domain-independent Dominance of Adaptive Methods. arXiv preprint: 1912. 01 0283,,13000 2019.
Tom Schaul and Yann LeCun. Adaptive learning rates and parallelization for stochastic, sparse, non-smooth gradients. In 1st International Conference on Learning Representations, ICLR, 2013.
Tom Schaul, Sixin Zhang, and Yann LeCun. No more pesky learning rates. In 30th International Conference on Machine Learning, ICML, 2013.
Fanhua Shang. Kaiwen Zhou, Hongying Liu, James Cheng, Ivor W. Tsang, Lijun Zhang, Dacheng Tao, and Licheng Jiao. VR-SGD: A Simple Stochastic Variance Reduction Method for Machine Learning. IEEE Trans. Knowl. Data Eng., 32(1), 2020.
Leslie N. Smith. Cyclical Learning Rates for Training Neural Networks. In IEEE Winter Conference on Applications of Computer Vision, WACV, 2017.
Leslie N. Smith and Nicholay Topin. Super-Convergencee Very Fast Training of Neural Networks Using Large Learning Rates, arXiv preprint: 1 1708. 07120, 2017.
29

---

Under review as a conference paper at ICLR 2021
Huikang Sun, Lize Gu, and Bin Sun. Adathm: Adaptive Gradient Method Based Estimates of Third-Order Moments. In 4th IEEE International Conference on Data Science in Cyberspace, DSC, 2019.
Wonyong Sung, Iksoo Choi, Jinhwan Park, Seokhyun Choi, and Sungho Shin. S-SGD: Symmetrical Stochastic Gradient Descent with Weight Noise Injection for Reaching Flat Minima. arXiv preprint: 2009. 02479, 2020.
Conghui Tan, Shiqian Ma, Yu-Hong Dai, and Yuqju Qian. Barzilai-Borwein Step Size for Stochastic Gradient Descent. In Advances in Neural Information Processing Systems 29, NIPS, 2016.
Zeyi Tao, Qi Xia, and Qun Li. A new perspective in understanding of Adam-Type algorithms and beyond, 2019.
Brian Teixeira, Birgi Tamersoy, Vivek Singh, and Ankur Kapoor. Adaloss: Adaptive Loss Function for Landmark Localization. arXiv preprint: 908. 01070, 2019.
Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5 RMSProp: Divide the gradient by a running average of its recent magnitude, 2012.
Qianqian Tong, Guannan Liang, and Jinbo Bi. Calibrating the Adaptive Learning Rate Improve Convergence of ADAM. arXiv preprint: 1908. 70000,000000000000000000000000.0 2019.
Phuong Thi Tran and Le Trieu Phong. On the Convergence Proof of AMSGrad and a New Version. IEEE Access, 7, 2019.
Rasul Tutunov, Minne Li, Alexander Gowen-Rivers, Jun Wang, and Haitham Bou-Ammar, Compo- sitional ADAM: An Adaptive Compositional Solver. arXiv preprint: 2002. 03755, 2020.
2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. In Advances in Neural Information Processing Systems 30, NIPS, 2017.
Sharan Vaswani, Aaron Mishkin, Issam Laradji, Mark Schmidt, Gauthier Gidel, and Simon Lacoste-Julien. Painless Stochastic Gradient: Interpolation, Line-Search. and Convergence Rates. In Advances in Neural Information Processing Systems 32, NeurlPS, 2019.
Thijs Vogels, Sai Praneeth Karimireddy, and Martin Jaggi. PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization. In Advances in Neural Information Processing Systems 32, NeurlPS, 2019.
Bao Wang, Tan M. Nguyen, Andrea L. Bertozzi, Richard G. Baraniuk, and Stanley J. Osher. Scheduled restart momentum for accelerated stochastic gradient descent. arXiv preprint: 2002. 1583,, 2020a.
Dong Wang, Yicheng Liu, Wenwo Tang, Fanhua Shang, Hongying Liu, Qigong Sun, and Licheng Jiao. SignADAM++: Learning Confidences for Deep Neural Networks. In International Conference on Data Mining Workshops, ICDM, 2019a.
Guanghui Wang, Shiyin Lu, Quan Cheng, Weiwei Tu, and Lijun Zhang. SAdam: Variant of Adam for Strongly Convex Functions. In 8th International Conference on Learning Representations, ICLR, 2020b.
Jiaxuan Wang and Jenna Wiens. AdaSGD: Bridging the gap between SGD and Adam. arXiv preprint: 2006. 1654, 2020.
Shipeng Wang, Jian Sun, and Zongben Xu. HyperAdam: A Learnable Task-Adaptive Adam for Network Training, In 33rd AAAI Conference on Artificial Intelligence, AAAI, 2019b.
Less Wright.
Deep Memory:
https://github. wttp:///....mmmmss22220000000000 2020a.
Seee-DDee-Learatn Cap--pptimme
Less
Wright.
Ranger.
https://github..com comp://woo/ccmmmes//0/000000
Rarger--Dee--eaaammn
2020b.
30

---

Under review as a conference paper at ICLR 2021
Xiaoxia Wu, Rachel Ward, and Léon Bottou. WNGrad: Learn the Learning Rate in Gradient Descent. arXiv preprint: 1803. 02865, 2018.
Cong Xie, Oluwasanmi Koyejo, Indranil Gupta, and Haibin Lin. Local AdaAlter: Communication- Efficient Stochastic Gradient Descent with Adaptive Learning Rates. arXiv preprint: 1911. 09030, 2019.
Chen Xing, Devansh Arpit, Christos Tsirigotis, and Yoshua Bengio. A Walk with SGD, arXiv preprint: 1802. 08770, 2018.
Yangyang Xu. Momentum- based variance-reduced proximal stochastic gradient method for composite nonconvex stochastic optimization. arXiv preprint: 2006. 00425, 2020.
Minghan Yang, Dong Xu, Yongfeng Li, Zaiwen Wen, and Mengyun Chen. Structured Stochastic Quasi-Newton Methods for Large-Scale Optimization Problems. arXiv preprint: 2006. 09606, 6, 2020.
Zhewei Yao, Amir Gholami, Sheng Shen, Kurt Keutzer, and Michael W. Mahoney, ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning arXiv preprint: 2006. 00719, 2020.
Yang You, Igor Gitman, and Boris Ginsburg. Large Batch Training of Convolutional Networks. arXiv preprint: 1 7778.0. 03888, 2017.
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large Batch Optimization for Deep Learn- ing: Training BERT in 76 minutes. In 8th International Conference on Learning Representations, ICLR, 2020.
Jihun Yun, Aurelie C. Lozano, and Eunho Yang. Stochastic Gradient Methods with Block Diagonal Matrix Adaptation. arXiv preprint: 1905. 10757. 2019.
Manzil Zaheer, Sashank J. Reddi, Devendra Singh Sachan, Satyen Kale, and Sanjiv Kumar. Adaptive Methods for Nonconvex Optimization. In Advances in Neural Information Processing Systems 31, NeurlPS, 2018.
Matthew D. Zeiler. ADADELTA: An Adaptive Learning Rate Method, arXiv preprint: 1212.5701. 2012.
Guodong Zhang, Shengyang Sun, David Duvenaud, and Roger Grosse. Noisy Natural Gradient as Variational Inference. In 35th International Conference on Machine Learning, ICML, 2018.
Jian Zhang and Ioannis Mitliagkas. YellowFin and the Art of Momentum Tuning. In Machine Learning and Systems, MLSys, 2019.
Jiawei Zhang and Fisher B. Gouza. GADAM: Evolutoorry ADAM for Deep Neural Network Optimization, arXiv preprint: 1805. 07500, 2018.
Jingzhao Zhang, Sai Pranceth Karimireddy, Andreas Veit, Seungycon Kim, Sashank Reddi, Sanjiv Kumar, and Suvrit Sra. Why are adaptive methods good for attention models? In Advances in Neural Information Processing Systems 33, NewrlPS, 2020.
Michael R. Zhang, James Lucas, Geoffrey Hinton, and Jimmy Ba. Lookahead Optimizer: k steps forward, step back. Advances in Neural Information Processing Systems 32, NeurIPS, 2019.
Zijun Zhang, Lin Ma, Zongpeng Li, and Chuan Wu. Normalized Direction-preserving Adam. arXiv preprint: 1709. 0454 6, 2017.
Shen-Yt Yi Zhao, Yin-Peng Xie, and Wu-Jun Li. Stochastic Normalized Gradient Descent with Momen- tum for Large Batch Training. arXiv preprint: 2007. 13985, 2020.
Bingxin Zhou, Xuebin Zheng, and Junbin Gao. ADAMT: A Stochastic Optimization with Trend Correction Scheme. arXiv preprint: 2001 06130, 2020.
31

---

Under review as a conference paper at ICLR 2021
Zhiming Zhou, Qingru Zhang, Guansong Lu, Hongwei Wang. Weinan Zhang, and Yong Yu. AdaShift: Decorrelation and Convergence of Adaptive Learning Rate Methods. In 7th International Confer- ence on Learning Representations, ICLR, 2019.
Liu Ziyin, Zhikang T. Wang, and Masahito Ueda. LaProp: Better Way to Combine Momentum with Adaptive Gradient, arXiv preprint: 2002. 04839, 2020.
32


Reviews:
[{"review_id": "uwB9TctXzfq", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper benchmarks popular optimizers for training neural networks. The experiments consider all possible combinations of 3 different tuning budgets, and 4 different fixed learning rate schedules on 8 deep learning workloads for 14 optimizers. The paper highlights two main observations: 1) there is no clear dominating optimizer, and 2) selecting from a pool of optimizers with their default parameters is often as good as tuning a fixed optimizer.\nThe main contribution of this work is the open-sourced experiment results for a multitude of cases which can serve as a baseline for future research in optimizers for deep learning.\n\nStrengths:\n- The biggest strength of this work is that all the details regarding the benchmarking protocol are presented and justified. In addition to this, the caveats of the protocol are stated explicitly. The explicit and transparent nature of this work can help practitioners make more informed decisions, and prevent them from misinterpreting the results to something more than what is presented.\n- To my knowledge, this is the first paper to compare a large number of optimizers, selected based on popularity in the research community.\n- The problems considered are of varying difficulty, and includes tasks other than image classification. \n- Different levels of tuning budgets are considered, with the smallest budget being 1 trial (evaluating the default setting), and the largest being 50 trials.\n\nWeaknesses\n- The current tuning procedure is unstable. Tuning with the seed fixed is like optimizing for the specific seed. Furthermore, the hyperparameters that produce the best performance for a specific seed tends to be more unstable (which the authors agree to in appendix C); evaluating such an unstable setting on different seeds unnecessarily penalizes the optimizer. What makes more sense to me is to tune with the same number of trials, with each trial having a different seed, and using bootstrapping to compute the statistics (mean, standard deviation, etc). What we want to see with the tuning experiments is how well the optimizer can do (as an upper bound), and how stable it is. With the current approach, it’s hard to observe the best performance, at which point, I’m not sure how meaningful the error measurements are. All in all, I think it’s more meaningful to show how varying the best optimizer performance can be when tuning with a different set of seeds (since everyone uses different seeds), than to show the variance of a specific set of hyperparameters that is most likely unstable, on many seeds.\n- I believe the experiments lack results for the “well-tuned” case. The optimizers all use a fixed hyperparameter search range for all problems, which can’t be competitive over different tasks of varying difficulty. I understand that this study assumes the model practitioner to be someone who doesn’t have prior knowledge about the optimizer, let alone the search ranges. However, I think it’s reasonable to believe that a practitioner would try to verify the search range by testing some hyperparameter values before committing 25 or 50 trials to the search range. Likewise, it would be useful to see results with a more calibrated search space per test problem. This can be done with the 50 trial budget by, for example, using 25 on a wide search space, and the other 25 on a more refined search space. At the very least, it would be useful to see the performance vs hyperparameter value plotted for the existing experiments to see whether the ranges could have been trivially improved (for example, if the performance tends to increase/decrease with the learning rate, but the best performance lied on the boundary of the search space, the range could have been shifted). This sort of tuning procedure is not unknown in the community. See [1, 2, 3].\n\nCurrently, I think the weaknesses outweigh the strengths of the paper. It is my understanding that optimizer comparisons should be done between reasonably good versions of the optimizers, and I think better versions of the optimizers could have been presented with a different methodology, given the same computational budget.\n \n\n\n[1] Wilson, Ashia C., et al. \"The marginal value of adaptive gradient methods in machine learning.\" Advances in neural information processing systems. 2017.\n\n[2] Shallue, Christopher J., et al. \"Measuring the effects of data parallelism on neural network training.\" arXiv preprint arXiv:1811.03600 (2018).\n\n[3] Choi, Dami, et al. \"On empirical comparisons of optimizers for deep learning.\" arXiv preprint arXiv:1910.05446 (2019).\n\n\nUpdate:\n\nI have read over the changes made by the authors, and also the other reviewer’s responses. I am maintaining my score, because I don’t think the current version of the paper is enough of a contribution to get accepted. As mentioned in my responses below, I would be happy to accept a future version of the paper that addresses my comments above. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "WG5IBU04WgD", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overview:\nOverall I believe this paper is extremely well written and organized. The introduction and limitations are very useful groundings of optimization research, and I hope that the community reads this paper and internalizes its message of making more meaningful research in optimization (instead of yet-another-Adam-variant!).\n\nIn a previous comment I brought up a serious concern about the tuning ranges of momentum-like parameters, which I believe could bias the results towards optimizers that were tuned with ranges whose lower end was 0.5 (Adam, AMSBound, AMSGrad, AdaBound, LA(RAdam), NAdam, RAdam). Besides this concern, I overall would normally argue for a very strong acceptance, but until this is corrected I am unsure I can recommend accepting. I look forward to discussing correcting this with the authors however, and am willing to dramatically raise my score!\n\n**NOTE: updated score after seeing author replies and updated draft. I believe that this work is exemplary in terms of being careful about baseline construction, something that is unfortunately too often overlooked in our field. Additionally, it rigorously highlights another important point that I believe many often overlook, that \"there are now enough optimizers\"; community effort should be diverted from introducing small variations around Adam and instead invest focus on more meaningful improvements in scaling machine learning optimization. I do still believe that ImageNet and a larger transformer experiment would be extremely valuable to add to a later version, and hope the authors can eventually secure the computing power to add these.**\n\nPros:\n-Table 2 is extremely useful and I think should be additionally put on a GitHub where it can be updated with links to papers.\n-The authors clearly put a lot of careful thought into how to study and present these results, taking into account numerous caveats that almost all other papers totally ignore, such as the limitations of their study, the importance of tuning ranges, learning rate schedules, etc.\n\nConcerns:\n-I am very understanding and sympathetic to the issue of compute constraints, but I do believe that the study would be even more useful if a ResNet-50/Imagenet and a Transformer pipeline were used. The authors discuss that GANs and RL are not included, and those seem like very different types of optimization to me so I am more understanding of not including them. If the authors are academics then I have seen researchers have success in the past with getting grants from cloud providers, namely the lottery ticket hypothesis line of work and the TF Research Cloud (I found out about it through this paper https://openreview.net/forum?id=S1gSj0NKvB), which can easily be used to heavily tune ImageNet and Transformer runs. Outside of that, according to this benchmark https://dawn.cs.stanford.edu/benchmark/ImageNet/train.html, the cost of ImageNet runs has come down considerably in recent years, to $10-20 per run.\n-The experimental results could partially be explained by the No Free Lunch theorem, and the authors could at least reference this in the section.\n-What regularization, if any, was used for these problems, and was that also tuned? One could argue that, while optimization and regularization are in theory orthogonal to each other in what they try to accomplish (train vs test performance), they are both part of the update rule whose hyperparameters are being tuned. Also I believe it is important to be careful and note if coupled or decoupled weight decay/L2 is being used is important; I assume it is coupled because the DeepOBS code that is referenced uses L2 regularization, which means that optimizers that use preconditioning (Adam-like optimizers) could be impacted by this differently than those that do not (SGD/Momentum).\n\nWriting:\n-In the “Tuning method” paragraph in section 2.3, “In case there is no prior knowledge provided in the op.cit. we chose” seems like a syntax error.\n\nPrior work:\nThe authors do a very thorough literature search, and properly reference and discuss similar prior studies.\n\nAdditional feedback, comments, suggestions for improvement and questions for the authors:\n-Awesome job providing per-step values for results, it would be further useful to have code that could easily plot them side-by-side so that future researchers would be further encouraged to include them in their figures.\n-May be worth noting that, in addition to the optimizer hyperparameters, one could also tune the batch normalization momentum/epsilon, for additional performance gains.\n-Figure 3 seems very useful, however I believe it would be much better presented as a series of box plots. A nice recent example of this is Figure 2 in https://arxiv.org/abs/1906.02530. This could also be done for Figure 4 where each nested box is an individual optimizer, and it shows the distribution over runs for each optimizer for each problem.\n-The trapez schedule always seems to be the best, and I wonder if this is due to only one of the learning rate ramp up, which has been shown to be beneficial to stabilize training (although it is unclear if this is required), or the learning rate becoming quite small at the end, which has been shown to be necessary so that the optimizer can better learn the noisy directions of the objective. It would be useful for future work to consider each of these learning rate schedules (ramp-up, ramp-down) separately, although that requires even more compute.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "TTKV3qe6ok7", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an extensive independent benchmark of 14 popular optimizers on a variety of deep learning tasks from DeepOBS (Schneider et al. 2019). They compare them at three different tuning budgets and with 4 learning rate schedules. The authors are realistic about their setup. They acknowledge that different people might have different desires for such a benchmark, and they are clear about the choices they made to keep the experiments feasible. While there is no clear-cut answer that tells practitioners which optimizer to use in what scenario and how to tune it, these experiments are valuable and I believe it is important that these results are shared with the community.\n\nThe quality of the presentation and the writing is good. \n\nIn terms of novelty, the authors model the target audience slightly differently from previous work (Schneider et al. 2019, Choi et al. 2019, Sivaprasad et al. 2020). I am not convinced that this approach is better per se than others, but a different angle and a different set of optimizers is a valuable contribution to the community. I believe that the description of (Sivaprasad et al. 2020) in Section 1.1. is not entirely accurate. They do not compare hyperparameter tuning methods, but rather benchmark optimizers similarly to this work at a continuum of hyperparameter tuning budgets (all with random search).\n\nFinally, let me share two concerns:\n\n1. The intro mentions three contributions: (i) performance varies greatly, (ii) trying different optimizers works as well as tuning a single one, (iii) they identify a significantly reduced subset of algorithms and parameter choices that perform well across experiments. Points (ii) and (iii) are interesting, but (ii) is formulated quite imprecisely and it is hard to see on which results this is based. I inspected Figures 9 to 12 in the Appendix and conclude \"this might be true, but it is hard to see\". I believe a quantative statement would be more useful/meaningful. Similarly, for point (iii) it is not clear from which results this is concluded, and what the high-performing subset is. Such a list would be valuable to many practitioners and should be clearly stated in the main text.\n\n2. Figure 3 shows relative improvement across tasks. Any such measurements of 'improvement' are dependent on re-shifting or re-scaling of the loss, and are not necessarily meaningful when aggregated into a plot like this. Consider accuracy: measuring relative improvement (accuracy 1 / accuracy 2) would yield drastically different numbers than (error 1 / error 2 = (1-acc1) / (1-acc2)).", "extra_scores": {}, "extra_text": {}}, {"review_id": "4c3EviB6hxx", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\nThe authors of this paper conducted a thorough evaluation of deep learning optimizers across different compute budgets and learning rate schedules. They provide detailed analysis of the results. The design decisions are well-reasoned and explained throughout the paper.\n\n### Comments\n* As the authors note, there is certainly value in understanding the practical tradeoffs between optimizers: \"for most algorithms, the only formal empirical evaluation is offered by the original work introducing the method\"\n* The writing is clear and easy to follow.\n* Many of the findings are useful in the context of the DeepOBS dataset. For instance, Figure 3 highlights the diminishing returns of increasing the budget when tuning hyperparameters.\n* Open-sourcing the data is great and beneficial for the community.\n\n* The benchmark would benefit from a larger scale dataset(s). I'm not in favor of solely adopting DeepOBS, as past papers have shown systematic differences in evaluation at different scales [1][2]. Investigating whether there are systematic differences in optimization on larger problems e.g. machine translation or ImageNet would be valuable. \n* As Reviewer 4 mentions, the momentum parameter should be tuned as 1 - \\rho.\n\n### Recommendation / Justification\nI vote that this paper is below the acceptance threshold. There are many things to like about the approach taken is this paper, as highlighted above. However, the lack of larger scales datasets lessens the significance of the conclusions.\n\nI'd increase my score if concerns about the datasets used were addressed. I understand it is challenging to do so during the rebuttal period, but I strongly believe that larger scale datasets would strengthen the work significantly.\n\n### Minor feedback \n* A tabular form of Figure 4 would improve clarity.\n* I think it is worth acknowledging techniques for averaging the weights of neural networks, as these can have a substantial impact on final performance (Polyak averaging, exponential moving average, Stochastic Weight Averaging).\n* I believe it is also worthwhile to benchmark a second-order optimizer. While the compute per step is more expensive, the comparison could be made fair by using the same compute budget for each optimizer.\n* I am surprised by the choice of \\alpha when tuning the lookahead optimizer. My suspicion is that tuning the momentum and learning rate is more fruitful than trying low values of \\alpha.\n\n\n[1] Frankle, Jonathan, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. \"Stabilizing the lottery ticket hypothesis.\" arXiv preprint arXiv:1903.01611 (2019).\n\n[2] Gale, Trevor, Erich Elsen, and Sara Hooker. \"The state of sparsity in deep neural networks.\" arXiv preprint arXiv:1902.09574 (2019).\n\n\nEdit: After the rebuttal period, I maintain my original rating but am increasing the confidence of my evaluation from 4 to 5. I thank the authors for their hard work and engaging in discussion. I agree with Reviewer 3 that tuning with a fixed seed and the lack of search space refinement is a major weakness. The lack of a larger dataset further limit the applicability of the results. As such, I do not believe the paper in its current form should be accepted to ICLR. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Contributions of this type are very important for the community. There is a great deal of confusion among practitioners about how to pick optimizers. Perhaps worse, there is confusion among optimization researchers about how to demonstrate the effectiveness of their novel algorithms on deep learning tasks. I applaud this paper as one of the best attempts to make sense of this confusion.

Unfortunately, I am recommending that it is rejected. This was an extremely difficult decision. This paper was very thoroughly discussed by reviewers, both with the authors and after the feedback phase. I agree with R4 that this paper is exemplary in terms of its breadth of optimizer choices. I also agree with R3 that this paper's choices regarding hyperparameter search spaces and seed fixing significantly diminish the contribution of the paper at hand. The key issue that persuaded my decision centered on whether the paper's evidence supported its conclusions.

The two key conclusions that I want to highlight are:

1. *evaluating multiple optimizers with default parameters works approximately as well as tuning the hyperparameters of a single, fixed optimizer*

2. *different optimizers exhibit a surprisingly similar performance distribution compared to a single method that is re-tuned or simply re-run with different random seeds*

These conclusions can only be supported if optimizers are well-tuned. Based on R3's remarks and a quick reading of the paper, I am concerned that the use of fixed search spaces means that these optimizers cannot be considered well-tuned. This concern splits into two sub-concerns.

1. I appreciate the author's desire to encode "no prior knowledge about well-working hyperparameter values". Unfortunately, I don't think this is realistic or possible. The learning rate range used in this paper did not include 1e100 for good reasons, all of which depend on the prior knowledge of our community. This isn't just a glib concern, the apparently neutral search spaces may bias the conclusions towards well-known methods whose hyperparameters are well-understood.

2. I am also skeptical of the choice to use the same range for hyperparameters with "similar naming". The reason is that these hyperparameters *may have been misnamed by the inventors* and may, in fact, play very different roles in the dynamics of optimization.

Top-line conclusions have a way of becoming memes in our community. Therefore, it is critical that conclusions, as stated, are actually supported by the experimental design and the empirical evidence. Unfortunately, I am not confident that this is is the case for the paper at hand.

It is clear that this paper represents a heroic effort by the authors. I am aware of the challenges involved in getting this type of paper published and of the urgent need for them. I hope that the authors address the concerns that I expressed and the concerns of the reviewers in a future submission.
Decision: Reject