On the Tunability of Optimizers in Deep Learning
Authors: Prabhu Teja S*
Florian Mai*
Thijs Vogels
Martin Jaggi
Francois Fleuret
Venue:  (2020)
Keywords: Optimization
Benchmarking
Hyperparameter optimization

Abstract: There is no consensus yet on the question whether adaptive gradient methods like Adam are easier to use than non-adaptive optimization methods like SGD. In this work, we fill in the important, yet ambiguous concept of ‘ease-of-use’ by defining an optimizer’s tunability:  How easy is it to find good hyperparameter configurations using automatic random hyperparameter search? We propose a practical and universal quantitative measure for optimizer tunability that can form the basis for a fair optimizer benchmark.  Evaluating a variety of optimizers on an extensive set of standard datasets and architectures, we find  that Adam is the most tunable for the majority of problems, especially with a low budget for hyperparameter tuning.

Under review as a conference paper at ICLR 2020
ON THE TUNABILITY OF OPTIMIZERS IN DEEP LEARNING
Anonymous authors Paper under double-bling review
ABSTRACT
There is no consensus yet on the question whether adaptive gradient methods like Adam are easier to than non-adaptive optimization methods like SGD. In this work, we fill in the important, yet ambiguous concept of "ease-of-use" by defining an optimizer's tunability: How easy is it to find good hyperparameter configurations using automatic random hyperparameter search? We propose a practical evaluation protocol for optimizer tunability that can form the basis for a fair optimizer benchmark. Evaluating a variety of optimizers on an extensive set of standard datasets and architectures, we find that Adam is the most tunable for the majority of problems, especially with low budget for hyperparameter tuning.
INTRODUCTION
With the ubiquity of deep learning in various applications, a multitude of first-order stochastic optimizers (Robbins & Monro, 1951) have been in vogue. They have varying algorithmic components like momentum (Sutskever et al., 2013) and adaptive learning rates (Tieleman & Hinton, 2012; Duchi et al., 2011; Kingma & Ba, 2015). With all these choices, picking the optimizer is among the most important design decisions for machine learning practitioners. For this decision, the best possible generalization performance is certainly an important characteristic to be taken into account. However, we argue that in practice, an even more important characteristic is whether the best possible performance can be reached with the available resources.
The performance of optimizers strongly depends on the choice of hyperparameter values such as the learning rate. the machine learning research community, the sensitivity of models to hyperparameters has been of great debate recently, where in multiple cases, reported model advances did stand the test of time because they can be explained by better hyperparameter tuning (Lucic et al., 2018; Melis et al., 2018; Henderson et al., 2018). This has led to calls for using automatic hyperparameter optimization methods with a fixed budget for a fairer comparison of models (Sculley al., 2018; Feurer & Hutter, 2019; Eggensperger et al., 2019). For industrial applications, automated machine learning (AutoML, Hutter et al., 2019), which has automatic hyperparameter optimization of its key concepts, is becoming increasingly more important. In both cases, optimization algorithm that achieves good performances with relatively little tuning effort is arguably substantially more useful than an optimization algorithm that achieves top performances, but reaches only with lot of careful tuning effort. Hence, we advocate that the performance obtained by an optimizer is not only the best performance obtained when using that optimizer, but also has to account for the cost of tuning its byperparameters to obtain that performance, thus being dichotomous. We term this concept tunability in this paper.
Despite the importance of this concept, there is no standard way of measuring tunability. Works that propose optimization techniques show their performance on various tasks as depicted in Table 1. is apparent that the experimental settings, as well as the network architectures tested, widely vary, hindering a fair comparison. The introduction of benchmarking suites like DEEPOBS (Schneider et al., 2019) have standardized the tested architectures, however, this does not fix the problem of selecting the hyperparameters themselves, and the effort expended in doing so. Previous studies treat tunability to be the best performance obtained on varying hyperparameter (Schneider et al., 2019) or by measuring the improvement in performance by tuning a hyperparameter (Probst et al., 2019), but do not take any cognizance the intermediate performance during the tuning process.

---

Under review as a conference paper at ICLR 2020
Table 1: Experimental settings shown in the original papers of popular optimizers. The large differences in test problems and tuning methods make them difficult to compare. denotes learning rate, μ denotes momentum, is the weight decay coefficient.
Method
Datasets
Network archilecture
Parameter tuning methods
SGD with (Sutskever et al., 2013)
Artificial MNIST
Pully-connected
for first 1000 updates
LSTM
iben μa. É {0,00 0.9, 0.98,0.9955 0.995). & (ogro(1)) {-3,
for are
-6}
Adagrad
ImageNet ranking
Single layer Handcrafted features Histogram features
Perfomance dev
(Duchi et al., 2011)
Reuter RCVI
MNIST
KDD Census
Adam (Kingma & Ba, 2015)
IMDb MNIST
Logistic regression
8, {0,0.3) 0.9)
Multi-layer perceptron
82 {0.99, 0.999, 0.9999)
CIAAR
Convolational network
(ogro(7) {-5, 33 22
AdamW
CIFAR
ResNet CNN
log2(1) (-11,-10-
1.
(Loshchilov & Hutter, 2019)
ImageNet 32
(ogs(A) loga(10->) + -4.....4
In this paper, we introduce fair evaluation protocol for tunability based on automatic hyperparameter optimization, and simple evaluation measures that allow to compare the performance of optimizers under varying resource constraints. By evaluating on a wide range diverse tasks, we aim to contribute to the debate of adaptive vs. non-adaptive optimizers (Wilson et al., 2017; Shah et al., 2018; Chen & Gu, 2018). To reach a fair comparison, we experiment with several SGD variants that often needed to reach good performance. Although well-tuned SGD variant is able to reach the top performance in some cases, our overall results clearly favor adaptive gradient methods. We therefore conclude that there is substantial value in adaptive gradient methods.
MEASURING TUNABILITY
Given the dichotomy of the problem tunability, we argue that it needs to take into account
how difficult it is to find good hyperparameter configuration for the optimizer,
2. the absolute performance of the optimizer.
To see why both are needed, consider Figure L.a, which shows the performance in terms of loss of four different optimizers function of its only hyperparameter 0 (by assumption). If we only consider requirement #1, optimizer C would be considered the best, since every hyperparameter value is the optimum. However, its absolute performance is poor, making it of low practical value. Moreover, due to the same shape, optimizers A and B would be considered equally good, although optimizer clearly outperforms B. On the other hand, if we only consider requirement #2, optimizers and D would be considered equally good, although optimizer D's optimum is harder to find.
As we show in Section 5, no existing definition of tunability takes both requirements into account. In the following, we present formulation that does so.
2.1 PRELIMINARIES: HYPERPARAMETER OPTIMIZATION
We define hyperparameter optimization (HPO) (Feurer & Hutter, 2019) follows:
Definition. Let M be optimization algorithm with N hyperparameters (f1....,0y E ② Leta
specific instantiation of M with E θ be denoted by Mg. Thus, given a dataset Drrain Deal. the following objective is minimized
0- = aragggm min C(Me. Deat)
where Mø is trained on Dteain. In our work, we use L to be validation loss.
We use L()) to refer to [(Mg, D Daat) for brevity. In our experiments, we use the time-tested Random Search (Bergstra & Bengio, 2012) algorithm for HPO for simplicity.
2

---

Under review as a conference paper at ICLR 2020
2.2 DEFINING TUNABILITY
The ability to easily find good minima depends on the loss surface & itself. Thus, tunability is characterization of the HPO's loss function L. Here we present a quantification of this idea, which we illustrate with Figure 1.b.
Let us assume that there are two optimizers E & F, both with hyperparameter 0 both of them used to minimize function (e.g. train neural network). Let the loss functions of HPO be LE and Cp respectively, As the figure shows, the minimum of L lower than that of Cr (denoted by 0 and 0%) i.e. ( < (8 However, the minimum of Eæ is much sharper than that of Cp, and
Ontimizer C Optimizer B Optimizer D
Expected loss C())
Expected loss
Optimizer A
Optimizer R Optimizer
0%
0
Hyperparameter 0
Hyperparameter 0
La: Illustration. It important to consider both the absolute performance of optimizers as well as the tuning effort to get to good performances.
1.b: Illustration. While optimizer achieve the best performance after careful tuning, optimizer is likely to provide better performance under constrained HPO bud- get.
in most regions of the parameter space performs much better than E. This makes it easier to find configurations that already perform well. This makes optimizer F an attractive option when have no prior knowledge of the good parameter settings.
The key-difference between the two interpretations is whether prefers 'good enough' perfor- mance through fewer hyperparameter configuration searches (in the case of optimizer F), or whether one is willing spend computational time get the best possible performance (02 in the case of optimizer E). In this work, we search for the hyperparameter through an HPO like Random Search. Thus, the difference of the interpretations of tunability lies whether one values results from late stages of the HPO process (i.e. optimizer E is preferable due to better optimum) more than results from early stages of the HPO process (i.e. optimizer is preferable).
Motivated by these observations, we propose the following metric for tunability.
u-tunability's Definition. Let (0,, L(O))) be the incumbents (best performance attained till t) of the HPO algorithm at iteration and T' be the hyperparameter tuning budget. For w. V and E, We < oC, we define w-tumability as
w-tunability
Ct
ie, w-tunability a weighted sum of the incumbents (())). In our experiments we use Σ, we = 1.
By appropriately choosing the weights {wr). J. In the extreme case where we are only interested in the peak performance of the optimizer, we can set wy = 1 and set the other weights to zero. In the opposite extreme where the "one-shot tunability" of the optimizer, we can set W1 1. In general, we can answer the question of "How well does the optimizer perform with budget of K iterations?" by setting W; 1;
interpolate between two notions of tunability.
interested in
While the above weighting scheme intuitive, merely computing the performance after expending HPO budget of K does not consider the performance obtained after the previous K -1 iterations
3

---

Under review as a conference paper at ICLR 2020
Le. would like to differentiate the cases where requisite performance is attained by tuning an optimizer for K iterations and another for K1 iterations, where K > K. Therefore, we employ three additional weighting schemes. By setting W; x (T - t), our first one puts more emphasis on the earlier stages of the hyperparameter tuning process. We term this weighting scheme Cumulative Performance-Faryy (CPE). In contrast, the second weighting scheme, Cumulative Performance-late (CPL) puts more emphasis on late stages of tuning, and thus on obtaining a better performance at higher tuning cost: wi X As intermediate of the two, we also report a uniform weighting Cumulative Performance- Uniform (CPU) W/ 1/T.
3 OPTIMIZERS AND THEIR HYPERPARAMETERS
3.1 PARAMETERS OF THE OPTIMIZERS
To compare the tunability of adaptive gradient methods to non- non-adaptive methods, choose the most commonly used optimizers from both the strata; SGD and SGD with momentum for non-adaptive methods, and Adagrad and Adam for adaptive gradient methods. Since adaptive gradient methods said to work well with their default hyperparameter values already, we additionally employ a default version of Adam where we only tune the initial learning rate and set the other hyperparameters values recommended in the original paper (Kingma & Ba, 2015) (termed AdamLR), Such a scheme has been used by Schneider et al. too. A similar argument can be made for SGD with momentum (termed SGDM): thus we experiment with fixed momentum value of 0.9 (termed SGDMS).
the
In addition standard parameters in all optimizers, we consider weight decay with SGD too. SGD with weight decay can be considered as optimizer with two steps where the first step is to scale current weights with the decay value, followed by a normal descent step (Loshchilov & Hutter, 2019). Thus we devise two additional experiments for SGD with weight-decay where tune weight-decay along with momentum (termed SGDMW), and one where we fix it to 10-5 (termed SGDM WC) along with the momentum being fixed to 0.9, which is the value for weight decay found to be consistently better through HPO. The full list of optimizers we consider is provided in Table 4
Manually defining specific number of epochs can be biased towards one optimizer, as optimizer may reach good performance in the early epochs of a single HPO iteration, whereas another may reach higher peaks more slowly. In order to alleviate this, it would be possible to add the number of training epochs as an additional hyperparameter to be searched. Since this would incur even higher computational cost, we instead use validation set performance stopping criterion. Thus stop training when the validation loss plateaus for more than 2 epochs or if the number of epochs exceeds the predetermined maximum number as set in DEEPOBS.
3.2 CALIBRATION OF HYPERPARAMETER DISTRIBUTIONS
As mentioned previously, we use Random Search for optimizing the hyperparameters, which requires distributions of random variables to sample from. Choosing poor distributions sample from impacts the performance, and may break requisite properties (e.g. learning rate is non-negative). For some of the parameters listed in Table 2, obvious bounds exist due their mathematical properties, or have been prescribed by the optimizer designers themselves. For example, Kingma & Ba (2015) bound 3,, 82 to [0, 1) and specify that they are close 1. In the absence of such prior knowledge, we devise simple method to determine the priors.
We train each task specified in the DEEPOBS with a large number of hyperparameter samplings and retain the hyperparameters which resulted in performance within 20% of the best performance obtained. For each of the hyperparameters in this set, we fit the distributions in the third column of Table 2 using maximum likelihood estimation. In doing so, we make a simplifying assumption that all the hyperparameters are independent each other. We argue that these distributions are appropriate; the only condition on learning rate is non-negativity that is inherent to the log-normal distribution, momentum is non-negan -egaaivve with a usual upper bound of 1, B8 in Adam have been prescribed to be less than but close to it, is used to avoid divide- by-zee- error and thus is small positive value close to O. We report the parameters of the distributions obtained after the fitting in Table 2. The calibration procedure's 's performance is not included in the tunability computation.
4

---

Under review as a conference paper at ICLR 2020
Table Optimizers evaluated. For each hyperparameter, we calibrated a 'sampling distribution' to give good results across tasks (Section 3.2). (/ b] is the continuous uniform distribution on (a, 6]. Log-uniform(a, b) distribution whose logarithm is Lf bj. (og-norma((j..)) is distribution whose logarithm is normally distributed with mean μ and standard deviation a.
Optimizer
Tunable parameters
Sampling distribution
Stochastic Gradient Descent
Learning rate
Log-normal(-2.09, 1.312)
Momentum Weight decay
M(0, Log-uniform(-5.-I)
Adagrad
Learning rate
Log-normal(2..000 1.20)
Adam
Learning rate 81, .2
Log-normal( 1.42)
Log-uniformn-S5-- -1)
Log-uniform(-8,
Table 3: Models and datasets used. We use the DeepOBS benchmark set (Schneider et al., 2019). Details are provided in Appendix A.
Table 4: Optimizers and tunable parameters. y is learning rate, μ is momentum, À is the
weight decay coefficient.
Architecture
Datasets
Optimizer
Tunable parameters
Convolutional net
FMNIST, CIFARIO/100
SGD
7 (p=0, X=0)
Variational autoencoder
FMNIST, MNIST SVHN
SGDM
7, (A=))
Wide residual network
SGDM
7 0.9 A=0)
Character RNN
Tolstoi's s War and Peace
SGDM w
7 (W=0.9. X=10-
Quadratic function
Artificial datatset IMDb
SGDMW
7. M.H.M. A
LSTM
Adagrad
AdamLR
Y (,0-0.9,3=000900 0,0.0
Adam
82,5
4 EXPERIMENTS AND RESULTS
To assess the tunability of optimizers' hyperparameters for the training of deep neural networks, benchmark using the spen-s suite DEEPOBS (Schneider et al., 2019). The architectures and datasets we experiment details of the architectures. To obtain a better balance between vision and NLP applications, we added LSTM network with the task of sentiment classification in the IMDB dataset (Maas et al., 201 1), details for which are provided in Appendix A.
given in Table 3. We refer the reader to Schneider al. (2019) for specific
4.1 HYPERPARAMETERS AND IMPLEMENTATION
The performance automatic hyperparameter search methods is dependent on its own hyperpa- rameters, which play an important role in the outcome, e.g number of configurations to test. In our experiments, we evaluate 100 configurations with each of the hyperparameter optimization methods. As we random search, we simulate multiple runs of these 100 configurations through shuffling. This gives us the variance of performance at each step.
4.2 ANALYSIS OF TUNABILITY
We analyze the tunability for the various weighting schemes proposed. For the weighting scheme wi = =l,K. for increasing values of K. we show the performance as well as its variance in Figure 3. For readability, we show only results for Adam, AdamLR, Adagrad and SGDMW, and the rest are given in Appendix C. It is quite apparent that in most of the tasks, a well tuned SGD with momentum and weight decay is as good as Adam (for large K). However, the gap the performance is quite noticeable when AdamLR outperforms SGD in the VAE tasks and the IMDB task. In the case of image classification problems, SGD variants fare the best, as it has been reported before (Keskar & Socher, 2017). It is interesting notice that for K=4, the decreasing order of variance is nearly always SGDMW, Adam, Adagrad, AdamLR (10 out of 11 cases), even if AdamLR marginally underperforms it is in the case of Quadratic Deep. Given this formulation, ask the following
5

---

Under review as a conference paper at ICLR 2020
question: given HPO budget of K, what is the best choice for optimizer? We answer this in Appendix D.
For the other weighting schemes, tumability scores reported in Table 5. We see that there is one optimizer that is best across three schemes, and tasks presented. A similar trend of SGD doing better than the adaptive gradient methods image classification tasks is evident. Considering CPE, observe that R performs the best in 6 out 9 tasks, where as the other three times SGDM WC performs the best. The trend is not very obvious for CPU and CPL. In the case of CPU, AdamLR wins 5 out of 9 tasks, SGDM wins twice, and SGDM wins once. For CPL, AdamLR wins out of 9, and the SGDM wins once, and SGDM and SGDM win twice each. Summarizing, if peak- geak-performance or even evolving to better performance at larger hyperparameter search cost, SGD variants are better out 9 times. However, if good performance is expected in the earlier iterations of hyperparameter search, AdamLR is very competitive. Also, the default parameters of 82, of Adam optimizer result in quite good performance, to the point that Adam is rarely the better alternative over AdamLR. A known exception is training Inception networks (Abadi et al., 2015), where E is recommended to be set to 0.1.
For some of the cases, the tunablities reported are very similar for the AdamLR and SGD variants. However, tuning Adam is very different from tuning SGD from a wall-clock time measurement. For example, we find that for the case of CIFAR-10. 10, AdamLR requires to complete training than SGDMC (the top perfomer); thus being that much faster than SGDM in wall-clock time. It can be argued that more practical form of hyperparameter tuning budget is wall-clock time i.c. if wall-clock time budget of K minutes is given, how do our findings vary? In short, find similar trends we noticed before. We provide the details in Appendix E.
average 39% fewer epochs
4.3 SUMMARIZING ACROSS DATASETS
To get a better understanding of an optimizer's aggregate tunability across datasets compared the rest, we compute summary statistics for an optimizer o's performance after k iterations in the following way:
1.00
Agrrepted rebtuue turstilty
11.0-
01.0
AdanLR
1 P
o(k, p)
0.55
SCD
S(o,k) k)
SCDMWV
max o' pk.,)) o'EO
0.50
SCDM
SCDM W
where o(k,p) denotes the performance of optimizer o O on test problem p P after l iterations of the HPO pro- cess (i.e. w- u-tunabiiity with w; = 1,=k). In other words, we compute the average relative performance of optimizer to the best performance of any optimizer over all tasks.
0.75-
20
60
80
100
*xpprppramber co configuattons (bodgee))
Figure 2: Aggregated relative tunability of each optimizer across datasets
The results in Figure 2 and show that AdamLR performs very close the best optimizer throughout the HPO process and is the best till about the 60th iteration. In early stages of HPO, the SGD variants perform 10-20% worse than Adam, but improve as the HPO progresses.
5 RELATED WORK
There exist few works that have tried to define and investigate tunability formally, Assessing the impact of hyperparameter tuning for decision tree models, Mantovani et al. (2018) count the number of times the tuned hyperparameter values are (statistically significantly) better than the default values. Probst et al. (2019) define tunability of an ML algorithm as the performance difference between a reference configuration (e.g., the default hyperparameters of the algorithm) and the best possible configuration on each dataset. This metric is comparable across MI. algorithms, but it disregards entirely the absolute performance of ML algorithms. Schneider al. (2019) recently released benchmark for optimizers that evaluates their peak performance and speed. Tunability is assessed as the sensitivity of the performance to changes of the learning rate. In all three aforementioned studies, the definitions of tunability would fail to identify the superiority of optimizer over optimizer B in Figure L.a.
The study by Wilson et al. (2017) finds SGD-based methods as easy to tune as adaptive gradient methods. However, their study lacks clear definition of tunability and tunes the algorithms

---

Under review as a conference paper at ICLR 2020
Table 5: Performance of various experiments.
Optimizer
CPE
CPU
CPL
Optimizer
CPE
CPU
CPL
Optimizer
CPE
CPU
CPL
Adagrad
91.3
91.4
91.6
Adagrad
76.4
77.1
77.9
Adagrad
30.4
31.8
33.1
Adam
91.3
91.5
91.8
Adam
77.2
78.4
79.5
Adam
39.4
42.2
45.1
AdamLR
91.3
91.6
91.9
AdamLR
78.8
79.4
80.0
AdamLR
42.2
43.0
43.8
SGD
90.4
90.8
91.2
SGD
77.0
77.8
78.6
SGD
31.8
34.2
36.6
SGDM
90.5
90.9
91.3
SGDM
77.8
78.6
79.5
SGDM
40.6
43.3
46.0
SGDM
90.7
90.9
91.1
SGDMC
78.6
79.4
80.1
SGDM
42.1
43,3
44.5
SGDMOWC
90.7
90.9
91.1
SGDM
81.1
81.6
82.0
WEDMWWE
39.2
40.3
41.5
SGDMW
90.4
90.8
91.3
SGDMW
79.7
80.4
81.2
SGDMW
33.5
37.2
41.0
5.a: FMNIST 2C2D. Higher better
5.b: CIFAR 10. Higher better
5.c: CIFAR 100. Higher the better
Optimizer
CPE
CPU
CPL.
Optimizer
CPE
CPU
CPL
Optimizer
CPE
CPU
CPL.
Adagrad
84.3
84.8
85.3
Adagrad
94.8
94.9
95.0
Adagrad
55.6
56.2
56.7
Adam
83.6
84.5
85.5
Adam
94.5
94.8
95.2
Adam
54.4
55.7
57.0
AdamI.R
86.0
86.3
AdamI.R
95.1
95.3
95.4
AdamL.R
56.9
57.2
57.5
69.3
70.5
SGD
94.6
94.9
95.2
SGD
40.3
42.5
44.6
SGDM
75.9
77.5
SGDM
94.8
95.2
95.6
SGDM
51.4
54.0
56.5
SGDMC
79.3
80.1
81.0
SGDMC
94.9
95.1
95.3
SGDMC
55.6
57.0
58.3
SGDMOWC
78.8
79.4
80.0
SGDM
95.2
95.4
95.5
WG
54.2
55.6
57.0
SGDMW
75.7
77.1
78.6
SGDMW
95.0
95.2
95.3
SGDMW
45.1
48.2
51.2
5.d: IMDB. Higher better
5.e: WRN-16(4). Higher better
5.f: Char-RNN. Higher is better
Optimizer
CPE
CPU
CPL
Optimizer
CPE
CPU
CPL
Optimizer
CPE
CPU
CPL.
Adagrad
25.5
24.7
24.0
Adagrad
30.3
29.4
28.4
Adagrad
91.5
89.6
87.6
Adam
26.0
24.8
23.7
Adam
33.2
31.2
29.1
Adam
94.8
92.1
89.4
AdamLR
24.6
24.0
23.5
AdamLR
29.2
28.6
27.9
AdamLR
91.2
89.5
87.7
SGD
26.2
25.5
24.8
SGD
53.3
53.1
52.9
SGD
90.5
89.6
88.7
SGDM
26.2
25.4
24.6
SGDM
36.0
32.9
29.9
SGDM
89.5
88.7
87.9
SGDM
28.6
26.7
24.9
SGDM
54.1
53.5
53.0
SGDMC
89.6
88.6
87.5
SGDMOWC
27.9
26.4
24.8
SGDM
54.0
53.5
53.0
SGDM
88.6
88.4
88.1
SGDMW
26.5
25.7
24.8
SGDMW
34.6
32.2
29.8
SGDMW
89.3
88.9
88.4
5.g: FMNIST-VAE, Lower better.
5.h: MNIST-VAE, Lower is better
5.1: Quadratic deep. Lower better
manually selected, dataset dependent grid values. The study by Shah et al. (2018) applies similar methodology and comes to similar conclusions regarding tunability. Since both studies only consider the best parameter configuration, their approach would be unable to identify the better optimizer among B and in Figure l.a. In contrast, the methodology in our study is able to distinguish all the cases depicted in Figure l.a.
In a concurrent study, Choi et al. (2019) show that there exist a hierarchy among optimizers that such some be viewed as specific cases of others e.g. SGDM shown to be special case of Adam as its + po, and thus Adam should never underperform SGDM with appropriate hyperparameter search). Like our study, they suggest that the performance comparison of optimizers strongly depends on the hyperparameter tuning protocol. They also argue that the search space needs to be chosen optimizer specific. However, their focus is on the best possible performance achievable by an optimizer and does not take into account the tuning process. Moreover, while the authors claim their search protocol to be relevant for practitioners, the search spaces are manually chosen per datasel, constituting significant difference to the AutoML scenario considered in our paper.
Tunability is related to measuring hyperparameter importance (Hutter al., 2013), where van Rijn Hutter (2018) have recently shown that learning the priors for hyperparameter distributions can yield to better HPO performance, akin to the calibration phase in our study.
There has been recent interest in building optimizers termed the APROX family (Asi & Duchi, 2019a;b) that are provably robust to hyperparameter choices. Asi & Duchi experimentally find that, training Residual network (He et al., 2016) CIFAR-10 10, SGD converges only for small range of initial learning rate choices, whereas Adam exhibits better robustness to learning rate choices. This is inline with our findings of tunability.
7

---

Under review as a conference paper at ICLR 2020
CONCLUSION
Our work proposes new notion of lunability for optimizers that takes into account the tuning efforts of an HPO. The results of our experiments support the hypothesis that adaptive gradient methods are easier to tune than non-adaptive methods: In setting with low budget for hyperparameter tuning, tuning only Adam optimizer's learning rate is likely to be a very good choice; doesn't 't guarantee the best possible performance, but it is evidently the easiest find well-performing hyperparameter configurations for. While SGD yields the best performance is tedious find, and Adam often performs close to it. We, thus, state that the substantial value
some cases, its best configuration
3.a: CIFAR 10
3.b. CIFAR 100
Ascemec
Ada
SCIDAO
3.c: SVHN WRN-16-4
3.d: IMDb LSTM
G
SCIAAN
ABn
SCIDINN
Magel
MarE
Mam
3.c: FMNIST 2C2D CNN
3.f. Tolstoi Char-RNN
S01000
AA
Atlaraa
Ada
Mexld
MaeD
25
3.g: MNIST VAE
3.h: F-MNIST VAE
Mazaal
Marth
Mag
22
3.i: Quadratic Deep
19
Solee H
Figure 3: w-tunability with w, 1 for various experiments. We plot the the x-axis the number of the hyperparameter configuration searches, on the y-axis the appropriate performance on log scale. Figures a-f: higher is better and g-i: lower is better.
8

---

Under review as a conference paper at ICLR 2020
of the adaptive gradient methods, specifically Adam, is its amenability to hyperparameter search. This is in contrast to the findings of Wilson et al. (2017) who observe no advantage in tunabilty for adaptive gradient methods, and thus deem them to be of 'marginal value'. Unlike them, we base experiments on a standard hyperparameter optimization method that allows for an arguably fairer comparison.
Our study is certainly not exhaustive: We do not study the effect the inclusion of learning rate schedule, using a different HPO algorithm on the results. However, their inclusion would result in a large increase the number of experiments, and constitutes our future work.
We hope that this paper encourages other researchers conduct future studies the performance of optimizers from more holistic perspective, where the cost of the hyperparameter search is included.
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL https: / /wwww//wwww///////////w www.wttws.e....o tensor flow. org/. Software available from tensorflow.org.
Olah,
Hilal Asi and John C. Duchi. The importance of better models in stochastic optimization. Proceedings of the National Academy of Sciences, 2019a. ISSN 0027-8424. doi: 10.1073/pnas. 191117///paa..0001881111111 1908018116. URL https:: /www. www.aa/.or/oooooeoooooooooo/ooee
wwtp:///oooeeeeeeaaa///////0//// 00000/////00///1001111111111 998001816..
Hilal Asi and John Duchi. Stochastic (approximate) proximal point methods: Convergence,
optimality, and adaptivity. SIAM Journal on Optimization, 29(3):2257-2290, 2019b.
James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13(Feb):281-305, 2012.
Jinghui Chen and Quanquan Gu. Closing the generalization gap of adaptive gradient methods in training deep neural networks. CoRR, abs/1806.06765 2018. URL http::/arxiv....../////////
abs/1805 06763.
Dami Choi, Christopher J Shallue, Zachary Nado, Jaehoon Lee, Chris J Maddison, and George E Dahl. On empirical comparisons of optimizers for deep learning. arXiv preprint :XXv:1910.004466, 2019.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(uul:2221-22 2011.
Katharina Eggensperger, Marius Lindauer, and Frank Hutter. Pitfalls and best practices in algorithm
configuration. Journal of Artificial Intelligence Research, 64:861-893, 2019.
Matthias Feurer and Frank Hutter. Hyyperparameter Optimization, pp. 3-33. Springer International Publishing, Cham, 2019, ISBN 978-3-030-00318-- doi: bttpss///ol. htt./oii.ogrrgg/eee /togg/1o0 1000/1110009000 0-1-778--3300000---------
URL
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Thirry-Second AAAI Conference on Artificial Intelli- gence, 2018
Frank Hutter, Holger H Hoos, and Kevin Leyton- Identifying key algorithm parameters and instance features using forward selection. In International Conference on Learning and Intelligent Optimization, pp. 364-381. Springer, 2013.

---

Under review as a conference paper at ICLR 2020
Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren. Automated machine dearning-methon systems, challenges, 2019.
Nitish Shirish Keskar and Richard Socher. Improving generalization performance by switching from adam to sgd. arXiv preprint arXiv: / 17.00766 2017.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Iyya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019. URL https::/ wttpeeeiiiiieeee/rrnnneee Forum?? - Bkg6RiCqY7.
Mario Lucic, Karol Kurach, Marein Michalski, Sylvain Gelly, and Olivier Bousquet. Are gans created equal? a large-scale study. Advances neural information processing systems, pp. 700-709, 2018.
Andrew I.. Maas, Raymond E. Daly, Peter Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http: // //www. /twwwaacllet/ aclweb............ org/ anthhoogyyy..///o/o.o wwwphhhooggyPP1--10101 PP11-1015.
Rafael Gomes Mantovani, Tomáš Horváth, Ricardo Cerri, Sylvio Barbon Junior, Joaquin Vanschoren, André Carlos Ponce de de Carvalho, and Leon Ferreira. An empirical study on hyperparameter tuning of decision trees. arXiv preprint arXiv: / /tt.::/122220000 022007 2018.
Gábor Melis, Chris Dyer, and Phil Blunsom. On the state of the art of evaluation in neural language models. In International Conference on Learning Representations, 2018. URL https:/ openreview. n for.mfidd-yyyuuuuiiaa
Philipp Probst, Anne-Laure Boulesteix, and Bernd Bischl. Tunability: Importance hyperparameters of machine learning algorithms. Journal of Machine Learning Research, 20(53):1-32, 2019, URL http:: jm1r org/ hatpppperra rs vv2 1--11-4444 4.hml.
Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pp. 400-407, 1951.
Frank Schneider, Lukas Balles, and Philipp Hennig. DeepOBS: A deep learning optimizer benchmark suite. In International Conference on Learning Representations, 2019. URL
htttps://speneeviee net Fot
D. Sculley, Jasper Snoek, Alexander B. Wiltschko, and Ali Rahimi. Winner's 's curse? on pace, progress, and empirical rigor. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Workshop Track Proceedings, 2018. URL https: //tpsre//peeervieew net/ forum? fatammmdd=JJiie FFF
Vatsal Shah, Anastasios Kyrillidis, and Sujay Sanghavi. Minimum norm solutions do not always generalize well for wer-paraameeerized problems. arXiv preprint arXiv:1811.07033 2018
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In Sanjoy Dasgupta and David McAllester (eds.), Proceedings of the 30th Internutional Conference Muchine Leurning, volume 28 of Proceedings of Machine Learning Research, pp. 1139-1147, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR. URL http:/ / ppwpeediinggg .ml.......... press/v22. sutskever13 html.
T. Tieleman and G. Hinton. Lecture 6.5 RmsProp: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 2012.
Jan N van Rijn and Frank Hutter. Hyperparameter importance across datasets. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp- 2367-2376. ACM, 2018.
Ashia C Wilson, Rebecca Roelofs, Mitchell Stern, Nati Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. In Advances in Neural Information Processing Systems, pp. 4148-4158, 2017.
10

---

Under review as a conference paper at ICLR 2020
A ARCHITECTURES OF THE MODELS USED IN EXPERIMENTS
Along with the architectures examined by Schneider et al. (2019), we experiment with an additional network and dataset. We included an additional network into our experimental setup, as DEEPOBS does not contain an word level LSTM model. Our model uses 32-dimensional word embedding table and single layer LSTM with memory cell size 128, the exact architecture given Table We experiment with the IMDB sentiment classification dataset (Maas et al., 2011). The dataset contains 50, 000 movie reviews collected from movie rating website IMDB. The training set has 25, 000 reviews, each labeled as positive or negative. The rest 25, 000 form the test set. We split 20% of the training set to use as the development set. We refer the readers to DEEPOBS (Schneider et al., 2019) for the exact details of the other architectures used in this work.
Table 6: Architecture of the LSTM network used for IMDb experiments
Layer name
Description
Embedding Layer
Emb
Vocabulary of 10000
Embedding dimension: 32.
LSTM
LSTM 1
Input size: 32
Hidden dimension: 128
FC Layer
Linear(125 2)
Classifier
Softmax (2)
B Q- TUNABILITY
We provide additional methods to analyze tunability here. Let p(f) denote the best performance observed after using budget of hyperparameter optimization algorithm. We call an optimizer a-tunable (ax E [o, I)) at if p(t) 2 a p()). Thus a a-tunability is the ratio of number of times the neural network needs to be retrained with optimizer's hyperparameters being provided by an automatic method, to the total budget T (maximum number of configurations tested).
For each optimizer, define its a-tunability ((a) + for a {0.9, 0.95, 0.99). This metric provides an intuitive and simple quantification of how easy it is to tune an optimizer to reach requisite performance. We extend a-tunability to indicate the sharpness of the minima by computing the difference A ((a1) {(a2) where
and term it Sharpness. In our experiments, we choose 0n = 0.99 and a2 = 0.9. Sharpness(A)) the relative time taken by the HPO to improve from a2 to on and thus quantifies the flatness of the minima in the space of hyperparameters. If the minima is sharper, then we expect mandom-searee also takes longer time to find it, thus the time required to go from 01 and ag is higher. We provide Sharpness for optimizers in table:7.
MNIST VAE
FMNIST 2CID
CIFAR 100
CIEAR 10
SVHN WRN
IMDB LSTM
FAINST VAE
Quadratic
Char RNN
Adagrad
99.0
16.0
66.0
71.0
75.0
83.0
96.0
94.0
93.0
97.0
SGD
98.0
95.0
93.0
95.0
47.0
SGDM
55.0
97.0
50.0
91.0
59.0
72.0
90.0
30,0
59.0
SGDM
95.0
83.0
82.0
94.0
$2.0
83.0
75.0
95.0
57.0
WC
45.0
95.0
16.0
SGDMW
65.0
Table 7: Sharpness for various optimizers examined.
The above definition not without faults. An optimizer's 's a-tunability depends only on how fast it get close to its own best performance, pitfall it shares with Probst et al. (2019). That is, an optimizer that peaks at the performance of a random classifier may be considered well-tunable because it reaches its peak performance in the first iteration. It is apparent from tables 7 and 4 that

---

In our experiments so far, we defined tunability in terms number hyperparameter configurations. However, due to varying convergence speeds, different optimizers require varying amounts of time/epochs per configuration. We choose to work with number of epochs per configuration, as a hardware agnostic measure, but still indicates the relative time required. To incorporate epochs into our definition tunability, we conduct the following analysis: For each dataset, we consider minimum total epochs for running all 100 trials across all optimizers, and consider it the (virtual) maximum epoch budget maxx i.e., we disregard all trials after this point. We divide this maximum K. We modify the definition of w-tunability from Under review as a conference paper at ICLR 2020
the top performance does not imply lower sharpness. Take the of IMDB Bi-LSTM, the lowest sharpness is for SGDM while the best performance attained by AdamLR, implying that SGDM&WC settled to minima faster which isn't necessarily better than the one AdamLR found. In other terms, the flatness of the minima does not indicate how deep it is.
C PERFORMANCE ANALYSIS
We show the full performance plots of all variants of SGD experimented with, in figures 5, 6, 7.
D How LIKELY ARE WE TO FIND GOOD CONFIGURATIONS!:
natural question that arises is: given a budget K, what is the best optimizer one can pick? In other words, for given budget what is probability of each optimizer finding the best configuration? We answer this with a simple procedure. We repeat the runs of HPO for a budget K, and collect the optimizer that gave the best result in each of those runs. Using the classical definition of probability, we compute the required quantity. We plot the computed probability in Figure 8. It is very evident for nearly all budgets. AdamLR is always the best option for of the problems. SGD variants emerge to be better options for CIFAR- 100 and Char-RNN at later stages of HPO. For some of the problems like VAEs, LSTM, it is very obvious that AdamL_ nearly always the best choice. Thus further strengthens our hypothesis that adaptive gradient methods are more tunable, especially in constrained HPO budget scenarios.
E TUNABILITY BY COMPUTATION BUDGET
into K 100 intervals 4, Smaxii 1
Section 2.2 as follows: ppoch -tuabbiittyss Definition. Let (0,, [(O))) be the incumbents (best performance attained till of the HPO algorithum at iteration t, et be the total number of epochs required until iteration t has finished. We define É, max L, the maximum performance among all configurations that
e,S1,
have finished before interval Ir, where Lo = eD = 0. For w; > 4' 1 and E. wi < oc, we define w spach -unability as
spoch -umability
Please note that due to the case where no trial has finished before the first interval has concluded, we assign performance of 0 for that task to that interval (bin). The above definition does not lend itself to VAE tasks, as an appropriate value there oc. We, therefore, report the results for classification problems in Table 8, where better understand the results, we also report the average number of epochs each configuration takes on average. Comparing the relative performances of optimizers to the results from Table 5, we can observe that considering spoch yields to the same conclusions as before, and even amplifies Adam's 's strengths: The performance gap SGD variants widens in the cases where Adam(LR) was already better (FMNIST, IMDB, Char-RNN), and narrows considerably in the cases where an SGD variant was previously better (CIFAR-1O, CIFAR-100, WRN-16). On CIFAR-100 and WRN-16, this even results in Adam outperforming the SGD variants slightly. Considering the average number of epochs, the results can easily be explained by the fact that the adaptive gradient methods tend to take less time to converge.
the same weighting schemes (CPE, CPU, and CPL) as before. To
12

---

Under review as a conference paper at ICLR 2020
Table 8: wpoch tunability performance of optimizers on the classification tasks for CPE, CPU, and CPL weighting schemes. We additionally provide the average number of epochs required by each optimizer for single configuration.
Optimizer
CPE
CPU
CPL
Epochs
Optimizer
CPE
CPL
CPL
Epochs
Optimizer
CPE
CPU
CPL
Epochs
Adagrad
Adagrud
Adugrad
Adam
90.2
91.0
91.8
24.1
Adam
75.6
71.5
79.3
27.7
Adam
36.0
39.4
42.8
36.8
AdamLR
90.4
91.1
19.5
AdamL
78.2
79.1
80.0
24.9
AdumLR
41.8
42.8
43,8
24.6
34.5
73.4
77.9
29.9
SGDM
83.1
89.5
91.1
33.6
SGDM
74.3
76.5
78.8
46.9
SGDM
30.5
34.7
39.0
80.6
SGDMC
88.6
89.8
28-2
SGDMC
75.6
71.6
79.6
44.4
SGDMC
36.9
42.7
SGDMCWC
89.1
90.1
91.1
27.4
SGDMCWC
78.2
79.9
81.7
47.6
SGDMCWC
34.1
39.4
71.7
SGDMW
83.0
89.5
91.0
32.4
SGDMW
75.5
78.1
80.6
52.4
SGDMW
21.5
25.6
29.7
102.3
8.a: FMNIST 2C2D. Higher better
8.b: CIFAR 10. Higher better
8.c: CIFAR 100. Higher the better
Optimizer
Epechs
Optimizer
Epochs
Optimizer
Epochs
Adagrad
83.1
84.1
85.2
34.4
Adagrad
93.8
94.4
95.0
19.7
Adagrad
54.5
55.5
56.6
166.6
Adain
82.1
83.7
85.2
24.3
53.7
$7.0
131.0
AdamLR
84.8
85.5
86.2
28.6
AdanLR
94.5
95.0
95.4
17.2
AdanLR
55.9
56.7
57.4
170.9
SGD
65.8
67.7
69.6
42.2
SGD
92.6
91.8
95.1
29.9
SGD
38.1
40.8
43.4
183.2
SGDM
72.2
SGDM
188.3
SGDMC
77.9
79.3
80.8
32.3
SGDMC
93.3
94.2
95.2
26.0
SGDMC
53.3
55.6
58.0
194.5
SGDMCWC
71.3
75.0
78.6
109.7
SGDMOWC
92.5
94.0
95.4
31.8
SGDMCWC
51.7
54.2
56.6
197.3
SGDMW
1840
8.d: IMDB. Higher is better
8.c: WRR-16(4). Higher better
8.f: Char- RNN. Higher better
13

---

Under review as a conference paper at ICLR 2020
Figure 4: Performance analysis of various experiments. We plot the on the x-axis the number of the hyperparameter configuration searches, the y-axis the appropriate performance.
10a
Avcuricy
101
Adgerad
Adam
agrad Adau Adam
Adlmm
Adam
Adamn
Adaer
Adauu
Adaam!
Adamp
Adamm
Adam
SGD SGDM
SC1
SGG
SG
16
32
64
100
4.a: CIFAR- evolution
Acouracy
104
00
Adam
Ad agrad
Adamm Aupun
Ad aogra
Adlamt Adam
Adamp Adam
Ad Maou
Adiann
Adiam
Adamm
Adimm
SGDMO
SGD
SGD
SGD
SGD
SGL
SCD
16
32
64
100
4.b: CIFAR- 100 evolution
10a
Aveursyy
10a
Adagraat
Adamn
Adan
Auhun
Anham
Adlama
Adama
SGD
SGI
SGD
16
32
64
100
4.c: WRN evolution
Figure 5: Performance analysis of various experiments. We plot the on the x-axis the number of the
hyperparameter configuration searches, on the y-axis the appropriate performance on a log scale
14

---

Under review as a conference paper at ICLR 2020
10
9 10a
8 10
10
6 10'
10
x 10
Adagrad
Ad WaraI Adau
Adau
Adigaan
Adieg Iagrard
Adau
Ad Aerad
AdauB
Adam
Adam
Adamm
Adomj
Adamp
Adam
Adaml
SGDMO
Ad
SCDM
SCD
SGD
SGD
SGD
SCD
32
64
100
6.a: Bi-LSTM evolution
10a
9 10
Accurncy
8 10
7 X 10
6 6 10
Adgezad
Adamm
Aduaaa
Adamp Adem
Kchera aoad
Adamn Adsu
Adaan
Adamg
Adauy
Adhin
SCD
SGD
SGD
SGD
SC1
1
2
16
32
64
100
6.b: FMNIST CNN evolution
10a
6 10
4 10
3 10
2 2 10
10
Adagrad
Adauy
Adaayl
SCD Ad Sared
Adau
Adarg
SGD aard
Adaon Musu
Adaru
Adagraa
Adagg
Adggraa aggead md
Adgul
Adarg
Adias SeEa
Adaam
SGD
SGD
SGD
2
32
64
100
6.c: Tolstoi Char-RNN evolution
Figure 6: Performance analysis of various experiments. We plot on the x-axis the number
hyperparameter configuration searches, the y-axis the appropriate performance log scale
15

---

Under review as a conference paper at ICLR 2020
10
10a
4 x 10
10
Adagrad
Adam Adam
Ad Anous
Adam
Ad weran Adain
Adamm
Adamn
Adimy
Adgggan
Adam
Add pereaa
A0Ru
Adam
Ad WERRI AdRUR
Adam
SGDMO
SCDM
SGD
SGD
SGD
SGD
SCb
64
100
7.a: MNIST- VAE evolution
10a
Loss
6 10a
4 x 10a
3 x 10
Adgerad
Adlay Adamml
Adagrau Hares
Adamp Adem
Kdhera coaa Adimn Adou
Adaip
Adauy
SCIA
SGD
SGD
SGD
2
16
32
64
100
7.b: MNIST-VAE cvolution
4 x 10a
3 x 10a
Loss
2 x 10a
10a
Adagrad
Adayy Adary
SCD Ad BaEEd Adeu Adarg
Adaan Kissu
Adaau
Adagradd
Adamg
Adgul
Adarg
Adlan
Add
SGD
SGD
SGD
SGD
2
32
64
100
7.c: Quadratic deep evolution
Figure 7: Performance analysis of various experiments. We plot on the x-axis the number
hyperparameter configuration searches, the y-axis the appropriate performance log scale
16

---

Under review as a conference paper at ICLR 2020
FIMNST 2C20
MMST VAE
CIPAR 100
Pob of feeing best optenzer
80
40
WINNOM
Pich. of feeling best qutiberr
80
40
Quadratic Deep
Prob. thidig best optibber
80
SGD
SCDM
40
SCDM
20
SCDNFAM
Budget 80
20
40
40
60
Budge
Budger
Figure 8: Which optimizer for which budget? Given a tuning budget K (æ-axis), the stacked area plots above show how likely each optimizer (colored bands) is to yield the best result after K steps of hyperparameter optimization. For example, for the IMDB LSTM problem, for small budget, 'AdamLR' is the best choice (with 0.8 probability). whereas for larger search budget >> 50, tuning the additional parameters of 'Adam' is likely to pay off.
17


Reviews:
[{"review_id": "SJx0zByAYr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The main contributions of the submission are:\n\n1. A comprehensive empirical comparison of deep learning optimizers, with their performance compared under different amount of hyper-parameter tuning (they perform hyper-parameter tuning using random search).\n2. The introduction of a novel metric that tries to capture the \"tunability\" of an optimizer. This metric attempts to trade off the performance of an optimizer when tuned only with a small number of hyper-parameter trials, and its performance when carefully tuned. The metric is defined as a weighted average of the performance after tuning with i random trials, with i that goes from 1 to K. The weights of this weighted average and K are \"hyper-parameters\" of the metric itself. They use K=100 and suggest 3 possible choices of weights.\n\nThe paper appears to treat 2. as the main contribution. However, I do not think the metric they introduce is good enough to be recommended in future work, when comparing tunability of optimizers (or other algorithms with hyperparameters). The reason is that simpler methods provide just as much information, and do not rely on the need of interpreting the choice of the weights and K. This point is proven in the paper itself, where for example Figure 2 provides a more concrete and easier to interpret information than the tunability metric, similar graphs could be easily provided per dataset. Similarly, figure 3 as well as figures 5-7 and 8 in the appendix provide very good information about the tunability of the various optimizers without using the introduced metrics. Information similar (although not identical) to that summarized in table 5 could be captured by substituting the 3 metrics with the best performance after tuning for 4, 16 and 64 iterations respectively (just as examples).\n\nA stronger contribution is 1., which however is somewhat incremental compared to similar comparisons made in the past. Comparisons which, while mentioned, should perhaps have been discussed and compared more in detail in this work. Overall, I do not feel the comparisons dramatically change the qualitative understanding the field has of the different optimizers and their tunability.\n\nThey also suggest that when the tuning budget is low, using Adam but tuning only the learning rate is beneficial, which could be a valuable and practical suggestion.\n\nI enjoyed reading the submission, which is very clearly written, but due to the relatively limited value of the contributions, and excessive focus on the tunability metric which I do not feel is giustified, I slightly lean against acceptance here at ICLR. I do think, however, that it would make a great submission to a smaller venue or workshop.\n\nOther comments/notes:\n* One aspects that is mostly left out of the discussion (except from one side comment) is the wallclock time, as some optimizers might be on average quicker to train (for example due to quicker convergence), this can easily lead it to be quicker to tune even though it requires a higher budget of trials. I think it would be worth discussing this more.\n* minor: in figure 8 in the appendix, the results after 100 iterations is, as far as I understand, over a single replication, so is not particularly reliable (and will always be 100% of a single optimizer)\n* similarly to the above, if the configurations are always sampled from the same 100, confidence intervals in the graphs become less reliable as the budget increases.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BylSI7T6qr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a simple measure of tunability that allows to compare optimizers under varying resource constraints. The tunability of the optimizer is a weighted sum of best performance at a given budget. The authors found that in a setting with low budget for hyperparameter tuning, tuning only Adam optimizer’s learning rate is likely to be a very good choice; it doesn’t guarantee the best possible performance, but it is evidently the easiest to find well-performing hyperparameter configurations.\n\nComments:\n\nThe paper is easy to follow. The motivation of defining tunability of optimizer is a very interesting question, however, the study seems to preliminary and the conclusion is not quite convencing due to several reasons:\n\nIn section 3.2, to characterize its difficulties of finding best hyperparameters or tunability, the authors seem to try to connect the concept of “sharpness” of a minima in loss surface to the tunability of an optimizer, which is similar to comparing the loss landscape of minimums. However, while the authors made intuitive explanation about the tunability in section 2.2, I did not see the actual plot of the true hyperpaparameter loss surface of each optimizer to verify these intuitions. Can the author be more specific about the x-axis in the illustration 1.a and 1.b? If I understand correctly, they are not the number of trails.\n\nIn addition, the proposed stability metric seems not quite related with the above intuitions, as the illustrations (1.a and 1b) define the tunability to be the flatness of hyperparameter space around the best configurations, but the proposed definition is a weighted sum of the incumbents in terms of the HPO budgets. \n\nThe definition of the tuning budgets is not clear, is it the number of trials or the time/computation budgets? The authors seems interchangeably using “runs” and “iterations”, which makes the concept more confusable. \n\nThe authors further proposed three weighting schemes to emphasize the tunability of different stage of HPO. My concern is that is highly dependent  on the order of hyperparameter searched, which could impact the tunability significantly. For instance, in case of grid search HPO and 0.1 is the best learning rate, different search order such as [10, 1, 0.01, 0.1] and [0.1, 0.01, 1, 10] could results in dramatic different CPE and CPL. \n\nMy major concern is the hyperparameter distributions for each optimizer highly requires prior knowledge. A good prior of one optimizer could significantly affect the HPO cost or increase the tunability, i.e., the better understanding the optimizer, the less tuning cost. My major concern is that the authors assume the hyperparameters to be independent (section 3.2), which is not necessarily true. Actually hyperparameters are highly correlated, such as momentum, batch size and learning rate are correlated in terms of effective learning rate [1,2], so as weight decay and learning rate are [3], which means using non-zero momentum is equivalent to using large learning rate as long as the effective learning rate is the same. This could significantly increase the tunability of SGDM. Another concurrent submission [4] verified this equivalence and showed one can also just tune learning rate for SGDM. The assumption of independent hyperparameters might be fine for black box optimization or with the assumption that practitioners have no knowledge of the importance of each hyperparameter, then the tunability of the optimizer could be different based on the prior knowledge of hyperparameter and their correlations. But it is not rigorous enough to make the conclusion that Adam is easier to tune than SGD.\n\n\nThe author states their method to determine the priors by training each task specified in the DEEPOBS with a large number of hyperparameter samplings and retain the hyperparameters which resulted in performance within 20% of the best performance obtained. Could the authors be more specific on the hyperparameters searched? Is this process counted in the tunability measurement?\n\n[1] Smith and Le, A Bayesian Perspective on Generalization and Stochastic Gradient Descent, https://arxiv.org/abs/1710.06451\n\n[2] Smith et al, Don't Decay the Learning Rate, Increase the Batch Size, https://arxiv.org/abs/1711.00489\n\n[3] van Laarhoven et al, L2 Regularization versus Batch and Weight Normalization, https://arxiv.org/abs/1706.05350\n\n[4] Rethinking the Hyperparameters for Fine-tuning\n https://openreview.net/forum?id=B1g8VkHFPH\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed a new metric to define the quality of optimizers as a weighted average of the scores reached after a certain number of hyperparameters have been tested.

While reviewers (and myself) understood the need to better be able to compare optimizers, they failed to be convinced by the proposed solutions. In particular (setting aside several complaints of the reviewers with which I disagree), by defining a very versatile metric, this paper lacks a strong conclusion as the ranking of optimizers would clearly depend on the instantiation of that metric.

Although that is to be expected, by the very behaviour of these optimizers, it makes it unclear what the added value of the metric is. As one reviewer pointed out,  all the points made could have been similarly made with other, more common plots.

Ultimately, it wasn't clear to me what the paper was trying to achieve beyond defining a mathematical formula encompassing all "standard" evaluation metric, which I unfortunately see of limited value.
Decision: Reject