How much progress have we made in neural network training? A New Evaluation Protocol for Benchmarking Optimizers
Authors: Yuanhao Xiong
Xuanqing Liu
Li-Cheng Lan
Yang You
Si Si
Cho-Jui Hsieh
Venue:  (2021)
Keywords: deep learning
optimization
benchmarking

Abstract: Many optimizers have been proposed for training deep neural networks, and they often have multiple hyperparameters, which make it tricky to benchmark their performance. In this work, we propose a new benchmarking protocol to evaluate both end-to-end efficiency (training a model from scratch without knowing the best hyperparameter) and data-addition training efficiency (the previously selected hyperparameters are used for periodically re-training the model with newly collected data). For end-to-end efficiency, unlike previous work that assumes random hyperparameter tuning, which over-emphasizes the tuning time, we propose to evaluate with a bandit hyperparameter tuning strategy. A human study is conducted to show our evaluation protocol matches human tuning behavior better than the random search. For data-addition training, we propose a new protocol for assessing the hyperparameter sensitivity to data shift. We then apply the proposed benchmarking framework to 7 optimizers and various tasks, including computer vision, natural language processing, reinforcement learning, and graph mining. Our results show that there is no clear winner across all the tasks. 


Under review as a conference paper at ICLR 2021
How MUCH PROGRESS HAVE WE MADE IN NEURAL NETWORK TRAINING? A NEW EVALUATION PROTO- COL FOR BENCHMARKING OPTIMIZERS
Anonymous authors Paper under double-blind review
ABSTRACT
Many optimizers have been proposed for training deep neural networks, and they often have multiple hyperparameters, which make it tricky to benchmark their performance. In this work, we propose a new benchmarking protocol to evalu- ate both end-to-end efficiency (training model from scratch without knowing the best hyperparameter) and data-addition training efficiency (the previously se- used for periodically re-training the model with newly collected data). For end-to-end efficiency, unlike previous work that assumes ran- dom hyperparameter tuning, which over-emphasizes the tuning time, we propose to evaluate with a bandit hyperparameter tuning strategy. human study is con- ducted to show that our evaluation protocol matches human tuning behavior better than the random search. For data-addition training, we propose a new protocol for assessing the hyperparameter sensitivity to data shift. We then apply the proposed benchmarking framework to 7 optimizers and various tasks, including computer vision, natural language processing, reinforcement learning, and graph mining. Our results show that there no clear winner across all the tasks.
lected hyperparameters
INTRODUCTION
Due to the enormous data size and non-convexity, stochastic optimization algorithms have be- come widely used in training deep neural networks. In addition to Stochastic Gradient Descent (SGD) (Robbins & Monro 1951). many variations such as Adagrad (Duchi et al., 201T) and Adam (Kingma& & Bal 2014) have been proposed. Unlike classical, hyperparameter free optimizers such as gradient descent and Newton's 's method stochastic optimizers often hold multiple hyperpa- rameters including learning rate and momentum coefficients. Those hyperparameters are critical not only to the speed, but also to the final performance, and are often hard to tune.
It is thus non-trivial to benchmark and compare optimizers in deep neural network training. And benchmarking mechanism that focuses on the peak performance could lead false sense of improvement when developing new optimizers without considering tuning efforts. In this paper, we aim to rethink the role of hyperparameter tuning benchmarking optimizers and develop new benchmarking protocols to reflect their performance in practical tasks better. We then benchmark seven recently proposed and widely used optimizers and study their performance on wide range of tasks. In the following, we will first briefly review the two existing benchmarking protocols, discuss their pros and cons, and then introduce contributions.
Benchmarking performance under the best hyperparameters A majority of previous bench- marks and comparisons on optimizers are based on the best hyperparameters, Wilsson et al. (2017); Shah et al. (2018) made comparison of SGD-baser methods against adaptive ones under their best hyperparameter configurations. They found that SGD can outperform adaptive methods on several datasets under careful tuning. Most of the benchmarking frameworks for ML training also assume knowing the best hyperparameters for optimizers (Schneider et al. 2019 Coleman al. 2017 Zhu et al. 2018). Also, the popular MI Perf benchmark evaluated the performance of optimizers under the best hyperparameter. It showed that ImageNet and BERT could be trained in minute using the combination of good optimizers, good hyperparameters, and thousands of accelerators.
'The step sizes of gradient descent and Newton's method be automatically adjusted by line search procedure (Nocedal & Wright 2006).

---

Under review as a conference paper at ICLR 2021
Despite each optimizer's peak performance being evaluated, benchmarking under the best hyper- parameters makes the comparison between optimizers unreliable and fails to reflect their practical performance. First, the assumption of knowing the best hyperparameter is unrealistic. In practice, requires lot of tuning efforts to find the best hyperparameter, and the tuning efficiency varies greatly for different optimizers. It is also tricky to define the "best hyperparameter", which de- pends on the hyperparameter searching range and grids. Further, since many of these optimizers are sensitive to hyperparameters, some improvements reported for new optimizers may come from insufficient tuning for previous work.
Benchmarking performance with random hyperparameter search It has been pointed out in several papers that tuning hyperparameter needs to be considered in evaluating optimizers (Schnei- der et al. 2019; Asi & Duchi 2019), but having a formal evaluation protocol on this topic is non- trivial. Only recently, two papers Choi et al. (2019) and Sivaprasad et al. (220)) take hyperparameter tuning time into account when comparing SGD with Adam/ Adam/Adagrad. However, their comparisons among optimizers are conducted on random hyperparameter search. We argue that these compar- isons could over-emphasize the role of hyperparameter tuning. which could lead to pessimistic and impractical performance benchmarking for optimizers. This is due to the following reasons: First, in the random search comparison, each bad hyperparameter has to run fully (c.g.. 200 epochs), In practice, a user can always stop the program early for bad hyperparameters if having limited time budget. For instance, if the learning rate for SGD is too large, a user can easily observe that SGD diverges in few iterations and directly stops the current job. Therefore, the random search hypothesis will over-emphasize the role of hyperparameter tuning and does not align with a real user's practical efficiency. Second, the performance of the best hyperparameter is crucial for many applications. For example, in many real-world applications, we need to re-train the model every day every week with newly added data. So the best hyperparameter selected in the beginning might benefit all these re-train tasks rather than searching parameters from scratch. In addition, due to the expensive random search, random search based evaluation often focuses on the low-accuracy regior while practically we care about the performance for getting reasonably good accuracy.
Our contributions Given that hyperparameter tuning is either under-emphasized (assuming the best hyperparameters) or over-emphasize (assuming random search) in existing benchmarking pro- tocols and comparisons, develop new evaluation protocols to compare optimizers to reflect the real use cases better. Our evaluation framework includes two protocols. First, to evaluate the end-to-end training efficiency for a user to train the best model from scratch, we develop an ef- ficient evaluation protocol to compare the accuracy obtained under various time budgets, including the hyperparameter tuning time. Instead of using the random search algorithm, we adopt the Hyper- band (Li et al., 2017) algorithm for hyperparameter tuning since it can stop early for bad configura- tions and better reflect the real running time required by user. Further, we also propose to evaluate the data addition training efficiency for a user to re-train the model with some newly added train- ing data, with the knowledge of the best hyperparameter tuned in the previous training set. We also conduct human studies to study how machine learning researchers are tuning parameters in optimizers and how that aligns with our proposed protocols.
Based on the proposed evaluation protocols, we study how much progress has recently proposed algorithms made compared with SGD or Adam. Note that most of the recent proposed opti- mizers have shown outperforming SGD and Adam under the best hyperparameters for some par- ticular tasks, but it is not clear whether the improvements are still significant when considering hyper-parameter tuning, and across various tasks. To this end, we conduct comprehensive exper- iments comparing state-of- oftte--f-theaaae the--httt art training algorithms, including SGD (Robbins & Monro! 1951), Adam (Kingma & Ba 2014), RAdam (Liu et al. 2019) Yogi (Zaheer et al. 2018). LARS (You ctt al. 2017). LAMB YYou et al. 2019). and Lookahead (Zhang et al. 2019), on a variety of training tasks including image classification, generated adversarial networks (GANs), sentence classification (BERT fine-tuning), reinforcement learning and graph neural network training. Our main conclu- sions are: 1) On CIFAR-10 10 and CIFAR- 100, all the optimizers including SGD are competitive. 2) Adaptive methods are generally better on more complex tasks (NLP, GCN, RI.. 3) There no clear winner among adaptive methods. Although RAdam is more stable than Adam across tasks, Adam still a very competitive baseline even compared with recently proposed methods.
Forr instance, Sivaprasad al. 2020) only reaches < 50% accuracy in their CIFAR-100 comparisons.
2

---

Under review as a conference paper at ICLR 2021
2 RELATED WORK
Optimizers, Properties of deep learning make it natural apply stochastic first order methods, such as Stochastic Gradient Descent (SGD) (Robbins & Monro 1951). Severe issues such as a Zig-zag training trajectory and uniform learning rate have been exposed, and researchers have then drawn extensive attention to modify the existing SGD for improvement. Along this line of C
work, tremendous progresses have been made including SGDM (Qian 1999), Adagrad (Duchi et al.) 2011]. RMSProp (Tieleman Hinton| 2012), and Adam (Kingma Ba 2014). These methods utilize momentums to stabilize and speed up training procedures. Particularly, Adam is regarded as the default algorithm due to its outstanding compatibility. Then variants such as Amsgrad (Reddi) et al. 2019), Adabound (Luo et al. (2119) Yogi (Zaheer et al.) 2018). and RAdam (Liu et al. 2019) have been proposed resolve different drawbacks of Adam. Meanwhile, the requirement large batch training has inspired the development of LARS (You et al. (017)) and LAMB (You et al.) 2019). Moreover, /Zhang al. (2019) has put forward a framework called Lookahead to boost optimization performance by iteratively updating two sets of weights.
Hyperparameter tuning methods. Random search and grid search (Bergstra Bengio 2012) can be a basic hyperparameter tuning method in the literature. However, the inefficiency of these methods stimulates the development of more advanced search strategies. Bayesian optimization methods including Bergstra et al. (2011) and Hutter et al. (211)) accelerate random search by fitting black-box function of hyperparameter and the expected objective adaptively guide the search direction. Parallel to this line of work, Hyperband (Li et al. 2017) focuses on reducing evaluation cost for each configuration and early terminates relatively worse trials. Falkner et al.) (2018) proposes BOHB to combine the benefits of both Bayesian Optimization and Hyperband. All these methods still require huge computation resources. A recent work (Metz al.) (2020) has tried to obtain a list potential hyperparameters by meta-learning from thousands of representative tasks. We strike a balance between effectiveness and computing cost and leverage Hyperband in our evaluation protocol to compare a wider range of optimizers.
3 PROPOSED EVALUATION PROTOCOLS
In this section, we introduce the proposed evaluation framework for optimizers. We consider two evaluation protocols, each corresponding to an important training scenario:
Scenario I (End-to-end training): This is the general training scenario, where user is given an unfamiliar optimizer and task, the goal is to achieve the best validation performance after several trials and errors. In this case, the evaluation needs to include hyperparameter tuning time. We develop an efficiency evaluation protocol to compare various optimizers in terms of CPE and peak performance.
Scenario (Data-addition training): This is another useful scenario encountered in many ap- plications, where the same model needs to be retrained regularly after collecting some fresh data. this case, a naive solution is to reuse the previously optimal hyperparameters and retrain the model. However, since the distribution is shifted, the result depends on the sensitivity to that shift.
We describe the detailed evaluation protocol for each setting in the following subsections.
3.1 END-TO-END TRAINING EVALUATION PROTOCOL
Before introducing our evaluation protocol for Scenario I, we first formally define the concept of optimizer and its hyperparameters.
Definition 1. An optimizer is employed to solve minimization problem ming C(0) and can be defined by a tuple o E O (U, D), where contains all types of optimizers. U is a specific update rule and Q (Wi......4 ww) E N represents a vector of N hyperparameters. Search space of these hyperparameters is denoted by F. Given an initial parameter value Og. together with a trajectory of optimization procedure H. (0., L(O.), VL(..)). the optimizer updates 0 by
0,41 ().
We aim to evaluate the end-to-end time for a user to get the best model, including the hyperparameter tuning time. A recent work (Sivaprasad et al., 2020) assumes that user conducts random search for finding the best hyperparameter setting. Still, we argue that the random search procedure will over-emphasize the importance of hyperparameters when tuning is considered it assumes user
3

---

Under review as a conference paper at ICLR 2021
never stops the training even if they observe divergence or bad results in the initial training phase, which is unrealistic.
Figurel1 Tieeeiilluuttt Illustrates why random search might not lead to a fair comparison of optimizers. In Figure]] 1 we are given two optimizers, and B, and their corresponding loss w.r.t. hyperparameter. According to Sivaprasad et al. (2020), optimizer B is considered better than optimizer A under a constrained budget since most regions of the hyperparameter space of A outperforms B. For instance, suppose we randomly sample the same hyperparamter setting for A and B. The final config wt (B) found under wf (A), as shown in Figure Ia| However, there exists a more practical search strategy which can invalidate this statement with the assumption of a limited searching budget: user can early terminate a configuration trial when trapped in bad results or diverging. Hence, we can observe in Figure [b] that for optimizer A, this strategy early- stops many configurations and only allow a limited number of trials to explore the deeper stage. Therefore, the bad hyperparameters will not affect the overall efficiency of Optimizer A too much. In contrast, for optimizer B, performances of different hyperparameters and hard to distinguish, resulting in similar and long termination time for each trial. Therefore, may be easier for practical search strategy find the best configuration w of optimizer A than wp (B), given the same constrained budget.
this strategy can have lower expected loss than that of w
relatively satisfactory
Tominaatin aim
Expected loss
Optinizer A Optimizer
Optimizer
w,10)
Hyperperameteraa
Hyperparameter
Figure 1:
An illustration example showing that different hyperparamter tuning methods are likely to affect comparison of optimizers. Optimizer A is more sensitive to hyperparamters than optimizers B, but it may be prefered if bad hyperparameters can be terminated in the early stage.
This example suggests that random search may over-emphasize the parameter sensitivity when benchmarking optimizers. To better reflect practical hyperparameter tuning scenario, our eval- uation assumes a user applies Hyperband (Li et al.] 2017). simple but effective hyperparame- ter tuning scheme to get the best model. Hyperband formulates hyperparameter optimization unique bandit problem. It accelerates random search through adaptive resource allocation and early- stopping, as demonstrated in Figure Tb Compared with its more complicated counterparts such as BOHB (Flkner et al. 2018). Hyperband requires less computing resources and performs similarly within a constrained budget. The algorithm is presented in Appendix A
Despite different hyperparameter tuning algorithms, human tuning by experts is still regarded as the most effective. To verify and support that Hyper- band is an effective method and even competitive with humans, we conduct human study as fol- lows: for image classification on CIFARIO, given 10 learning rate configurations of SGD in the grid [1.0 x 10 1.0 x 10-7, 1.0 x 10-6 ...., 10), par- ticipants are requested to search the best one at their discretion. Namely, they can stop or pause a trial any time and continue to evaluate a new configuration until they feel it has already reached the best perfor- mance. 10 participants are sampled randomly from Ph.D. students with computer science backgrounds. We collect their tuning trajectories and average them as human performance, which is considered "optimal" in this human study. In Figure 2 we plot curves for hyperparameter tuning of human, Hyperband, random search, random search with an early stopping (ES) strategy in Sivaprasad et al. 2020). and Hyperband with ES. We find that Hyperband matches humans' behavior better, while random search tends to trap in suboptimal configurations although random with early stopping can mitigate this issue some extent. This finding shows the advantage of Hyperband over random
0.9
0.8
Accuracy
0.7
0.6
Human
0.5
Hyperband
Random
Hyperband with ES
Random with
30
300
600
Epach
Figure 2: Hyperband tuning used in our eval- uation protocol is closer to human behavior than random search.
4

---

Under review as a conference paper at ICLR 2021
search regardless of early stopping. and justifies the use of Hyperband in optimizer benchmarking. More details of this human study can found Appedix B
With Hyperband incorporated in end-to-end training, we assume that each configuration is run se- quentially and record the best performance obtained at time step / as P.. Specifically, P. represents the evaluation metric for each task, e.g., accuracy for image classification and return for reinforce- ment learning. {Pa} forms a trajectory for plotting learning curves on test like Figure Although it is intuitive to observe the performance of different optimizers according to such figures, summarizing learning curve into a quantifiable, scalar value can be more insightful for evaluation. Thus, as shown in Eq. [] we use X-tunability defined in Sivaprasad et al. (2020) to further measure the performance of optimizers:
X-tunability E
1=1 A, PP, where Σ, X = 1 and V, > 0.
One intuitive way to set X 1:=T determine which optimizer can reach the best model performance after the whole training procedure, However, merely considering the peak performance not good guidance on the choice of optimizers. In practice, we tend take into account the complete trajectory and exert more emphasis on the early stage. Thus, we employ the Cumulative Performance- Early weighting scheme where X x (T - i). to compute X-tunablity instead of the extreme assignment X = 1t=r. The value obtained is termed as CPE for simplicity.
We present evaluation protocol in Algorithm As can see, end-to-end training with hy- perparameter optimization is conducted for various optimizers on the given task. The trajectory [P:) is recorded to compute the peak performance well CPE value. Note that the proce- dure is repeated M times to obtain a reliable result. We use M = 3 in all experiments. More details of time cost and acceleration of the algorithm can be found in Appendix B
Algorithm End-to-Ent Efficiency Evaluation Protocol
Input: A set of optimizers O fo: (4, task A, feasible search space F
1: for O do
2: for i= to MM do
3:
Conduct hyperparameter search in F with the optimizer using HyperBand
4: Record the performance trajectory {Pr) explored by HyperBand
5: Calculate the peak performance and CPE by Eq.
6: end for
7:
Average peak and CPE values over M repetitions for the optimizer
8:
end for
9: Evaluate optimizers according to their peak and CPE values
3.2 DATA-ADDITION TRAINING EVALUATION PROTOCOL
In Scenario we have a service (e.g., search or recommendation engine) and we want to re-train the model every day or every week with some newly added training data. One may argue that an online learning algorithm should be used in this case, but in practice online learning unstable and industries still prefer this periodically retraining scheme which is more stable.
In this scenario, for every training, so no hyperparameter tuning is required and the performance (including both ef- ficiency and test accuracy) under the best hyperparameter becomes important. However, an implicit assumption made in this process is that "the best hyperparameter will still work when the training task slightly changes". This can be viewed transferability of hyperparameters for particular optimizer, and our second evaluation protocol aims to evaluate this practical scenario.
the best hyperparameters
chosen in the beginning,
reuse them
We simulate data-addition training with all classification tasks, and the evaluation protocol works as follows: 1) Extract subset D3 containing partial training data from the original full dataset D with a small ratio 8; 2) Conduct a hyperparameter search on Dj to find the best setting under this scenario; 3) Use these hyperparameters to train the model on the complete dataset; 4) Observe the potential change of the ranking of various optimizers before and after data addition. For step 4) when comparing different optimizers, we will plot the training curve in the full-data training stage in Section 4 and also summarize the training curve using the CPE value. The detailed evaluation protocol is described in Algorithm
5

---

Under review as a conference paper at ICLR 2021
Algorithm 2 Data- Data-Addition Training Evaluation Protocol
Input: set of optimizers O {o o (4, St) task a E A with a full dataset D, a split ratio d
1: for E do
2: for i to M do
3:
Conduct hyperparameter search with the optimizer using Hyperband on a with a partial dataset Dg, and record the best hyperparameter setting 9 Qpartal found under this scenario
4: Apply the optimizer with Paaiial on D3 and D, then save the training curves
5: end for
6: Average training curves of over M repetitions to compute CPE
7: end for
8: Compare performance of different optimizers under data-addition training
4 EXPERIMENTAL RESULTS
Optimizers to be evaluated. As shown in Table 1 we consider 7 optimizers including non- adaptive methods using only the first-order momentum, and adaptive methods considering both first- order and second-order momentum. We also provide lists of tunable hyperparameters for different optimizers in Table [I Morcover, consider following two combinations of tunable hyperparame- ters to better investigate the performance of different optimizers: a) only tuning initial learning rate with the others set to default values and b) tuning full list of hyperparameters. A detailed descrip- tion of optimizers as well as default values and search range of these hyperparameters can be found in Appendix D Note that we adopt unified search space for fair comparison following [etz al. (2020). eliminate biases of specific ranges for different optimizers. The tuning budget of Hyperband is determined by three items: maximum resource (in this paper we use epoch) per con- figuration R. reduction factor 17, and number of configurations ne. According to Li al. (2017), single Hyperband execution contains п, logg, []] + of SuccessiveHalving each reterred to as bracket. These brackets take strategies from least to most aggressive carly-stopping, and each one is designed to use approximately B R resources, leading to a finite total budget. The number of randomly sampled configurations in Hyperband run is also fixed and grows with R. Then given R and 17, n, determines the repetition times of Hyperband. We set v) = 3 as this default value performs consistently well, and R value which each task usually takes for complete run. For ne, it assigned as what required for a single Hyperband execution for all tasks, except for BERT fine-tuning, where larger number of configurations is necessary due relatively small R. In Appendix we give assigned values of R, y), and n. for each task.
Optimizer
Tunable hyperparameter
Non-udaptive
SGD LARS
Og. μ Og Pt,
Adam, RAdam, Yogi Lookahead, LAMB
Adaptive
0o. 3r. 82,
Table I: Optimizers to be evaluated with their tunable hyperparameters. Specifically, on represents the initial learning rate. μ is the decay factor of the first-order momentum for non-adaptive methods while 3, and 32 are coefficients to compute the running averages first-order and second-order momentums. is a small scalar used to prevent division by 0.
Tasks for benchmarking. For comprehensive and reliable assessment of optimizers, we con- sider wide range of tasks in different domains. When evaluating end-to-end training efficiency, we implement our protocol on tasks covering several popular and promising applications in Table 2 Apart from common tasks computer vision and natural language processing, we introduce two extra tasks in graph neural network training and reinforcement learning. For simplicity, we will use the dataset represent each task in our subsequent tables of experimental results. (For the rein- forcement learning task, we just use the environment name.) The detailed settings and parameters for each task can be found in Appendix D

---

Under review as a conference paper at ICLR 2021
Domain
Task
Metric
Model
Dataset
Image
CIFARIO
Classification
Accuracy
Re
CIFARIO0
Computer Vision
VAE
NLL
CNN Autoencoder
CelebA
GAN
FID
SNGAN network
CIFARIO
NLP
GLUE benchmark
Accuracy
RoBERTa-basc
MRPC
Graph network training
Node labeling
Fl score
Cluster-GCN
PPI
Reinforvement Learning
Walker2d-v3
Return
PPO
X
Table 2: Tasks for benchmarking optimizers. Details are provided in Appendix D
4.1 END-TO-END EFFICIENCY (SECNARIO I)
To evaluate end-to-end training efficiency, we adopt the protocol in Algorithm 1 Specifically, we record the average training trajectory with Hyperband ( (P:) for each optimizer on benchmarking tasks, where P: is the evaluation metric for each task (e.g., accuracy, return). We visualize these trajectories in Figure for CIFARIO and CIFARI00, and calculate CPE and peak performance in Table 3 and 9 respectively. More results for other tasks and the peak performance can be found in Appendix I Besides, in Eq. task, and then utilize the distribution function of a performance metric called performance profile Po(T) to summarize the performance of different optimizers over all the tasks. For tasks where lower CPE is better, we just use To.a = a min CPE a.a instead to guarantee ro.a 2 1. The function PT(T) for all optimizers is presented in Figure 4 Based the definition of performance profile (Dolan & Moré, 2002), the optimizers with large probability Pa(T) are to be preferred. In particular, the value of P. (T) is the probability that optimizer will win over the rest and can be reference for selecting the proper optimizer for an unknown task. We also provided a probabilistic performance profile to summarize different optimizers in Figure 7 in Appendix FF
compute performance ratio To.a for each optimizer and each
To,a max CPE..a :0E00 O}
1 [
Po(T)
size Aor ST).
(2)
CPE.,e
Our findings are summarized below:
It should be emphasized from Table and that under our protocol based Hyperhand, SGD performs similarly to Adam in terms of efficiency as well as peak performance, and can even surpass it in some cases like training on CIFAR100. Under Hyperband, the best configuration of SGD is less tedious to find than random search because Hyperband can early-stop bad runs and thus they will affect less to the search efficiency and final performance.
For image classification tasks all the methods are competitive, while adaptive methods tend to perform better in more complicated tasks (NLP, GCN, RL).
There is no significant distinction among adaptive variants. Performance of adaptive optimizers
tends to fall in the range within 1% of the best result.
According to performance profile in Figure the RAdam achieves probability with the smallest and Adam is the second method achieving that. This indicates that RAdam and Adam are achieving relatively stable and consistent performance among these tasks.
Table 3: CPE for different optimizers on benchmarking tasks. The best performance is highlighted in bold and blue and results within the 1% range of the best are emphasized in bold only.
Optimizer
CIFARIO (
CIFAR100 (
CelchA (VAE)
MRPC (NLP)
PPI (P) (GCN)
Walker2d-13 1 (RL)
(classification)
(classification)
Tune leamiin
rate anly:
SGD
$8.87 0,23
66.85 # 0.12
0.1430 0.0038
69.50 0.69
36.77 2 0.08
2795 4 275
Adam
901.2 0.10
65.55 1 1.2.3
0.1356 0.0001
84,90
0.72
95,08 0.01
3822 1 78
RAdam
90..9
0.1362 0.0001
0.04
Yogi
901.42 0.04
67.37 : 0.50
0.1371 0.0004
70.19.+
93.39
4132 205
LARS
90.25 0.07
67.48 : 0.04
0.1367 0.0002
69:97 +0.54
93.79
0.01
2986 105
LAMB
90.19 =: 0.08
65.08 + 0.06
0.1358 0.0003
82.23 + 1.49
87.79
01.77
3401 235
Lookahead
90.60 0.06
65.60 0.07
0.1358 0.0004
72.99
1.33
94,69 0.02
4141 264
every Ieperparammttt::
SGD
90,20 0.16
67.36 1 0.10
0.1407 0.0011
71.53 1.21
91.64
0.01
2978 1 91
Adam
89.27 1. 1.40
67.57 a 0.23
0. 0.1389 0.0002
65.23
1.44
92.62 1.222.1000 0.04
4080 459
RAdam
9I.14 0.44
66. 90 0.05
0.1366 0.0006
84.32
1.91
93.05 =: 0100
3813 103
Yogi
89.83 0.21
67.65 0.08
0.0019
68.42
88.94 0.11
249
LARS
90.42
67.78
0.0005
96.34
LAMB
90.27 a: 0.40
65.59 +: 0.03
0.1382 0.0001
84.66 2.61
93.18 0.05
2035 :
Lookahead
90.44 0.11
66.46 0.45
0.1360 0.0005
79.05
2.99
94,30 à: 0.04
3786 137

---

Under review as a conference paper at ICLR 2021
CIFARLE (uuse learning noe unly)
CIPARII [oan extry bypepeemeee)
CIFARIM fune leaming mie only]
CIFARUS (tune isees lepeprrntted)
(a)
(b)
(d)
Figure 3: End-to-end training curves with Hyperband on CIFARIO and CIFARIOO.
11
Phclolnity
5GD Mles BACEE
100
1.30
110
1.10
3.15
(a) Tune learning rate only.
(b) Tune every hyperparameter.
Figure 4: Performance profile of 7 optimizers in the range [1.0, 1.3).
4.2 DATA-ADDITION TRAINING (SCENARIO II)
We then conduct evaluation on data-addition training based on the protocol in Algorithm We choose four classification problems on CIFAR1O, CIFAR100, MRPC and PPI since the setting do not apply to RL. We search the best hyperparameter configuration, denoted by Sppriiats under the sub training set with the ratio å = 0.3. Here we tune all hyperparameters. Then we directly apply Dartial on the full dataset for a complete training process. The training curves are shown in Figure 5 and we also summarize the training curve with CPE by Eq. in Table 4 We have the following findings:
There no clear winner in data- addition training. RAdam outperforming other optimizers in 2/4 tasks so is slightly preferred, but other optimizers except Lookahead are also competitive (within 1% range) on least 2/4 tasks.
To investigate whether the optimizer's 's ranking will change when adding 70% data, we compare the training curve on the original 30% data versus the training curve on the full 100% data in Figure 5) We observe that the ranking of optimizers slightly changes after data addition.
Table 4: CPE of different optimizers computed under curves trained with 9 Qpanial on four full datasets.
Optimizer
CIFARIO (%))
CIFARI00 (%))
MRPC (%)
PPI
SGD
90.04 0.16
67.91 0.23
66.62 : 3.47
66.830 0.010
Adarn
90.52 0.03
67.04 + 0.27
73.13 + 1.16
70.420 + 0.007
RAdam
90.30 i 0.14
67.06
79.01 1 3.10
70,840 : 0.010
Yogi
89.63 0.39
67.58
68.40
67.990 0.003
LARS
90.17 + 0.13
67.29
64.43
68.400 1 0.005
LAMB
90.51 + 0.07
66.13
78.94 + 1.25
70.110 = 0.008
Lookahead
0.06
67.10
68.81 + 1.22
69.710 + 0.003
5 CONCLUSIONS AND DISCUSSIONS
In conclusion, we found there is no strong evidence that newly proposed optimizers consistently outperform Adam, while each of them may be good for some particular tasks. When deciding the choice of the optimizer for specific task, people can refer to results in Tan If the task is contained in T he/she can directly choose the one with the best CPE or best peak performance based on his/her goal of the task (easy to tune or high final performance). On the other hand, even though the desired task covered, people can also gain some insights from the results of the most similar task in Table 2 or refer to the performance profile in Figure 4 to pick adaptive methods like Adam. Besides choosing the optimizer, it will contribute designing new optimizer as well. Using our protocol to evaluate a new optimizer can show whether it has obvious improvement over existing optimizers, and can serve routine to judge the performance of the optimizer thoroughly.
8

---

Under review as a conference paper at ICLR 2021
Traising partad CIPARIO
Trateing on fut CHEARO
Trening on pertial CIPARII
Trstnlig on CHEAR
4psa
Ryeck
(a)
(b)
(c)
(d)
Figure 5: Training curves under Qparial for both partial and full datasets.
In addition to the proposed two evaluation criteria, there could be other factors that affect the prac- tical performance of optimizer. First, the memory consumption is becoming important for training large DNN models. For instance, although Lookahead performs well in certain tasks, requires more memory than other optimizers, restricting their practical use in strained applications. Another important criterion is the scalability of optimizers. When training with massively distributed system, optimizing the performance of a large batch regime (e.g.. 32K batch size for ImageNet) is important. LARS and LAMB algorithms included in our study are de- veloped for large batch training. We believe it is another important metric for comparing optimizers worth studying further.
memory
REFERENCES
Joshua Achiam. Spinning Up in Deep Reinforcement Learning, 2018.
Hilal Asi and John C Duchi. The importance of better models in stochastic optimization. Proceed- ings of the National Academy of Sciences, 116(46):22244-229)) 2019.
André MS Barreto, Heder S Bernardino, and Helio JC Barbosa. Probabilistic performance pro- files for the experimental evaluation stochastic algorithms. In Proceedings of the 12th annual conference on Genetic and evolutionary computation, pp. 751-758, 2010.
James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. The Journal of Machine Learning Research, 13(1):2813305, 2012.
James S Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for optimization. In Advances in neural information processing systems, pp. 2546-2554, 2011.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Wei-Lin Chiang, Xuanqing Liu, Si Si, Yang Li, Samy Bengio, and Cho-Jui Hsieh. Cluster-gen: An efficient algorithm for training deep and large graph convolutional networks. In Proceedings of the 25th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pp- 257-266, 2019.
Dami Choi, Christopher J Shallue, Zachary Nado, Jachoon Lee, Chris J Maddison, and George Dahl. On empirical comparisons of optimizers for deep learning. arXiv preprint arXiv:1910.05446, 2019. 2019.
Cody Coleman, Deepak Narayanan, Daniel Kang, Tian Zhao, Jian Zhang, Luigi Nardi, Peter Bailis, Kunle Olukotun, Chris Ré, and Matei Zaharia. Dawnbench: An end-to-end deep learning bench- mark and competition. Training, 100(101):102,
2017. 2017.
Elizabeth D Dolan and Jorge Moré. Benchmarking optimization software with performance pro- files. Mathematical programmingg 91(2):211-213, 2002.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(11):2121-2159, 2011.
Stefan Falkner, Aaron Klein, and Frank Hutter. Bohb: Robust and efficient hyperparameter opti- mization at scale. arXiv preprint luXiv=1807001744 2018.

---

Under review as a conference paper at ICLR 2021
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Frank Hutter, Holger H Hoos, and Kevin Leyton-Brown. Sequential model-based optimization for general algorithm configuration. In International conference on learning and intelligent optimiza- lion, pp. 507-523. Springer, 2011.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: wet2.69126.6 2014.
Diederik P Kingma and Max Welling. arXiv:13/2.61 14, 2013.
Auto-encoding variational bayes. arXiv preprint
Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel bbandit-bascd approach to hyperparameter optimization. The Journal of Machine Learning Research, 18(1):6765-68) 16, 2017.
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. arXiv preprint arXiv:1908.00 2019.
Liangchen Luo, Yuanhao Xiong, Yan Liu, and Xu Sun. Adaptive gradient methods with dynamic bound of learning rate. arXiv preprint arXiv: 1902.0//99 2019.
Luke Metz, Niru Maheswaranathan, Ruoxi Sun, C Daniel Freeman, Ben Poole, and Jascha Sohl- Dickstein. Using thousand optimization tasks to learn hyperparameter search strategies. arXiv preprint arXiv:2002.11888 11887, 2020.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957, 2018.
Jorge Nocedal and Stephen Wright. Numerical optimization. Springer Science & Business Media, 2006.
Ning Qian. On the momentum term in gradient descent learning algorithms. Neural networks, 12 (1):145-151, 1999.
Sashank Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. arXiv preprint arXiv:1904.09237, 2019.
Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathemati- cal statistics, pp. 400-407, 1951.
Frank Schneider, Lukas Balles, and Philipp Hennig. Deepobs: A deep learning optimizer benchmark suite. arXiv preprini 1XXi::9903005005 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 2017.
Vatsal Shah, Anastasios Kyrillidis, and Sujay Sanghavi. Minimum weight norm models do not probleessss......s.oeoe.. arXiv preprint 1rXXv::811100055. 2018.
always generalize well for over-parameterized problems.
Christopher Shallue. Jachoon Lee, Joseph Antognini. Jascha Dohl-Dicksteen, Roy Frostig, and George E Dahl. Measuring the effects of data parallelism on neural network training. arXiv preprint arXiv:1811.03600 2018
Prabhu Teja Sivaprasad, Florian Mai, Thijs Vogels, Martin Jaggi, and Francois Fleuret. Optimizer benchmarking needs to account for hyperparameter tuning. Proceedings of the 37th Interna- tional Conference on Machine Learning, 2020.
Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop: Divide the gradient by running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2):66
31, 2012.
10

---

Under review as a conference paper at ICLR 2021
Ashia C Wilson, Rebecca Roelofs, Mitchell Stern, Nati Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. In Advances neurul information pro- cessing systems, pp. 4148-4158, 2017.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arviv /ttdv./70..000 03888, 2017.
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsich. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint arXiv: 00962, 2019.
Manzil Zaheer, Sashank Reddi, Devendra Sachan, Satyen Kale, and Sanjiv Kumar. Adaptive meth- ods for nonconvex optimization. In Advances in neural information processing systems, pp. 9793- 9803, 2018.
Michael Zhang, James Lucas, Jimmy Ba, and Geoffrey E Hinton. Lookahead optimizer: k steps forward, step back. In Advances in Neural Information Processing Systems, pp. 9597-9608, 2019.
Hongyu Zhu, Mohamed Akrout, Bojian Zheng, Andrew Pelegris, Amar Phanishayee, Bianca Schroeder, and Gennady Pekhimenko. Tbd: Benchmarking and analyzing deep neural network training, arXiv preprint arXiv: arXiv:1803.06990 2018.
A HYPERBAND
We present the whole algorithm for Hyperband in Algorithm 3 for more details.
and you can refer to Li et al. (0117)
Algorithm Hyperband Input: R, Initialization: 8max = [log, RJ.
(Amax + 1)R
1: for {smax, Smax
0, do
0}
2:
B [o+1)
Ro)=
3: // begin SuccessiveHalving with (n, r) inner loop
4:
T = random get configuration
5:
for i (0.....3) do
6:
n4 |mn |m J. = rry
7:
L = {run_th wwwttteetuunnnnceeeeeee
return val (al.)osst
8:
T top.k(T, [m//n])
9:
end for
10: end for
return Hyperparameter configuration with the smallest loss seen so far
B DETAILS OF HUMAN STUDY
In this human study, 10 participants are sampled from Ph.D. students with computer science back- grounds (machine learning backgrounds specifically). They are recruited as follows: We first asked the administrators to distribute the program of this human study to Ph.D. students in machine learn- ing labs in our institutions. Provided the population, we assumed that they had prior knowledge about some basic machine learning experiments, such as image classification on MNIST and CI- FARIO. They were requested to conduct hyperparameter tuning of learning rate based on their knowledge. They were informed that the target experiment was image classification on CIFARIO with SGD, and the search range learning rate was in the grid [1.0 x 10 1.0 10-7 7, 1.0 10 ,...,,,.,00 10). Each configuration was run for 200 epochs at most. Moreover, we also told them that they could pause any configuration if they wanted evaluate others, and stop the whole tuning process only when they thought the accuracy could not be further improved and the number of total tuning epochs exceeded 600 the same time. We collected results from 17 people and
11

---

Under review as a conference paper at ICLR 2021
determined the validity by checking whether the length of the trajectory greater than 600 and removed invalidity trajectories. Finally there remained trajectories we averaged them as the human performance in Figure
C OPTIMIZERS
Notations. Given vector of parameters 0 E 12, we denote sub-vector its i-th layer's parameters sequence of learning rates during the optimization procedure of horizon
by 060, {are ) T. for, 1/ J==1 represents sequence of functions to calculate the first-order and second-order momentum of the gradient Gt. which are m, and e respectively at time step t. Different optimization algorithms are usually specified by the choice of ((-) and ()(-). (ri)== is an additional sequence of adaptive terms to modify the magnitude of the learning rate in some methods. For algorithms only using the first-order momentum, is the while 31 and f32 are coefficients to compute the running averages m and u. is a small scalar (e.g., 10-8) used to prevent division by 0.
Generic optimization framework. Based on, we further develop thorough generic optimization framework including extra adaptive term Algorithm The debiasing term used in the original version of Adam is ignored for simplicity. Note that for {ar)/=1 different learning rate scheduling strategies can be adopted and the choice of scheduler is also regarded Without loss of generality, in this paper, we only consider a constant value and a linear decay (Shal-) lue al. 2018 in the following equation, introducing y as hyperparameter.
tunable hyperparameter.
Cip Oz = 00 - - 7)00
constant;
linear decay.
With this generic framework, specifying my U, and r. in Table 5 It should be clarified that Lookahead is an exception of the generic framework. In fact it is more like high-level mechanism, which can be incorporated with any other optimizer. However, as stated in Zhang et al., (2019), this optimizer is robust inner optimization algorithm, k, and a, in Algorithm [S we still include Lookahead here with Adam as the base for a more convineing and comprehensive evaluation. We consider Lookahead as a special adaptive method, and tune the same hyperparamters for it as other adaptive optimziers.
summarize several popular optimization methods by explicitly
Algorithm 4 Generic framework of optimization methods
Input: parameter value f.. learning rate with scheduling {at}, sequence of functions
(de, 2h ) compute m,, it, and r, respectively.
for 1 to T do
2:
9: f((()))
3:
m4 de(91," 99t)
4:
Vt t'r "'e gt)
5:
Ti (e(,,, m4, Ut)
6:
0.+1 = 0₂ c4rima! /
7:
end for
Table 5: summary of popular optimization algorithms with different choices of m,, it and re
me
Vt
SGD(M) Adam
pmn--
1
31m--1 + (1 B1)0:
32v-1 92)04
RAdam
8:m-1 + (1 11-8,1 B1)g:
32V1-1 (1 - -32)g²
[pt - 41(ps 2)pco
(p 4)ppx 2)
Yogi
31m-11 + (1 -B1)0+
Vt-1 (1 - sign o² )g?
LARS
1
110 11 ml
LAMB
31m-1 + (1 11 -11)00
32 + (1- -
10
Lookahead*
3im-1 + (1 B1)gg
32V1-1 (1- 92)94
12

---

Under review as a conference paper at ICLR 2021
Algorithm 5 Lookahead Optimizer
Input: Initial parameters 0o. objective function f. synchronization period slow weights step size as optimizer A
1: fort t = -,, do
2: Synchronize parameters 0z. <- O--1
3: for := 1.2...... do
4: Sample minibatch of data d E DD
5: Or.: +- 0,,-11 A(L, i,
6: end for
7: Perform outer update 0g + 0.-1 + a0(0 - O2-1)
8: end for
return Parameters 0
D TASK DESCRIPTION
We make a concrete description of tasks selected for our optimizer evaluation protocol:
Image classifcation. For this task, we adopt ResNet-50 (He et al. 2016) model on CIFAR1O 10 and CIFARION with batch size of 128 and the maximum epoch 200 per trial.
VAE. We use vanilla variational autoencoder in Kingma & Welling (013) with five convolu-
tional and five deconvolutional layers with latent space dimension 128 on CelebA. There no dropout layers. Trained with batch size of 144.
GAN. We train SNGAN with the same network architecture and objective function with spectral normalization for CIFARIO I0 in Miyato et al. (2018). and the batch size of the generator and the discriminator is 128 and 64 respectively.
Natural language processing. In this domain, we finetune RoBERTa- base on MRPC, one of the test suit in GLUE benchmark. For each optimizer, we set the maximal exploration budget to be 800 epochs. The batch size is 16 sentences.
Graph learning. Among various graph learning problems, choose node classification as semi- supervised classification. In GCN training, in there are multiple ways to deal with the neigh- borhood explosion of stochastic optimizers. We choose Cluster-GCN Chiang et al. (2019) as the backbone handle neighhorhood expansion and PPI as the dataset.
Reinforcement learning. We select Walker2d-v3 from OpenAI Gym (Brockman et al. (2016)) as our training environment, and PPO (Schulman et al. (017)), implemented by OpenAl Spinning (Achiam 2018),, , as the algorithm that required tuning. We use the same architectures for both action value network Q and the policy network We define 40,000 of environment interactions as one epoch, with a batch size of 4,000. The return we used is the highest average test return of epoch during the training.
E IMPLEMENTATION DETAILS
Implementation details of our experiments provided in this section. Specifically, we give the unified search space for all hyperparamters and their default values in Table(6) Note that we tune the learning decay factor for image classification tasks when tuning every hyperparamter. For the task on MRPC, 7 is tuned for all experiments. In other cases, we only tune original hyperparamters without learning rate scheduler.
In addition, Hyperband parameter values for each task are listed Table assigned based on properties of different tasks.
These parameters are
For time cost of our evaluation protocols, depends on how many budgets are available. Specifi- cally, in our paper, the unit of time budget is one epoch, then the total time will be B Bepoch Tepochs where Bepoch the total available budget and Tepoch the running time for one epoch. There is no additional computational cost, i.e., running our protocol once takes the same time as running one hyperparameter search with Hyperband. In our experiment CIFAR10, we roughly evaluated 200 hyperparameter configurations in one Hyperband running, while the same time can only allow about 50 configurations in random search.
13

---

Under review as a conference paper at ICLR 2021
Hyperparamter
Search space
Default value
Log-Uniformí -8, 1)
Uniform[(1 (0. 1]
0 for SGD and 0.9 for LARS
81
Log-Uniformí -4,0)
B2
Log-Uniformí -6.0)
0.001
Log- (oggUniform( -8,
1e-8
Log-Uniform:
-4.0)
x
Table 6: Hyperparamter search space and default value
Hyperband parameter
Task
π
Th
"
Image
200
172
Classification
VAE
50
62
GAN
200
172
17 = 3
GLUE benchmark
10
200
Graph learning
200
200
RL
200
172
Table 7: Hyperband parameters for each task.
Moreover, further accelerate evaluation protocol by resampling, shown in Algorithm 6
The basic idea is that we keep a library of different hyperparameter settings. At the beginning. the library is empty. And in each repetition, sample number of configurations required by running Hyperband once. During the simulation of Hyperband, we just retrieve the value from the library if the desired epoch of current configuration is contained in the library. Otherwise, we run this configuration based on Hyperband, and store the piece of the trajectory to the library.
Algorithm 6 Alternative End-to-Enn End Efficiency Evaluation Protocol
Input: set of optimizers O {o o (2, task a E A. search space F, library size
1: for E do
2: Sample S configurations from F, initialize the library with an empty list for each setting
3: for = to M do
4:
Simulate Hyperband with using configurations re-sampled from the library the desired accuracy pre-computed in the library then
5:
6:
Retrieve the value directly
7:
else
8:
Training normally, and store values to the library
9:
end if
10:
Record the performance trajectory {P.]T= explored by HyperBand Calculate the peak performance and CPE by Eq.
12: end for
Average peak and CPE values over M repetitions for the optimizer o end for
14:
Evaluate optimizers according to their peak and CPE values
ADDITIONAL RESULTS
More detailed experimental results reported in this section.
F.1
IMPACT OF 7)
Since there an extra hyperparameter, the reduction factor in Hyperband, we conduct an ex- periment with different values (1) 2,3, 4, 5) observe the potential impact of this additional
14

---

Under review as a conference paper at ICLR 2021
hyperparameter evaluation. Specifically, optimizers, SGD, Adam, and Lookahead on CIFARIO, and results are presented Table X As we can see, although the change of 1 may lead to different CPE values, the relative ranking among three optimizers remains unchanged. Besides, they all achieve comparable peak performance at the end of training. Considering the efficiency of Hyperband, we choose y = 3 based on the convention in Li al. 2017 in all our experiments.
Hyperband to tune the learning rate for three
Table 8: CPE on CIFAR1O with different The value in the round brackets is peak performance.
CIFAR10 (%)
Optimizer
=
1=
"J = 4
Tume learning rate only:
SGD
88.63 (93.42)
88.87 (93.39)
88.45 (93.34)
87.97(93.73)
Adam
90.35 (93.24)
90.42 (93.65)
90.06 (93.26)
88.75 (93.23)
Lookahead
90.46 (93.53)
90.60 (93.60)
90.33 (93.40)
89.05 (93.62)
F.2 END-TO-END TRAINING
Table peak performance for optimizers each task. For GAN, only conduct evaluation on optimizers tuning learning rate due to time limit, and present its CPE and peak performance in Table I0 There is also end-to-end training curve for GAN CIFAR10 in Figure Figures for end-to-end training curves on the rest of tasks are shown in Figure and 10
Table 9: Peak performance during end-to-end training. The best one for each task is highlighted in bold.
Optimizer
CIFARIO (%)
CIFAR100 (%)
CelebA (VAE)
MRPC (%) (NLP)
PPT ^ (GCN)
Walker2d-13 (RL)
(clsssfication)
(classification
Time learinng rale only:
73.68
Adam
93.65 0.
71.51 0.21
0.1326 0.0001
84.90 0.10
98.73
10
4735 91
RAdam
93.93
72.30 0.09
0.1325 0.0002
85.41 1.45
98.70
1.0
10
5020 = 112
Yogi
93.58 a 0.03
73.48 0,93
0.1334 0.0002
7.919
98.18 9.0
5013 439
LARS
9
73.53
0.1332 0.0001
98.45
4.7
10
4073 a
LAMB
93.39 0.03
70.38 0.08
0,13 0.0002
82.23
98.46
10-1
4219
Lookahcad
93.60 = 0.04
71.75 2 0.68
0,1327 0.0001
72.99 0.61
98.63 + 1.4
10
5246 2 666
every hyperparameter:
SGD
93.47 0.02
73.94 a 0.06
0,1 + 0.0003
72.80 1.58
98.64 1.5 10-3
3647 + 129
Adam
92.58
73.82 0.25
0.1327 0.0002
98.93 7.5
10
RAdam
94.47 2 0.24
73.50 2 0.32
0,1326 0.0001
88.78 0.72
98.92 1 6.0
4886 334
Yogi
93.75 a: 0.08
73.88 0.25
0.1333 0.0004
69.60
98.85 4.8
4612 370
LARS
0..22+00.10
74.08
0.04
0.1333 0.0003
79.83 6.50
98.88 8.4
3526 312
LAMB
93.88 +.382
71.31
+ 0.08
0.13 0 0.0002
87.80 1.07
98.80 1.0
10
3654 + 243
Lookahead
0.182
72.90
10
Table 10: CPE on GAN for end-to-end training. The value in the bracket is peak performance.
Optimizer
GAN-CIFARI0,
Tune learning rate only:
SGD
113.25 (50.08)
Adam
77.04 (25.14)
RAdam
73.85 (19.61)
Yogi
76.80 (25.36)
LARS
157.71 (73.82)
LAMB
68.47 (25.55)
Lookahead
65.61 (20.40)
15

---

Under review as a conference paper at ICLR 2021
CIPARIO-GAN (tune learning rate only)
700
Adam
RAvin
100
LARS
R1D
400
Looksbead
LAMB
200
100
300
1000
Epoch
Figure 6: End-to-end training curves for GAN on CIFARIO.
Besides the general performance profile, we present a probabilistic version described in Barreto et al. (010) in Figure 7 to account for randomness. The probabilistic performance profile can be formulated as
Po(T)
1 P(Foo ST), Fo.a N [A
Ho.
Fo. by
12
where Pooo and Jo.o are the mean and standard deviation of CPE of the optimizer o on the task a respetively, and be is just the best expectation of CPE among all optimizers. It can be seen in Figure 7 that the probabilistic performance profiles shows similar trend to Figure
LABS
LABS
LSMI
Lecoabodd
1.00
(a) Tune learning rate only.
(b) Tune every hyperparameter.
Figure Probabilistic performance profile of optimizers in the range [1.0, 1.3).
We also attach two end-to an-to-end training trajectories on CIFAR10 with the error in Figure g Since it hard to distinguish optimizers with the standard deviation added, we just instead report the std of CPE and peak performance in Table 3 and
CIFAR10 (tune learning rate only)
CIFAR10 (tune every hyperparameter)
Accunit
Accuracy
0.5
SGD
RAdom
Yogi
Yog!
0.
LAMB
62
LASLB
Loolshead
Locksboa
10
200
103
10
1001
1140
Epoch
Epach
(a)
(b)
Figure End-to-end training curves on CIFAR10
16

---

Under review as a conference paper at ICLR 2021
CIFAR10 (tune learning rate only)
CIEAR1O (tune every hyperparameter)
05
04-
Accunayy
Accuracy
80
SGD
SGD
Arim
Adum
Yng!
Veg!
LARS
LARS
02
Lcoishedd
Tankoshadl
Epoch
Epach
(a)
(b)
CIFAR100 (lame learning rate only)
CIFAR100 (tune every hy hyperparamter)
84
0.5
83
Acouray
Accurayy
444
SCD
SGD
Adum
Alum
Yog!
Yog
LARS
LAIS
Loolaheed
Locksbad
Epoch
Epoch
CelebA (tune learning rate only)
CelebA (tune every hyperparamter)
0.50
0130
SCD
SGD
Adan
RAdsu
Yog!
Yog!
0.44
0114
LARS
Looisdaad
Lose
Lookshrad
0.33
0122
0.33
0.36
0.337
500
100
Egech
Epoch
(e)
Figure 9: End-to-end training curves on CIFAR1O, CIFAR100, and CelebA
F.3 DATA ADDITION TRAINING
We provide training curves for data-addition training on full MRPC and PPI dataset in Figure |I
17

---

Under review as a conference paper at ICLR 2021
MRPC (tine learning rate only)
MRPC
SUD
SGD
83
RAdin
RAcso
LARS
LARS
22 73
Lookabead
Looksbrod
Epoch
Epoch
(a)
(b)
PPI (tune leanning only)
10
SGD
Admm
RAden
09
RAdam
Veg!
Yogi
Accurur
LARS
Accuray
0.8
077
Looksbead
Lookahead
05
0.4
130
8000
1000
Epoch
Epech
(c)
(d)
Walker2d- (tune learning rate only)
Walker2d v3 v3 (tune every hyperparameter
1000
500
1000
Reroard
Adam
Adon
2000
RAdun
RAdiun
Youi
1000
LARS
LARS
LAKB
LAMB
Lookolandd
Losisabed
1000
2000
3000
600
1000
2000
Epoch
Epoch
(e)
(f)
Figure 10: End-to-end training curves on MRPC, PPI, and Walker2d-v3.
MRPC
PPI
Accurary
Adamo
Adum
Vogi
83
Yog
35
LARS
LAMB
LAMUB
Lockstred
64
1000
100
Epoch
Epoch
(a) Training on full MRPC.
(b) Training on full PPI.
Figure 11: Data addition trainong on MRPC and PPI.
18


Reviews:
[{"review_id": "xbaJ3sbe5_W", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper mainly proposed an evaluation framework for optimizers in machine learning jobs. It points out that existing benchmarking often applies best hyperparameter or random search hyperparameter. Their proposed framework re-evaluates the role of hyperparameter tuning in machine learning. \nIt mainly deals with two cases, end-to-end training efficiency and data-addition training efficiency. The major findings are as follows.\n1. Random search might lead to unnecessary training when the loss does not converge. Therefore, given limited budgets, it is better to have a benchmark strategy for finding the best hyperparamater.\n2. Training on the same model repeatedly is necessary when there are new data. However, the existing hyperparameters might not be optimal when training data updates.\nFor end-to-end training efficiency, they assume users apply Hyperband and adopt \\lambda-tunability to meansure the performance of optimizers. For every optimizer, the framework computes the CPE value based on the complete trajectory and evaluates the optimizers. For data-addition training efficiency, the framework extracts a subset to tune the hyperparameter and then apply them to the entire dataset and evaluate the performance of various optimizers. \n\nStrengths:\n1. The overall structure is clear with a detailed explanation of the limitation of the existing optimizer benchmark scheme. The two cases the authors emphasized are practical and common in real-life scenarios.\n2. I think the idea of applying Hyperband instead of random searching is quite reasonable based on the result in figure 2.\n3. The experiments are conducted based on a variety of datasets and optimizer. The findings are well presented with different angles, task type, optimizer mechanism.\nQuestions:\n1. It seems the algorithms 1 and 2 are similar. If it is possible to combine the algorithm together, like run the model on the subset to find several good optimizers and conduct it on the overall dataset.\n2. I think overall the idea is good and easy to understand. However, I wonder if there is any way to support the effectiveness of the solution besides running different setups on the model.\n\nWeaknesses:\n1. I wonder the time cost of adopting this evaluation framework, like the time it needs to have a consolidated result on the optimizers' performance.\n2. Some explanations regarding the details of the algorithms should be added beforehand. It is better to have an explanation of all the variables mentioned in the algorithm for clear referencing. For example, in the equation 1, the meaning of P, I assume it should be accuracy. Variable M, how it is decided and how it will affect the ultimate performance.\n3. Some minor suggestions. There are some spelling mistakes in the paper. For example, in table 1, it should be non-adptive. And the format and location of figure and table could be improved. E.g. Figure 5 is mentioned in page 5 and located at page 8. It could be better to give a general picture of how the result looks like in the 3rd section first and then detailed experiment figures later.", "extra_scores": {}, "extra_text": {}}, {"review_id": "r-WhStwy63P", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes two protocols for evaluating and comparing the quality of different optimizers. It points out that some existing, commonly used ways of comparing optimizers may over- or under-represent the amount of time that hyperparameter search can take. To fix this, it proposes using the Hyperband algorithm to guide the hyperparameter search. \n\nI think the authors make a convincing argument that existing approaches for comparing optimizers can downplay the role of hyperparameter search, which can be significant and can vary greatly across optimizers. I think the paper presents a fairly convincing approach for comparing optimizers, and thus for evaluating new ones against existing ones. \n\nI find the argument that Hyperband is a good choice because it more closely resembles human behaviour somewhat weak. Instead I would be more convinced by something showing that Hyperband (or whatever alternative) does a good job of terminating bad runs early, since this is the point of using a bandit algorithm over random search. \n\nI would also like to see some discussion of how others could use the proposed procedures  when deciding which optimizer to choose for their task.\n\nFinally, I think the authors could do a better job of explaining why CPE is the right metric to use (i.e., why is considering peak performance not a good choice?) \n\nIn general I like this work and recommend accepting it. However, I think it could be strengthened by more discussion of how this could be of use to the community in the future. This is especially important given that the no one optimizer seems to be universally best. \n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "XrGAWLrZ83B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**The score does not represent the initial review. It was updated following the discussions below.**\n\nThe paper proposes a new benchmarking protocol for optimizers in deep learning. The main argument is that previous papers have either neglected hyperparameter tuning or have employed hyperparameter search method that is far from how humans tune hyperparameters in practice. To mitigate the latter, the paper proposes to use HyperBand for automatic hyperparameter search. They show through a human study that HyperBand resembles human tuning performance more closely than random search. They then evaluate several optimizers on a multitude of tasks, including many recently proposed methods, under two scenarios: 1. Given an unfamiliar task, the effectiveness of each optimizer is measured through a metric that is cognizant of the hyperparameter tuning time. 2. After initially tuning on a subset of the dataset, how well does the obtained best hyperparameter configuration transfer to the full dataset? The main result is that the recently proposed optimizers are not substantially better than Adam.\n\nBy independently comparing recently proposed optimizers in two realistic scenarios, the paper would constitute a valuable contribution to the machine learning community. However, listed below, I take several issues that negatively impact my confidence in the protocol. If these issues could be resolved by answering my questions, I am inclined to update my rating upwards.\n\nMajor questions:\n* The main argument for replacing random search with HyperBand is that HyperBand resembles human tuning behavior much more closely. You provide the results of a human study as evidence for this central claim. However, the paper provides little detail on the nature of the study. How many participants did you have? How were they sampled? What was the expertise of the participants? If they happen to be familiar with computer vision, there is a good chance that they have trained on CIFAR10 before, thus already knowing good hyperparameter values. This would contradict the scenario that you assume in your benchmark, which is unfamiliarity with the task. Non-experts in computer vision would likely take a lot longer to tune CIFAR10 to good performance, perhaps more closely resembling the curve of random search. In its current state, I have little confidence in this human study.\n\n* You claim that in the evaluation protocol of Sivaprasad et al. (2020) each bad hyperparameter has to run fully. This is not true, because the protocol incorporates early stopping after two successive epochs in which the validation performance doesn't improve, thereby preventing at least _very_ bad configurations. It is obvious that HyperBand is a more sophisticated solution, but there is no direct evidence that it is so much better that it justifies replacing Sivaprasad et al. (2020)'s protocol. Perhaps you could include random search with a simple early stopping criterion in Figure 2?\n\n* In Algorithm 1, the performance trajectory is computed M times, and you average the peak and CPE values over all repetitions, which is good to account for stochasticity. But you never mention how large M is. Is the cost of HyperBand low enough to allow for a sufficiently large M? Would it instead be possible to compute expected validation performance as is done in Sivaprasad et al. (2020) to decrease this cost significantly? If I understand HyperBand correctly, the hyperparameter configurations are still drawn independently via random search so that the expected validation performance could be computed, but I am not entirely sure.\n\n* One of Sivaprasad et al. (2020)'s motivations for using random search is that it requires no hyperparameters (except for the search space, which is however assumed to be given by optimizer designers), which could otherwise inject some human bias into the evaluation process. In contrast, HyperBand does have additional hyperparameters that could introduce human bias. E.g., you state \"We set $\\eta = 3$ as this default value performs consistently well, [...]\". Can we be sure that this choice is not biased towards some optimizers?\n\n* Choi et al. (2019) make the case for choosing hyperparameter search spaces independently for each optimizer, noting that a unified search space may contain biases towards certain optimizers. You consider a unified search space, making the opposite argument by citing Metz et al. (2020). I could not find that argument in Metz et al. (2020). Could you please elaborate or point directly to their argument in their paper?\n\n* Among your summarized findings you state that Sivaprasad et al. (2020) find Adam to usually outperform SGD. This may be misleading, since they only suggest Adam to be more likely to yield good performance than SGD if nothing is known about the task. This is a very similar result to what the performance profile shows in your Figure 4. On CIFAR10 and CIFAR100, Sivaprasad et al. (2020) also find an SGD variant to perform as well or better than Adam.\n\nSuggestions for improving the paper:\n* It would be good to provide not only average CPE and peak performance values, but also their variance.\n\n* In the description of Scenario I you state that you compute the expected performance under different time budgets. It would be good to clarify what you mean (I suspect peak performance vs. CPE?).\n\nMinor issues:\n\n* In the related work on hyperparameter tuning methods, Sivaprasad et al. (2020) is falsely cited as a Bayesian optimization method.\n\n* The paper claims that Sivaprasad et al. (2020) consider optimizer B in Figure 1 as better than optimizer A. This is not true; towards the end of Section 2 in Sivaprasad et al. (2020) acknowledge that the value of each optimizer depends on the available budget.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Mfjm89qPVEO", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the topic of evaluating the performance of optimizers for neural networks. The paper makes the argument that existing evaluation procedures either over emphasize the finding of optimal hyperparameters or under-evaluate the performance of an algorithm by randomly sampling hyperparameters. This paper's primary objective is to propose an evaluation procedure that better aligns with a practitioner's goal than existing evaluation procedures. The proposed procedure evaluates optimization algorithms by using the hyperband hyperparameter optimization algorithm to tune hyperparameters and then score the algorithm using a weighted combination of validation performance scores over regularly sampled training intervals. The aggregate performances of algorithms are then ranked using performance profiles. \n\nThe paper's main contributions are an evaluation procedure and a new problem setup where an algorithm is evaluated over its long term use as new data is added. The presented evaluation procedure better captures practitioners' interest in that they tend to care about how much \"effort\" is required to find a near-optimal solution when allowed to tune the algorithm's parameters. The second evaluation procedure best captures the practitioners' interest by considering performance over retraining the model as new data is added. This evaluation setup is a good direction for evaluating optimizers. I think the paper accomplishes its goals and could be a useful evaluation procedure for the community. \n\nDespite what this paper does well, I cannot recommend it for acceptance because there are issues with the paper's arguments and some gaps in the evaluation procedure.\n\nI believe the paper mischaracterizes the performances being reported. The performance of the algorithms being reported is not the performance of an optimization algorithm but a meta-algorithm that combines the optimization algorithm and hyperband. Furthermore, the performance depends directly on the hyperband algorithm's hyperparameters, but these are not accounted for in the evaluation. I think it is ok to evaluate these meta algorithms, but their performance should not be presented as representing the underlying optimization algorithm. Another way to view these meta algorithms is that they are performing a global search using successive applications of local search algorithms (the optimizers). In this view, it is evident that the random hyperparameter search method is inferior to hyperband. However, it also becomes clear that one should use whatever global search method is best and not rely solely on hyperband. \n\nCan the authors specify an exact research question this procedure is designed to answer? It should be evident directly from this question what the right way to evaluate the performance is. \n\nWhy is being similar in performance to a human's ability to tune hyperparameters desirable? Shouldn't it be better? How was the study using humans conducted? Did an institutional review board approve it? The primary support for using Hyperband is that it is similar to human performance. The details of this human experiment are needed to establish how and why they are similar. \n\nThe performance of all of these experiments are random. How is randomness accounted for in the results? How many trials are needed to ensure a statistically significant result? Quantifying uncertainty is needed at both the per task level and the aggregate measure, similar to that shown by Jordan et al. (2020). The author’s may also be interested in probabilistic performance profiles (Barreto et al., 2010). Quantification of uncertainty is a necessary component for a scientifically rigorous evaluation procedure. \n\n\nMinor notes: \nThe second paragraph in the intro says a \"biased benchmark.\" What does it mean for a benchmark to be biased? Every benchmark is biased to favor one method or another. \n\nPage 4: \"Still, we argue that the random search procedure will overemphasize the importance of hyperparameters\" - this depends on what question is being answered. For example, one could ask a question about an algorithm's performance without hyperparameter tuning. Random hyperparameter search is then a good route. \n\nIn the RL experiments, it is said that the reward is the metric used. This is incorrect. The metric for that environment is the return or cumulative reward. \n\n\n\nBarreto, A. M., Bernardino, H. S., & Barbosa, H. J. (2010, July). Probabilistic performance profiles for the experimental evaluation of stochastic algorithms. In Proceedings of the 12th annual conference on Genetic and evolutionary computation (pp. 751-758).\n\nJordan, S. M., Chandak, Y., Cohen, D., Zhang, M., & Thomas, P. S. (2020). Evaluating the Performance of Reinforcement Learning Algorithms. In Proceedings of the 37th International Conference on Machine Learning.\n\n\n-----------\nupdate\n-----------\nAfter the discussions below I have changed my score from a 5 to a 6. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper was referred to the ICLR 2021 Ethics Review Committee based on concerns about a potential violation of the ICLR 2021 Code of Ethics (https://iclr.cc/public/CodeOfEthics) raised by reviewers. The paper was carefully reviewed by two committe members, who provided a binding decision. The decision is "Significant concerns (Do not publish)". Details are provided in the Ethics Meta Review. As a result, the paper is rejected based Ethics Review Committee's decision .

The technical review and meta reviewing process moved proceeded independently of the ethics review. The result is as follows:

This paper studies the problem of evaluating optimiser's performance, which is important to show whether real progress in research has been made. It proposes several evaluation protocols, and used Hyperband (Li et al. 2017) to automate the tuning of each optimiser in the bench-marking study. Evaluations have been conducted on a wide range of deep learning tasks, and the paper reaches to a conclusion that none of the recently proposed optimisers in evaluation can uniformly out-perform Adam in all the tasks in consideration.

Reviewers agreed that the evaluations are extensive, however there are some shared concerns among reviewers. The paper argues that manual hyper-parameter tuning by humans is the right behavior to target for, which is the motivation to use Hyperband as an automating tool, and there is a human study to demonstrate that Hyperband tuning resembles human tuning behaviour. Some reviewers questioned about this desiderata choice that favours human tuning behaviour, also concerns on how the human study is conducted (and to what extend the human study itself is reflective enough for the human tuning behaviour in general).

Personally I welcome any empirical study that aims at understanding the real progress of a research topic, and I agree it is important to make rigorous automation tools in order to enable such a large scale study. Therefore, while the presented results are extensive, I would encourage the authors to incorporate the feedback from the reviewers to better examine their assumptions. 
Decision: Reject