How much progress have we made in neural network training? A New Evaluation Protocol for Benchmarking Optimizers

Abstract: Many optimizers have been proposed for training deep neural networks, and they often have multiple hyperparameters, which make it tricky to benchmark their performance. In this work, we propose a new benchmarking protocol to evaluate both end-to-end efficiency (training a model from scratch without knowing the best hyperparameter) and data-addition training efficiency (the previously selected hyperparameters are used for periodically re-training the model with newly collected data). For end-to-end efficiency, unlike previous work that assumes random hyperparameter tuning, which over-emphasizes the tuning time, we propose to evaluate with a bandit hyperparameter tuning strategy. A human study is conducted to show our evaluation protocol matches human tuning behavior better than the random search. For data-addition training, we propose a new protocol for assessing the hyperparameter sensitivity to data shift. We then apply the proposed benchmarking framework to 7 optimizers and various tasks, including computer vision, natural language processing, reinforcement learning, and graph mining. Our results show that there is no clear winner across all the tasks. 
      

Reviews and author responses:

The initial reviewers for this paper flagged a key issue in ethics:
1.	The paper presents a study with human experiments. However, there was no mention of an ethical review board being involved in the process. Additionally, there is a significant lack of information to who was involved with the study or how it was conducted to be able to say if appropriate standards were applied. 

The author’s response “As this human study is not involved in anything about ethics, we did not ask for approval from an institutional review board.”

In the paper, the authors conducted human studies to study how machine learning tuned parameters in optimizers.  Details of the human study were provided in the paper in Appendix “B DETAILS OF HUMAN STUDY” in which 10 participants were sampled from Ph.D. students with computer science backgrounds (machine learning backgrounds specifically). Subjects were requested to conduct hyperparameter tuning of learning rate based on their knowledge. Results were collected and averaged as
the human performance data as shown in Figure 2 in the paper. 

In the ICLR code of the ethics, it specifically states: “Where human subjects are involved in the research process (e.g., in direct experiments, or as annotators), the need for ethical approvals from an appropriate ethical review board should be assessed and reported.”  PIs can not make their own assessment. Only the appropriate ethical review board can make this designation, which typically involves the researchers sending a request to their ethical review board to ask. The review board then will indicate if ethical approvals are required or not. If these approvals are required, PIs are not allowed to perform the research (in fact, it’s a violation and can result in actions against both the PI and the institution where the PI works). 

As such, given that human subjects were involved in the research in direct experiments and their data was used in evaluation and to create new knowledge, there was a need to receive appropriate ethical review board approvals (or have the ethical review board indicate no ethical approvals were required). As such, this paper poses significant ethical concerns based on standard ethical review board regulations and should not be published.


This paper studies the topic of evaluating the performance of optimizers for neural networks. The paper makes the argument that existing evaluation procedures either over emphasize the finding of optimal hyperparameters or under-evaluate the performance of an algorithm by randomly sampling hyperparameters. This paper's primary objective is to propose an evaluation procedure that better aligns with a practitioner's goal than existing evaluation procedures. The proposed procedure evaluates optimization algorithms by using the hyperband hyperparameter optimization algorithm to tune hyperparameters and then score the algorithm using a weighted combination of validation performance scores over regularly sampled training intervals. The aggregate performances of algorithms are then ranked using performance profiles. 

The paper's main contributions are an evaluation procedure and a new problem setup where an algorithm is evaluated over its long term use as new data is added. The presented evaluation procedure better captures practitioners' interest in that they tend to care about how much "effort" is required to find a near-optimal solution when allowed to tune the algorithm's parameters. The second evaluation procedure best captures the practitioners' interest by considering performance over retraining the model as new data is added. This evaluation setup is a good direction for evaluating optimizers. I think the paper accomplishes its goals and could be a useful evaluation procedure for the community. 

Despite what this paper does well, I cannot recommend it for acceptance because there are issues with the paper's arguments and some gaps in the evaluation procedure.

I believe the paper mischaracterizes the performances being reported. The performance of the algorithms being reported is not the performance of an optimization algorithm but a meta-algorithm that combines the optimization algorithm and hyperband. Furthermore, the performance depends directly on the hyperband algorithm's hyperparameters, but these are not accounted for in the evaluation. I think it is ok to evaluate these meta algorithms, but their performance should not be presented as representing the underlying optimization algorithm. Another way to view these meta algorithms is that they are performing a global search using successive applications of local search algorithms (the optimizers). In this view, it is evident that the random hyperparameter search method is inferior to hyperband. However, it also becomes clear that one should use whatever global search method is best and not rely solely on hyperband. 

Can the authors specify an exact research question this procedure is designed to answer? It should be evident directly from this question what the right way to evaluate the performance is. 

Why is being similar in performance to a human's ability to tune hyperparameters desirable? Shouldn't it be better? How was the study using humans conducted? Did an institutional review board approve it? The primary support for using Hyperband is that it is similar to human performance. The details of this human experiment are needed to establish how and why they are similar. 

The performance of all of these experiments are random. How is randomness accounted for in the results? How many trials are needed to ensure a statistically significant result? Quantifying uncertainty is needed at both the per task level and the aggregate measure, similar to that shown by Jordan et al. (2020). The author’s may also be interested in probabilistic performance profiles (Barreto et al., 2010). Quantification of uncertainty is a necessary component for a scientifically rigorous evaluation procedure. 


Minor notes: 
The second paragraph in the intro says a "biased benchmark." What does it mean for a benchmark to be biased? Every benchmark is biased to favor one method or another. 

Page 4: "Still, we argue that the random search procedure will overemphasize the importance of hyperparameters" - this depends on what question is being answered. For example, one could ask a question about an algorithm's performance without hyperparameter tuning. Random hyperparameter search is then a good route. 

In the RL experiments, it is said that the reward is the metric used. This is incorrect. The metric for that environment is the return or cumulative reward. 



Barreto, A. M., Bernardino, H. S., & Barbosa, H. J. (2010, July). Probabilistic performance profiles for the experimental evaluation of stochastic algorithms. In Proceedings of the 12th annual conference on Genetic and evolutionary computation (pp. 751-758).

Jordan, S. M., Chandak, Y., Cohen, D., Zhang, M., & Thomas, P. S. (2020). Evaluating the Performance of Reinforcement Learning Algorithms. In Proceedings of the 37th International Conference on Machine Learning.


-----------
update
-----------
After the discussions below I have changed my score from a 5 to a 6. 


Thanks for your helpful questions and the response is presented below.


For the statement that human hyperparameter tuning is superior to automated hyperparameter optimization, we are sorry that there is currently no specific paper mentioning this point. However, this hypothesis is based on experience from people who have worked in the industry for a long time and they are still using human tuning instead of automated methods in practical applications. Besides, in most published papers, tuning hyperparameters by humans within some grids is still a prevailing way rather than fully automatic tuning. Perhaps it can be a future direction to investigate human tuning versus automated HPO algorithms.


The details of human study have been added in Appendix B and we restate them here. In this human study, 10 participants are sampled from Ph.D. students with computer science backgrounds (machine learning backgrounds specifically). They are recruited as follows: We first asked the administrators to distribute the program of this human study to Ph.D. students in machine learning labs in our institutions. Provided the population, we assumed that they had prior knowledge about some basic machine learning experiments, such as image classification on MNIST and CIFAR10. They were requested to conduct hyperparameter tuning of learning rate based on their knowledge. They were informed that the target experiment was image classification on CIFAR10 with SGD, and the search range of learning rate was in the grid $[1.0\times 10^{-8}, 1.0\times 10^{-7}, 1.0\times 10^{-6},\dots, 10]$. Each configuration was run for 200 epochs at most. Moreover, we also told them that they could pause any configuration if they wanted to evaluate others, and even stop the whole tuning process only when they thought the accuracy could not be further improved and the number of total tuning epochs exceeded 600 at the same time. We collected results from 17 people and we determined the validity by checking whether the length of the trajectory was greater than 600 and removed 7 invalidity trajectories. Finally there remained 10 trajectories and we averaged them as the human performance in Figure 2. We hope our explanation can solve your concern.


As to the variable M, the number of repetitions, generally it should be as large as possible to account for the randomness of experiments. At the preliminary stage, we ran the evaluation protocol on CIFAR10 five times and found that each running showed a similar result. The CPE results of five runnings are reported below. Therefore, considering the practical implementation, we use M=3 for all experiments, and find that the variance part in Table 3, 4, and 9 is relatively small except for reinforcement learning, which does need more trials as you mentioned. In addition, as shown in Figure 8, the obtained trajectory is only unstable within the very beginning 10 epochs for CIFAR10, which exerted a slight impact on the final performance as well as the weighted metric CPE considering the total budget of more than 4000 epochs. On the other hand, we are still running more experiments to further make sure that our results are reliable but due to time limitation, they might not be updated in time. 

| Optimizer      | CIFAR10 | 
| :---             |    :----:   |
| SGD      |     88.74 ± 0.25      | 
| Adam      |    90.45 ±  0.14      | 
| RAdam      |    90.30 ± 0.10    | 
| Yogi      |      90.39 ± 0.06    | 
| LARS      |     90.22 ± 0.08    | 
| LAMB      |     90.21 ± 0.07     | 
| Lookahead      |     90.62 ± 0.05     | 



For your final question concerning Algorithm 6, it is adapted from Procedure 1 in Sivaprasad et al. (2020). We are sorry that we did not consider the randomness of different runnings even with the same configuration. A workaround to this question could be that we can store several trajectories of each configuration with different random seeds in the dictionary. For your second point, since this protocol is a bootstrap algorithm, we first sample a number of configurations from the original search space, and simulate Hyperband by resampling from the sampled settings. Therefore, it works for both continuous-valued and discrete-valued hyperparameters, as long as sufficient configurations are sampled.


Sivaprasad, Prabhu Teja, et al. "Optimizer benchmarking needs to account for hyperparameter tuning." Proceedings of the 37th International Conference on Machine Learning. 2020.



Thank you for the detailed response and updates to the paper. The changes have improved the paper, but I have a few more questions and concerns about the paper. 

Is there a reference to the statement that human hyperparameter tuning is still superior to automated hyperparameter optimization? Without a citation here, it is a hypothesis, not a statement of fact. However, there is some support from the human study to back it up. 

Regarding the human study, all the details should be made available (in the appendix). How were they recruited? What criterion was used to remove participants? What is their prior experience? What were the exact directions they were given about the problem? Without this information, it is difficult (impossible?) to correctly interpret the results. This study is deserving of its own paper, and as presented, it cannot be repeated nor correctly evaluated. 

The use of M=3 trials is vastly insufficient to compare stochastic outcomes with high confidence. Even when using the t-test, the standard is to use at least 30 samples. To claim M=3 is enough to "obtain reliable results," there needs to be experiments or theory showing that the amount of information gained from more trials will not change the result with high probability. This study also includes results from reinforcement learning algorithms, which are notorious for having highly stochastic results, and three trials are known to be insufficient (Henderson et al. 2018, Colas et al. 2018). 

Using a dictionary of hyperparameters and performances (Algorithm 6) is an interesting idea but not a valid approach. This method can only work if the optimization process is deterministic. Since neural networks are randomly initialized, mini-batches are randomly sampled, and randomness is a part of reinforcement learning algorithms, performance results will be stochastic, thus invalidating this approach. Furthermore, since the probability of randomly sampling a continuous-valued hyperparameter is zero, then this approach could only work with discrete-valued hyperparameters, which is not the case for optimizers used in this work. Is there some point I am missing why these concerns are invalid?





Colas, C., Sigaud, O., & Oudeyer, P. (2018). How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments. ArXiv, abs/1806.08295.

Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., & Meger, D. (2018). Deep Reinforcement Learning that Matters. AAAI.

**The score does not represent the initial review. It was updated following the discussions below.**

The paper proposes a new benchmarking protocol for optimizers in deep learning. The main argument is that previous papers have either neglected hyperparameter tuning or have employed hyperparameter search method that is far from how humans tune hyperparameters in practice. To mitigate the latter, the paper proposes to use HyperBand for automatic hyperparameter search. They show through a human study that HyperBand resembles human tuning performance more closely than random search. They then evaluate several optimizers on a multitude of tasks, including many recently proposed methods, under two scenarios: 1. Given an unfamiliar task, the effectiveness of each optimizer is measured through a metric that is cognizant of the hyperparameter tuning time. 2. After initially tuning on a subset of the dataset, how well does the obtained best hyperparameter configuration transfer to the full dataset? The main result is that the recently proposed optimizers are not substantially better than Adam.

By independently comparing recently proposed optimizers in two realistic scenarios, the paper would constitute a valuable contribution to the machine learning community. However, listed below, I take several issues that negatively impact my confidence in the protocol. If these issues could be resolved by answering my questions, I am inclined to update my rating upwards.

Major questions:
* The main argument for replacing random search with HyperBand is that HyperBand resembles human tuning behavior much more closely. You provide the results of a human study as evidence for this central claim. However, the paper provides little detail on the nature of the study. How many participants did you have? How were they sampled? What was the expertise of the participants? If they happen to be familiar with computer vision, there is a good chance that they have trained on CIFAR10 before, thus already knowing good hyperparameter values. This would contradict the scenario that you assume in your benchmark, which is unfamiliarity with the task. Non-experts in computer vision would likely take a lot longer to tune CIFAR10 to good performance, perhaps more closely resembling the curve of random search. In its current state, I have little confidence in this human study.

* You claim that in the evaluation protocol of Sivaprasad et al. (2020) each bad hyperparameter has to run fully. This is not true, because the protocol incorporates early stopping after two successive epochs in which the validation performance doesn't improve, thereby preventing at least _very_ bad configurations. It is obvious that HyperBand is a more sophisticated solution, but there is no direct evidence that it is so much better that it justifies replacing Sivaprasad et al. (2020)'s protocol. Perhaps you could include random search with a simple early stopping criterion in Figure 2?

* In Algorithm 1, the performance trajectory is computed M times, and you average the peak and CPE values over all repetitions, which is good to account for stochasticity. But you never mention how large M is. Is the cost of HyperBand low enough to allow for a sufficiently large M? Would it instead be possible to compute expected validation performance as is done in Sivaprasad et al. (2020) to decrease this cost significantly? If I understand HyperBand correctly, the hyperparameter configurations are still drawn independently via random search so that the expected validation performance could be computed, but I am not entirely sure.

* One of Sivaprasad et al. (2020)'s motivations for using random search is that it requires no hyperparameters (except for the search space, which is however assumed to be given by optimizer designers), which could otherwise inject some human bias into the evaluation process. In contrast, HyperBand does have additional hyperparameters that could introduce human bias. E.g., you state "We set $\eta = 3$ as this default value performs consistently well, [...]". Can we be sure that this choice is not biased towards some optimizers?

* Choi et al. (2019) make the case for choosing hyperparameter search spaces independently for each optimizer, noting that a unified search space may contain biases towards certain optimizers. You consider a unified search space, making the opposite argument by citing Metz et al. (2020). I could not find that argument in Metz et al. (2020). Could you please elaborate or point directly to their argument in their paper?

* Among your summarized findings you state that Sivaprasad et al. (2020) find Adam to usually outperform SGD. This may be misleading, since they only suggest Adam to be more likely to yield good performance than SGD if nothing is known about the task. This is a very similar result to what the performance profile shows in your Figure 4. On CIFAR10 and CIFAR100, Sivaprasad et al. (2020) also find an SGD variant to perform as well or better than Adam.

Suggestions for improving the paper:
* It would be good to provide not only average CPE and peak performance values, but also their variance.

* In the description of Scenario I you state that you compute the expected performance under different time budgets. It would be good to clarify what you mean (I suspect peak performance vs. CPE?).

Minor issues:

* In the related work on hyperparameter tuning methods, Sivaprasad et al. (2020) is falsely cited as a Bayesian optimization method.

* The paper claims that Sivaprasad et al. (2020) consider optimizer B in Figure 1 as better than optimizer A. This is not true; towards the end of Section 2 in Sivaprasad et al. (2020) acknowledge that the value of each optimizer depends on the available budget.

Thank you for your careful reading and valuable comments. We provide our response to your questions below.
 
**Q1 & Q2:** Performance is meta-algorithm that combines hpo and optimizer and specify the research problem

**A1 & A2:** Yes, the reported performance is just like what you mentioned, representing a meta-algorithm that combines the optimizer and HPO method. And since we are benchmarking different optimizers considering hyperparameter tuning costs, reporting that performance is suitable to compare optimizers under the same hyperparameter tuning algorithm. Generally, our protocol for scenario 1 tries to compare the end-to-end training efficiency with Hyperband, and the protocol for scenario 2 is to evaluate the performance shift of optimizers when the dataset changes. Currently many works only report the peak performance that an optimizer can achieve, regardless of the efforts in tuning hyperparameters. Our paper tries to answer a more practical question that how these optimizers perform when taking hyperparameter tuning into account. And our framework can make a thorough comparison between various optimizers and provide insights to practitioners when choosing the right optimizer in their experiments, especially for those who are unfamiliar with machine learning. In addition, for your statement that “one should use whatever global search method is best and not rely solely on hyperband”, it is true that the more powerful the global search method is, the more convincing the conclusion will be. After balancing both efficiency and effectiveness of several hyperparameter tuning methods, Hyperband is preferred due to its relatively superior performance and simplicity.


**Q3:** Details of human study

**A3:** We are sorry that insufficient details of human study are provided in the paper. Despite different hyperparameter tuning algorithms, human tuning by experts is still regarded as the most effective since they can early terminate bad trials based on their knowledge. That’s why Hyperband is slightly worse than human tuning. We conducted this human study to verify and support that Hyperband is an effective method and is even competitive with humans. As this human study is not involved in anything about ethics, we did not ask for approval from an institutional review board. Specifically, there are 10 participants in this human study, each of whom is sampled from people with computer science related backgrounds. With computer science backgrounds, the tuning trajectory obtained by humans can be regarded as the “best” among all hyperparameter tuning methods. Therefore, in this case, similar performance of Hyperband and human tuning just supports that Hyperband is a strong and effective HPO algorithm even without prior knowledge like unfamiliar practitioners. Moreover, another interesting direction can be investigating optimizers under human expert tuning. Being similar to humans, Hyperband can be seen as a surrogate model of humans, and give us some insights how the optimizer will perform with human beings involved in.

**Q4:** Variance and uncertainty

**A4:** Thank you very much for your suggestion. In our paper, we conducted three independent experiments for each task and found that $M=3$ is enough to account for randomness. We added the variance to our results for both CPE and peak performance values in Table 3, 4, and 9. In addition, we employed the probabilistic performance profile in Barreto et al. (2010) to quantify uncertainty in the aggregate measure, as shown in Figure 7. The probabilistic performance profile shows a similar trend as what is demonstrated in Figure 4.

**Q5:** A biased benchmark

**A5:** It is true that each benchmark is biased to favor one method. We have modified the description in our paper.

**Q6:** The sentence in page 4

**A6:** For this question, we are sorry that we forget to add more details about the scenario, and we have modified the sentence to “we argue that the random search procedure will overemphasize the importance of hyperparameters when tuning is considered.”

**Q7:** Not reward, should be return or cumulative reward

**A7:** Thanks for pointing out that question. We mistook the metric of reinforcement learning by using the term “reward” and have now modified it to the correct one, return, in our latest version.


**Q5:** Why unified search space

**A5:**  We just use a wide range of hyperparameter search space as in Metz et al., (2020) which conducts comparisons on 1000 optimization tasks. They did not mention the explicit reason for using a unified range, but we think one reason may be that the range of each optimizer for each dataset is very subjective, and if we want to make the evaluation more objective and without using much domain knowledge, the search space has to be wide and unified. This sufficiently wide unified search space also matches better to our scenario of unfamiliar practitioners. 

On the other hand, we want to emphasize that our contribution of proposing two new optimizer evaluation protocol is orthogonal to the definition of the hyperparameter range. If one prefers a predefined hyperparameter range for each optimizer or task, our evaluation protocol can still be used. 

**Q6:**  SGD variant performs as well or better than Adam

**A6:**  Thanks for pointing out the statement which is not very rigorous in our summarized findings. We have modified that part, and in this finding we try to emphasize that SGD can also have superior performance in some cases like image classification on CIFAR10 and CIFAR100, and such a configuration of SGD is less tedious to find under Hyperband than under random search.

**Q7:**  Report variance

**A7:**  We have included the detailed statistics of variance in Table 3, 4, and 9. We also attach two figures showing the end-to-end training trajectories with the error in Figure 8. Since it is hard to distinguish different optimizers in such figures, we primarily use tables of CPE and figures without the added error in our paper. Furthermore, we adopt a probabilistic performance profile to account for randomness in the aggregate measure in Figure 7. The probabilistic performance profile shows a similar trend as what is demonstrated in Figure 4.

**Q8:**  The description of Scenario I

**A8:**  We apologize that we did not describe the protocol for scenario 1 clearly in section 3. Your suspicion is correct and what we want to express here is that we design a protocol to compare different optimizers based on CPE and peak performance values and we have modified that sentence in our paper.

**Q9:**  Wrong citation
**A9:**  We are sorry that we carelessly cited Sivaprasad et al. (2020) as a Bayesian optimization method and we have corrected it in our latest version.

**Q10:**  Not precise statement of optimizer A and B

**A10:**  After checking Sivaprasad et al. (2020) again, it indeed states that the performance of two optimizers depends on the available budget. We have modified that part in page 4 and what we want to express is that even with the constrained budget, optimizer A can outperform optimizer B due to the advantage of early-termination of Hyperband.


We appreciate your helpful suggestions and present the response below.

**Q1:** Details of human study

**A1:** We are sorry that insufficient details of human study are provided in the paper. Specifically, there are 10 participants in this human study, each of whom is sampled from people with computer science related backgrounds. Although they are familiar with machine learning, it does not have conflict in the scenario where we want to evaluate different optimizers from the perspective of amateurs. With computer science backgrounds, the tuning trajectory obtained by humans can be regarded as a “good and practical one” among all hyperparameter tuning methods. Therefore, in this case, similar performance of Hyperband and human tuning just supports that Hyperband is a strong and effective HPO algorithm. Moreover, another interesting direction can be investigating optimizers under human (e.g., a data scientist) tuning. Being similar to humans, Hyperband can be seen as a surrogate model of humans, and give us some insights how the optimizer will perform with human beings involved in.

**Q2:** Random search with early stopping

**A2:** Thanks for your suggestion. We have added a trajectory obtained from random search with the early stopping strategy in Sivaprasad et al. (2020) (stop training after two successive epochs where the validation performance does not improve), and it can be observed in Figure 2 that with early stopping, random search can be improved, but there is still a gap from Hyperband, especially in the peak performance. In fact, early stopping can be also employed in Hyperband, and we plot this trajectory in Figure 2 as well. We can see that Hyperband with early stopping performs slightly better at the initial stage by terminating bad configurations much earlier. Thus, compared with random search, Hyperband still has an advantage in efficiency and effectiveness regardless of early stopping. Moreover, we noticed that the search space in Sivaprasad et al. (2020) is well calibrated by the authors and they generally would hardly produce very bad results. In that scenario, even with early stopping, random search would still run a configuration to a relatively deeper stage.

**Q3:** How many repetitions? Expected validation performance

**A3:** In our experiments, we use $M=3$ to repeat all tasks three times with different random seeds. Yes, since Hyperband is also doing random sampling, if $M$ is very large, we can use a similar strategy as (Sivaprasad et al., 2020) to pre-compute a library of random configurations and then efficiently evaluate our protocol. Similar to (Sivaprasad et al., 2020), if all the configurations are pre-executed with max_epoch, we can then simulate the Hyperband algorithm without running actual training and compute the average performance. In addition, we can further reduce the cost since bad configurations may be early terminated. Although we cannot know beforehand how many epochs are needed for each configuration, the following procedure presented in Algorithm 6 in Appendix D can be used.  The basic idea is that we keep a library of different hyperparameter settings. At the beginning, each config is runned with 0 epoch, and then during the simulation of Hyperband, we just retrieve the value from the library if the desired epoch of the current configuration is contained in the library. Otherwise, we run this configuration for $X$ epochs ($X$ depends on Hyperband), and store the piece of the trajectory in the library.

**Q4:** Hyperparameters of HPO method

**A4:** Compared with random search, Hyperband only introduces one extra hyperparameter $\eta$. To reduce the concern that additional hyperparameters may induce bias, we conduct an experiment for three optimizers with the task of CIFAR10 training, under four different values, $\eta=2,3,4,5$. Results are demonstrated in Table 8, and we can find that the choice of eta has little impact on the relative ranking among optimizers. In fact, the reduction factor $\eta$ just controls the aggressiveness of the early-termination of Hyperband. Based on our experimental results and the convention in Li et. al (2017), we choose the default value $\eta=3$ in our paper.


First, we want to thank you for your helpful suggestions. Following are our responses to your questions:

**Q1:** Combine Algorithm 1 and 2.

**A1:** Although Algorithm 1 and 2 look similar, their goals are different. For Algorithm 1, it is used to evaluate the efficiency of optimizers with hyperparameter tuning and thus we need to conduct it on the full dataset. On the other hand, Algorithm 2 obtains the best hyperparameter configuration under the partial dataset, and utilizes that setting on the full dataset to observe potential changes of an optimizer. They are two separate protocols, which are hard to combine.

**Q2:** The effectiveness of the solution and time cost

**A2:** The time cost depends on how many budgets are available. Specifically, in our paper, the unit of time budget is one epoch, then the total time will be $B_{epoch}*T_{epoch}$, where $B_{epoch}$ is the total available budget and $T_{epoch}$ is the running time for one epoch. There is no additional computational cost in our evaluation protocol, i.e., running our protocol once takes the same time as running one hyperparameter search (Hyperband in our paper).  Besides, we can reduce time significantly by paralleling our experiments. In our experiment on CIFAR10, we roughly evaluated 200 hyperparameter configurations, while the same time can only allow about 50 configurations in a random search. 

Moreover, if a user wants to run the evaluation protocol several times and average the results, we can further accelerate our evaluation protocol by resampling, shown in Algorithm 6 in our latest version. The basic idea is that we keep a library of different hyperparameter settings. At the beginning, the library is empty. And in each repetition, we sample a number of configurations required by running Hyperband once. During the simulation of Hyperband, we just retrieve the value from the library if the desired epoch of the current configuration is contained in the library. Otherwise, we run this configuration based on Hyperband, and store the piece of the trajectory to the library.

**Q3:** Details of algorithms

**A3:** We are sorry that we did not make definitions of some variables very clear in the paper. We have added the details of these variables in our latest version in section 3. For the variable P you mentioned, it is the exact metric of each task, and it represents accuracy for tasks such as image classification, and GLUE benchmark while P becomes negative log likelihood for VAE training. You can see the specific metrics in Table 2 to get the meaning of P clearly. As to the variable M, the number of repetitions, generally it should be as large as possible to account for the randomness of experiments. At the preliminary stage, we ran the evaluation protocol on CIFAR10 5 times and found that each running showed a similar result. Therefore, considering the practical implementation, we use M=3 for all experiments, and this value is sufficient to mitigate the influence of stochasticity, as shown in the variance part in Table 3.

**Q4:** Minor suggestions

**A4:** We appreciate your suggestions. We have corrected typos in our paper. For the paper format and figure location, since section 3 only describes the two protocols without mentioning experimental results, we remove the reference to Figure 5 on page 5 to avoid confusion.


Thank you for your careful reading and helpful comments. Here is our response to your questions.

**Q1:** Why hyperband?

**A1:** To compare Hyperband with humans, we just want to emphasize that Hyperband can terminate bad trials effectively as what humans usually do in practice. Since the human study is conducted among people who are familiar with machine learning, the tuning trajectory of humans can be regarded as an “ideal” early stopping strategy. Therefore, resemblance of Hyperband and human tuning can exactly support that Hyperband is quite a strong algorithm that finds the best hyperparameter fast. 

Besides, in Figure 2, we show that Hyperband surpasses random search, which supports the claim that Hyperband can terminate bad configurations early. We further add another experiment to compare Hyperband with another baseline, which applies a simple early stopping criterion on random search, and Hyperband still outperforms this method. The results and detailed discussions can be found in Figure 2 as well as the paragraph surrounding it. 

Furthermore, since Hyperband performs closer to human tuning, it can be considered as a surrogate model of humans and provide some insights in optimizer evaluation with human’s hyperparameter tuning.

**Q2:** More discussion about how to use the framework to choose the optimizer

**A2:** When deciding which optimizer to use for a specific task, people can refer to Table 3 and Table 8 in our paper. If the task happens to be included in Table 2, he/she can directly choose the one with the best CPE or best peak performance based on his/her goal of the task (easy to tune or high final performance). On the other hand, even though the desired task is not covered, people can also gain some insights from the results of the most similar task in Table 2. Besides choosing the optimizer, it will contribute to designing a new optimizer as well. Using our protocol to evaluate a new optimizer can show whether it has obvious improvement over existing optimizers, and can serve as a routine to judge the performance of the optimizer thoroughly. We have also added this discussion to section 5 in our paper.

**Q3:** Why is cpe the right metric to use?

**A3:** Peak performance is also an important metric when evaluating optimizers and we report both peak performance as well as CPE in our paper. Since CPE is a weighted sum of performance w.r.t. training epoch during the training trajectory, it is much suitable when we are considering the efficiency of end-to-end training given the budget. But if we do not care about the budget, then we just turn to peak performance for choosing the optimizer.


This work proposes two protocols for evaluating and comparing the quality of different optimizers. It points out that some existing, commonly used ways of comparing optimizers may over- or under-represent the amount of time that hyperparameter search can take. To fix this, it proposes using the Hyperband algorithm to guide the hyperparameter search. 

I think the authors make a convincing argument that existing approaches for comparing optimizers can downplay the role of hyperparameter search, which can be significant and can vary greatly across optimizers. I think the paper presents a fairly convincing approach for comparing optimizers, and thus for evaluating new ones against existing ones. 

I find the argument that Hyperband is a good choice because it more closely resembles human behaviour somewhat weak. Instead I would be more convinced by something showing that Hyperband (or whatever alternative) does a good job of terminating bad runs early, since this is the point of using a bandit algorithm over random search. 

I would also like to see some discussion of how others could use the proposed procedures  when deciding which optimizer to choose for their task.

Finally, I think the authors could do a better job of explaining why CPE is the right metric to use (i.e., why is considering peak performance not a good choice?) 

In general I like this work and recommend accepting it. However, I think it could be strengthened by more discussion of how this could be of use to the community in the future. This is especially important given that the no one optimizer seems to be universally best. 





This paper mainly proposed an evaluation framework for optimizers in machine learning jobs. It points out that existing benchmarking often applies best hyperparameter or random search hyperparameter. Their proposed framework re-evaluates the role of hyperparameter tuning in machine learning. 
It mainly deals with two cases, end-to-end training efficiency and data-addition training efficiency. The major findings are as follows.
1. Random search might lead to unnecessary training when the loss does not converge. Therefore, given limited budgets, it is better to have a benchmark strategy for finding the best hyperparamater.
2. Training on the same model repeatedly is necessary when there are new data. However, the existing hyperparameters might not be optimal when training data updates.
For end-to-end training efficiency, they assume users apply Hyperband and adopt \lambda-tunability to meansure the performance of optimizers. For every optimizer, the framework computes the CPE value based on the complete trajectory and evaluates the optimizers. For data-addition training efficiency, the framework extracts a subset to tune the hyperparameter and then apply them to the entire dataset and evaluate the performance of various optimizers. 

Strengths:
1. The overall structure is clear with a detailed explanation of the limitation of the existing optimizer benchmark scheme. The two cases the authors emphasized are practical and common in real-life scenarios.
2. I think the idea of applying Hyperband instead of random searching is quite reasonable based on the result in figure 2.
3. The experiments are conducted based on a variety of datasets and optimizer. The findings are well presented with different angles, task type, optimizer mechanism.
Questions:
1. It seems the algorithms 1 and 2 are similar. If it is possible to combine the algorithm together, like run the model on the subset to find several good optimizers and conduct it on the overall dataset.
2. I think overall the idea is good and easy to understand. However, I wonder if there is any way to support the effectiveness of the solution besides running different setups on the model.

Weaknesses:
1. I wonder the time cost of adopting this evaluation framework, like the time it needs to have a consolidated result on the optimizers' performance.
2. Some explanations regarding the details of the algorithms should be added beforehand. It is better to have an explanation of all the variables mentioned in the algorithm for clear referencing. For example, in the equation 1, the meaning of P, I assume it should be accuracy. Variable M, how it is decided and how it will affect the ultimate performance.
3. Some minor suggestions. There are some spelling mistakes in the paper. For example, in table 1, it should be non-adptive. And the format and location of figure and table could be improved. E.g. Figure 5 is mentioned in page 5 and located at page 8. It could be better to give a general picture of how the result looks like in the 3rd section first and then detailed experiment figures later.

Meta-review: This paper was referred to the ICLR 2021 Ethics Review Committee based on concerns about a potential violation of the ICLR 2021 Code of Ethics (https://iclr.cc/public/CodeOfEthics) raised by reviewers. The paper was carefully reviewed by two committe members, who provided a binding decision. The decision is "Significant concerns (Do not publish)". Details are provided in the Ethics Meta Review. As a result, the paper is rejected based Ethics Review Committee's decision .

The technical review and meta reviewing process moved proceeded independently of the ethics review. The result is as follows:

This paper studies the problem of evaluating optimiser's performance, which is important to show whether real progress in research has been made. It proposes several evaluation protocols, and used Hyperband (Li et al. 2017) to automate the tuning of each optimiser in the bench-marking study. Evaluations have been conducted on a wide range of deep learning tasks, and the paper reaches to a conclusion that none of the recently proposed optimisers in evaluation can uniformly out-perform Adam in all the tasks in consideration.

Reviewers agreed that the evaluations are extensive, however there are some shared concerns among reviewers. The paper argues that manual hyper-parameter tuning by humans is the right behavior to target for, which is the motivation to use Hyperband as an automating tool, and there is a human study to demonstrate that Hyperband tuning resembles human tuning behaviour. Some reviewers questioned about this desiderata choice that favours human tuning behaviour, also concerns on how the human study is conducted (and to what extend the human study itself is reflective enough for the human tuning behaviour in general).

Personally I welcome any empirical study that aims at understanding the real progress of a research topic, and I agree it is important to make rigorous automation tools in order to enable such a large scale study. Therefore, while the presented results are extensive, I would encourage the authors to incorporate the feedback from the reviewers to better examine their assumptions. 

Decision: withdrawn-rejected-submissions