Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents an extensive independent benchmark of 14 popular optimizers on a variety of deep learning tasks from DeepOBS (Schneider et al. 2019). They compare them at three different tuning budgets and with 4 learning rate schedules. The authors are realistic about their setup. They acknowledge that different people might have different desires for such a benchmark, and they are clear about the choices they made to keep the experiments feasible. While there is no clear-cut answer that tells practitioners which optimizer to use in what scenario and how to tune it, these experiments are valuable and I believe it is important that these results are shared with the community.

The quality of the presentation and the writing is good. 

In terms of novelty, the authors model the target audience slightly differently from previous work (Schneider et al. 2019, Choi et al. 2019, Sivaprasad et al. 2020). I am not convinced that this approach is better per se than others, but a different angle and a different set of optimizers is a valuable contribution to the community. I believe that the description of (Sivaprasad et al. 2020) in Section 1.1. is not entirely accurate. They do not compare hyperparameter tuning methods, but rather benchmark optimizers similarly to this work at a continuum of hyperparameter tuning budgets (all with random search).

Finally, let me share two concerns:

1. The intro mentions three contributions: (i) performance varies greatly, (ii) trying different optimizers works as well as tuning a single one, (iii) they identify a significantly reduced subset of algorithms and parameter choices that perform well across experiments. Points (ii) and (iii) are interesting, but (ii) is formulated quite imprecisely and it is hard to see on which results this is based. I inspected Figures 9 to 12 in the Appendix and conclude "this might be true, but it is hard to see". I believe a quantative statement would be more useful/meaningful. Similarly, for point (iii) it is not clear from which results this is concluded, and what the high-performing subset is. Such a list would be valuable to many practitioners and should be clearly stated in the main text.

2. Figure 3 shows relative improvement across tasks. Any such measurements of 'improvement' are dependent on re-shifting or re-scaling of the loss, and are not necessarily meaningful when aggregated into a plot like this. Consider accuracy: measuring relative improvement (accuracy 1 / accuracy 2) would yield drastically different numbers than (error 1 / error 2 = (1-acc1) / (1-acc2)).
review_title: Important, well executed, experiments with unfortunately no clear-cut outcome

final_score: rating: 4: Ok but not good enough - rejection
review_content: ### Summary
The authors of this paper conducted a thorough evaluation of deep learning optimizers across different compute budgets and learning rate schedules. They provide detailed analysis of the results. The design decisions are well-reasoned and explained throughout the paper.

### Comments
* As the authors note, there is certainly value in understanding the practical tradeoffs between optimizers: "for most algorithms, the only formal empirical evaluation is offered by the original work introducing the method"
* The writing is clear and easy to follow.
* Many of the findings are useful in the context of the DeepOBS dataset. For instance, Figure 3 highlights the diminishing returns of increasing the budget when tuning hyperparameters.
* Open-sourcing the data is great and beneficial for the community.

* The benchmark would benefit from a larger scale dataset(s). I'm not in favor of solely adopting DeepOBS, as past papers have shown systematic differences in evaluation at different scales [1][2]. Investigating whether there are systematic differences in optimization on larger problems e.g. machine translation or ImageNet would be valuable. 
* As Reviewer 4 mentions, the momentum parameter should be tuned as 1 - \rho.

### Recommendation / Justification
I vote that this paper is below the acceptance threshold. There are many things to like about the approach taken is this paper, as highlighted above. However, the lack of larger scales datasets lessens the significance of the conclusions.

I'd increase my score if concerns about the datasets used were addressed. I understand it is challenging to do so during the rebuttal period, but I strongly believe that larger scale datasets would strengthen the work significantly.

### Minor feedback 
* A tabular form of Figure 4 would improve clarity.
* I think it is worth acknowledging techniques for averaging the weights of neural networks, as these can have a substantial impact on final performance (Polyak averaging, exponential moving average, Stochastic Weight Averaging).
* I believe it is also worthwhile to benchmark a second-order optimizer. While the compute per step is more expensive, the comparison could be made fair by using the same compute budget for each optimizer.
* I am surprised by the choice of \alpha when tuning the lookahead optimizer. My suspicion is that tuning the momentum and learning rate is more fruitful than trying low values of \alpha.


[1] Frankle, Jonathan, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. "Stabilizing the lottery ticket hypothesis." arXiv preprint arXiv:1903.01611 (2019).

[2] Gale, Trevor, Erich Elsen, and Sara Hooker. "The state of sparsity in deep neural networks." arXiv preprint arXiv:1902.09574 (2019).


Edit: After the rebuttal period, I maintain my original rating but am increasing the confidence of my evaluation from 4 to 5. I thank the authors for their hard work and engaging in discussion. I agree with Reviewer 3 that tuning with a fixed seed and the lack of search space refinement is a major weakness. The lack of a larger dataset further limit the applicability of the results. As such, I do not believe the paper in its current form should be accepted to ICLR. 
review_title: Official Review

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:
This paper benchmarks popular optimizers for training neural networks. The experiments consider all possible combinations of 3 different tuning budgets, and 4 different fixed learning rate schedules on 8 deep learning workloads for 14 optimizers. The paper highlights two main observations: 1) there is no clear dominating optimizer, and 2) selecting from a pool of optimizers with their default parameters is often as good as tuning a fixed optimizer.
The main contribution of this work is the open-sourced experiment results for a multitude of cases which can serve as a baseline for future research in optimizers for deep learning.

Strengths:
- The biggest strength of this work is that all the details regarding the benchmarking protocol are presented and justified. In addition to this, the caveats of the protocol are stated explicitly. The explicit and transparent nature of this work can help practitioners make more informed decisions, and prevent them from misinterpreting the results to something more than what is presented.
- To my knowledge, this is the first paper to compare a large number of optimizers, selected based on popularity in the research community.
- The problems considered are of varying difficulty, and includes tasks other than image classification. 
- Different levels of tuning budgets are considered, with the smallest budget being 1 trial (evaluating the default setting), and the largest being 50 trials.

Weaknesses
- The current tuning procedure is unstable. Tuning with the seed fixed is like optimizing for the specific seed. Furthermore, the hyperparameters that produce the best performance for a specific seed tends to be more unstable (which the authors agree to in appendix C); evaluating such an unstable setting on different seeds unnecessarily penalizes the optimizer. What makes more sense to me is to tune with the same number of trials, with each trial having a different seed, and using bootstrapping to compute the statistics (mean, standard deviation, etc). What we want to see with the tuning experiments is how well the optimizer can do (as an upper bound), and how stable it is. With the current approach, it’s hard to observe the best performance, at which point, I’m not sure how meaningful the error measurements are. All in all, I think it’s more meaningful to show how varying the best optimizer performance can be when tuning with a different set of seeds (since everyone uses different seeds), than to show the variance of a specific set of hyperparameters that is most likely unstable, on many seeds.
- I believe the experiments lack results for the “well-tuned” case. The optimizers all use a fixed hyperparameter search range for all problems, which can’t be competitive over different tasks of varying difficulty. I understand that this study assumes the model practitioner to be someone who doesn’t have prior knowledge about the optimizer, let alone the search ranges. However, I think it’s reasonable to believe that a practitioner would try to verify the search range by testing some hyperparameter values before committing 25 or 50 trials to the search range. Likewise, it would be useful to see results with a more calibrated search space per test problem. This can be done with the 50 trial budget by, for example, using 25 on a wide search space, and the other 25 on a more refined search space. At the very least, it would be useful to see the performance vs hyperparameter value plotted for the existing experiments to see whether the ranges could have been trivially improved (for example, if the performance tends to increase/decrease with the learning rate, but the best performance lied on the boundary of the search space, the range could have been shifted). This sort of tuning procedure is not unknown in the community. See [1, 2, 3].

Currently, I think the weaknesses outweigh the strengths of the paper. It is my understanding that optimizer comparisons should be done between reasonably good versions of the optimizers, and I think better versions of the optimizers could have been presented with a different methodology, given the same computational budget.
 


[1] Wilson, Ashia C., et al. "The marginal value of adaptive gradient methods in machine learning." Advances in neural information processing systems. 2017.

[2] Shallue, Christopher J., et al. "Measuring the effects of data parallelism on neural network training." arXiv preprint arXiv:1811.03600 (2018).

[3] Choi, Dami, et al. "On empirical comparisons of optimizers for deep learning." arXiv preprint arXiv:1910.05446 (2019).


Update:

I have read over the changes made by the authors, and also the other reviewer’s responses. I am maintaining my score, because I don’t think the current version of the paper is enough of a contribution to get accepted. As mentioned in my responses below, I would be happy to accept a future version of the paper that addresses my comments above. 
review_title: Review of "Descending through a Crowded Valley — Benchmarking Deep Learning Optimizers"

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: Overview:
Overall I believe this paper is extremely well written and organized. The introduction and limitations are very useful groundings of optimization research, and I hope that the community reads this paper and internalizes its message of making more meaningful research in optimization (instead of yet-another-Adam-variant!).

In a previous comment I brought up a serious concern about the tuning ranges of momentum-like parameters, which I believe could bias the results towards optimizers that were tuned with ranges whose lower end was 0.5 (Adam, AMSBound, AMSGrad, AdaBound, LA(RAdam), NAdam, RAdam). Besides this concern, I overall would normally argue for a very strong acceptance, but until this is corrected I am unsure I can recommend accepting. I look forward to discussing correcting this with the authors however, and am willing to dramatically raise my score!

**NOTE: updated score after seeing author replies and updated draft. I believe that this work is exemplary in terms of being careful about baseline construction, something that is unfortunately too often overlooked in our field. Additionally, it rigorously highlights another important point that I believe many often overlook, that "there are now enough optimizers"; community effort should be diverted from introducing small variations around Adam and instead invest focus on more meaningful improvements in scaling machine learning optimization. I do still believe that ImageNet and a larger transformer experiment would be extremely valuable to add to a later version, and hope the authors can eventually secure the computing power to add these.**

Pros:
-Table 2 is extremely useful and I think should be additionally put on a GitHub where it can be updated with links to papers.
-The authors clearly put a lot of careful thought into how to study and present these results, taking into account numerous caveats that almost all other papers totally ignore, such as the limitations of their study, the importance of tuning ranges, learning rate schedules, etc.

Concerns:
-I am very understanding and sympathetic to the issue of compute constraints, but I do believe that the study would be even more useful if a ResNet-50/Imagenet and a Transformer pipeline were used. The authors discuss that GANs and RL are not included, and those seem like very different types of optimization to me so I am more understanding of not including them. If the authors are academics then I have seen researchers have success in the past with getting grants from cloud providers, namely the lottery ticket hypothesis line of work and the TF Research Cloud (I found out about it through this paper https://openreview.net/forum?id=S1gSj0NKvB), which can easily be used to heavily tune ImageNet and Transformer runs. Outside of that, according to this benchmark https://dawn.cs.stanford.edu/benchmark/ImageNet/train.html, the cost of ImageNet runs has come down considerably in recent years, to $10-20 per run.
-The experimental results could partially be explained by the No Free Lunch theorem, and the authors could at least reference this in the section.
-What regularization, if any, was used for these problems, and was that also tuned? One could argue that, while optimization and regularization are in theory orthogonal to each other in what they try to accomplish (train vs test performance), they are both part of the update rule whose hyperparameters are being tuned. Also I believe it is important to be careful and note if coupled or decoupled weight decay/L2 is being used is important; I assume it is coupled because the DeepOBS code that is referenced uses L2 regularization, which means that optimizers that use preconditioning (Adam-like optimizers) could be impacted by this differently than those that do not (SGD/Momentum).

Writing:
-In the “Tuning method” paragraph in section 2.3, “In case there is no prior knowledge provided in the op.cit. we chose” seems like a syntax error.

Prior work:
The authors do a very thorough literature search, and properly reference and discuss similar prior studies.

Additional feedback, comments, suggestions for improvement and questions for the authors:
-Awesome job providing per-step values for results, it would be further useful to have code that could easily plot them side-by-side so that future researchers would be further encouraged to include them in their figures.
-May be worth noting that, in addition to the optimizer hyperparameters, one could also tune the batch normalization momentum/epsilon, for additional performance gains.
-Figure 3 seems very useful, however I believe it would be much better presented as a series of box plots. A nice recent example of this is Figure 2 in https://arxiv.org/abs/1906.02530. This could also be done for Figure 4 where each nested box is an individual optimizer, and it shows the distribution over runs for each optimizer for each problem.
-The trapez schedule always seems to be the best, and I wonder if this is due to only one of the learning rate ramp up, which has been shown to be beneficial to stabilize training (although it is unclear if this is required), or the learning rate becoming quite small at the end, which has been shown to be necessary so that the optimizer can better learn the noisy directions of the objective. It would be useful for future work to consider each of these learning rate schedules (ramp-up, ramp-down) separately, although that requires even more compute.

review_title: Very well done with one experimental concern

Meta-review: Contributions of this type are very important for the community. There is a great deal of confusion among practitioners about how to pick optimizers. Perhaps worse, there is confusion among optimization researchers about how to demonstrate the effectiveness of their novel algorithms on deep learning tasks. I applaud this paper as one of the best attempts to make sense of this confusion.

Unfortunately, I am recommending that it is rejected. This was an extremely difficult decision. This paper was very thoroughly discussed by reviewers, both with the authors and after the feedback phase. I agree with R4 that this paper is exemplary in terms of its breadth of optimizer choices. I also agree with R3 that this paper's choices regarding hyperparameter search spaces and seed fixing significantly diminish the contribution of the paper at hand. The key issue that persuaded my decision centered on whether the paper's evidence supported its conclusions.

The two key conclusions that I want to highlight are:

1. *evaluating multiple optimizers with default parameters works approximately as well as tuning the hyperparameters of a single, fixed optimizer*

2. *different optimizers exhibit a surprisingly similar performance distribution compared to a single method that is re-tuned or simply re-run with different random seeds*

These conclusions can only be supported if optimizers are well-tuned. Based on R3's remarks and a quick reading of the paper, I am concerned that the use of fixed search spaces means that these optimizers cannot be considered well-tuned. This concern splits into two sub-concerns.

1. I appreciate the author's desire to encode "no prior knowledge about well-working hyperparameter values". Unfortunately, I don't think this is realistic or possible. The learning rate range used in this paper did not include 1e100 for good reasons, all of which depend on the prior knowledge of our community. This isn't just a glib concern, the apparently neutral search spaces may bias the conclusions towards well-known methods whose hyperparameters are well-understood.

2. I am also skeptical of the choice to use the same range for hyperparameters with "similar naming". The reason is that these hyperparameters *may have been misnamed by the inventors* and may, in fact, play very different roles in the dynamics of optimization.

Top-line conclusions have a way of becoming memes in our community. Therefore, it is critical that conclusions, as stated, are actually supported by the experimental design and the empirical evidence. Unfortunately, I am not confident that this is is the case for the paper at hand.

It is clear that this paper represents a heroic effort by the authors. I am aware of the challenges involved in getting this type of paper published and of the urgent need for them. I hope that the authors address the concerns that I expressed and the concerns of the reviewers in a future submission.
Decision: Reject