Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper provides an empirical comparison of a set of first-order optimization methods for deep learning models. Those optimizers include stochastic gradient descent, momentum  method, RMSProp, Adam, Nesterov, and Nadam, which arguably covers all popular variants used in the literature. Although it is not the first empirical study on this topic, its conclusion differs slightly. The conclusion is a rather intuitive one: With proper parameter search, the 'richer', more powerful optimizers tend to work better, regardless of the downstream tasks. 

Pros: 
- Intuitive results with a well designed workloads and experiments. For practitioners that want to start their own hyperparameter search, the workloads and setups are likely to be useful. 

Cons:
- I am not entirely convinced that the inclusion relationship is indeed a major cause or indicator of different optimizers' performance. There is no theoretical justification; Empirically, if one takes two optimizers equally rich and tunes one of them more intensively, one should expect a better performance, too.

Suggestions:

- I think at least the basic definitions of different optimizers should be given in the main text. Otherwise, readers without detailed knowledge of all these optimizers cannot follow the paper. For example, the paper starts talking about the taxonomy of the optimizers with their corresponding hyperparameters in Section 3.2 before giving any functional form of the optimizers. 

- I would suggest the authors to follow the convention and use the term "hyperparameter" rather than "metaparameter". The readers of this paper are not primarily Bayesian, there is really no need to divert from the convention. Besides, the term "Bayesian hyperparameter tuning" is widely used even. 

- I wonder to which extent the network structures impact the choice of the hyperparameter (e.g., CNN vs. RNN). 
review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: 

First, I would like to note that the claim that SGD with momentum is a special case of Adam with large epsilon is technically wrong because Adam also includes the bias-corrected momentum estimates which SGD with momentum does not consider. It might seem like a small difference, however it is a form of learning rate schedule which most users of Adam are not aware of. In practice, however, Adam with large epsilon can approximate SGD with momentum. Just don't claim the equivalent since it is not there. 

I have some difficulties understanding the contribution of the paper. For example 
"When tuning all available metaparameters under a realistic protocol at scales common in deep learning,
we find that more general update rules never underperform their special cases."
In practice you do adjust hyperparameter search spaces to fit your conclusions, e.g., "We found that searching over (epsilon, alpha0/epsilon) was more efficient than searching over (epsilon, alpha)." Again, this alone invalidates your experimental setup since you biased it in order to fit your conclusion: "was more efficient" was found after running some prior experiments. 
Another situation where your experimental setup is unfairly tuned is when you used different hyperparameter ranges for similar hyperparameter, e.g. see D.2 for ResNet-32 on CIFAR-10 where 6 orders of magnitute difference was used for the initial learning of Momentum and 3 orders of magnitude difference for the initial learning rate of Adam. Similarly, there is a difference of 10x for ImageNet experiments. 

The paper suggests that 16 experiments is enough to produce good results. First, one should not forget the  special arrangements (see above) done for hyperparameter search space. Second, for any person working in black-box optimization it is clear that 16 experiments is next to nothing. It should give you something good in 1D, possibly in 2D if your search range is narrow. This is absolutely nothing in larger dimensions (providing that your benchmark in not super trivial and your hyperparameter search space is not absolutely boring when you already narrowed it around the optimum). After 16 evaluations you get pretty bad settings for most algorithms. 

Update: 

The paper uses a naive hyperparameter optimizer and runs it for a very small budget. The latter likely affects the conclusion of the paper that different training algorithms perform similarly. The authors seem to accept it by mentioning that this is the case for their tuning protocol/budget. 

If we would like to compare different training algorithms, we should optimize them on a set of problems using 2-3 state-of-the-art hyperparameter optimizers. Then, we should study how the best seen solutions so far and their robustness  change as a function of the computation budget (the maximum budget should be large enough). Then, one would see that the results are not that different for small budgets (a boring result) and somewhat different for larger budgets. Showing only the boring part seems more misleading than useful. 

Update#2:
As I mentioned in my review, Adam with large epsilon is not equivalent to momentum SGD but only approximates the latter. This is because the original Adam has a bias correction term and even if the same *global* learning rate schedule is used both for Adam with large epsilon and momentum SGD, they are not equivalent. In order to obtain the exact equivalence, one would need to either
1) drop the bias correction term of Adam and thus modify the algorithm in order to satisfy the claimed equivalence
or 
2) set a particular learning rate *for each batch pass* of Adam to simulate the effect of the bias correction term, this leads to a large number of hyperparameters - as many as the number of batch passes, this is intractable (the setup of the authors does not optimize such batch-wise hyperparameters, they are defined by a global scheduler as a function of batch/epoch index).  
If you avoid these modifications, then you can't claim the equivalence but only an approximation. If you don't have the equivalence of the two approaches and so momentum SGD is not a particular case of Adam, then the following sentence from the abstract is false: " As tuning effort grows without bound, more general optimizers should never underperform the ones they can approximate (i.e., Adam should never perform worse than momentum)". Again, strictly speaking, it is false that "Adam should never perform worse than momentum" because momentum SGD is not a particular case of the original Adam *unless* you drop the bias correction term or simulate it with tons of hyperparameters, one learning rate value per batch pass. Any global learning rate schedule *used for both* algorithms will not solve the issue because the bias correction term will remain. If you don't modify the learning rate schedule of Adam but only of momentum SGD, then you basically adjust your SGD by moving some part of Adam in it to claim the equivalence of the two, such actions can make pretty much every second algorithm equivalent to another. 

My main concern is described in the first Update. It is trivial that a more general optimizer is capable to perform at least as good as its particular case. What is not trivial is to clarify the interplay of computational budgets spent on hyperparameter tuning vs number of hyperparameters vs performance over time.  
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper presents experimental data supporting the claim that the under aggressive hyper-parameter tuning different optimizers are essentially ranked by inclusion --- if the hyper-parameters of method A can simulate any setting of the hyper-parameters of method B then under aggressive hyper-parameter tuning A will dominate B.  One way to achieve this rather trivially is to do a hyper-parameter search for B and then set the hyper-parameters of A so that A is simulating B.  But the point here is that direct and feasible tuning of A with dominate B even in the case where A has more hyper-parameters and where hyper-parameter optimization of A would seem to be more difficult.  An important conclusion is that without loss of generality one can always use Adam even in vision where SGD is currently the dominant optimizer used in practice.  Another important conclusion is that quasi-random hyper-parameter optimization is quite effective.
I find the claims to be intuitively plausible and I find the empirical results compelling.

Just a couple minor complaints.  First,"Hyper-parameter" please not "meta-parameter".  I find the attempt to overturn standard usage inappropriate.  Second, I found most of section 3 uninformative.  I don't think algorithm 1, or the definition of a first order optimizer adds anything to the paper.  Inclusion can be easily defined in complete generality for any parameterized algorithm.
review_title: Official Blind Review #2

Meta-review: This paper examines classifiers and challenges a (somewhat widely held) assumption that adaptive gradient methods underperform simpler methods.

This paper sparked a *large* amount of discussion, more than any other paper in my area. It was also somewhat controversial.

After reading the discussion and paper itself, on one hand I think this makes a valuable contribution to the community. It points out a (near-) inclusion relationship between many adaptive gradient methods and standard SGD-style methods, and points out that rather obviously if a particular method is included by a more general method, the more general method will never be worse and often will be better if hyperparameters are set appropriately.

However, there were several concerns raised with the paper. For example, reviewer 1 pointed out that in order for Adam to include Momentum-based SGD, it must follow a specialized learning rate schedule that is not used with Adam in practice. This is pointed out in the paper, but I think it could be even more clear. For example, in the intro "For example, ADAM (Kingma and Ba, 2015) and RMSPROP (Tieleman and Hinton, 2012) can approximately simulate MOMENTUM (Polyak, 1964) if the ε term in the denominator of their parameter updates is allowed to grow very large." does not make any mention of the specialized learning rate schedule.

Second, Reviewer 1 was concerned with the fact that the paper does not clearly qualify that the conclusion that more complicated optimization schedules do better depends on extensive hyperparameter search. This fact somewhat weakens one of the main points of the paper.

I feel that this paper is very much on the borderline, but cannot strongly recommend acceptance. I hope that the authors take the above notes, as well as the reviewers' other comments into account seriously and try to reflect them in a revised version of the paper.
Decision: Reject