Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper introduces a simple measure of tunability that allows to compare optimizers under varying resource constraints. The tunability of the optimizer is a weighted sum of best performance at a given budget. The authors found that in a setting with low budget for hyperparameter tuning, tuning only Adam optimizer’s learning rate is likely to be a very good choice; it doesn’t guarantee the best possible performance, but it is evidently the easiest to find well-performing hyperparameter configurations.

Comments:

The paper is easy to follow. The motivation of defining tunability of optimizer is a very interesting question, however, the study seems to preliminary and the conclusion is not quite convencing due to several reasons:

In section 3.2, to characterize its difficulties of finding best hyperparameters or tunability, the authors seem to try to connect the concept of “sharpness” of a minima in loss surface to the tunability of an optimizer, which is similar to comparing the loss landscape of minimums. However, while the authors made intuitive explanation about the tunability in section 2.2, I did not see the actual plot of the true hyperpaparameter loss surface of each optimizer to verify these intuitions. Can the author be more specific about the x-axis in the illustration 1.a and 1.b? If I understand correctly, they are not the number of trails.

In addition, the proposed stability metric seems not quite related with the above intuitions, as the illustrations (1.a and 1b) define the tunability to be the flatness of hyperparameter space around the best configurations, but the proposed definition is a weighted sum of the incumbents in terms of the HPO budgets. 

The definition of the tuning budgets is not clear, is it the number of trials or the time/computation budgets? The authors seems interchangeably using “runs” and “iterations”, which makes the concept more confusable. 

The authors further proposed three weighting schemes to emphasize the tunability of different stage of HPO. My concern is that is highly dependent  on the order of hyperparameter searched, which could impact the tunability significantly. For instance, in case of grid search HPO and 0.1 is the best learning rate, different search order such as [10, 1, 0.01, 0.1] and [0.1, 0.01, 1, 10] could results in dramatic different CPE and CPL. 

My major concern is the hyperparameter distributions for each optimizer highly requires prior knowledge. A good prior of one optimizer could significantly affect the HPO cost or increase the tunability, i.e., the better understanding the optimizer, the less tuning cost. My major concern is that the authors assume the hyperparameters to be independent (section 3.2), which is not necessarily true. Actually hyperparameters are highly correlated, such as momentum, batch size and learning rate are correlated in terms of effective learning rate [1,2], so as weight decay and learning rate are [3], which means using non-zero momentum is equivalent to using large learning rate as long as the effective learning rate is the same. This could significantly increase the tunability of SGDM. Another concurrent submission [4] verified this equivalence and showed one can also just tune learning rate for SGDM. The assumption of independent hyperparameters might be fine for black box optimization or with the assumption that practitioners have no knowledge of the importance of each hyperparameter, then the tunability of the optimizer could be different based on the prior knowledge of hyperparameter and their correlations. But it is not rigorous enough to make the conclusion that Adam is easier to tune than SGD.


The author states their method to determine the priors by training each task specified in the DEEPOBS with a large number of hyperparameter samplings and retain the hyperparameters which resulted in performance within 20% of the best performance obtained. Could the authors be more specific on the hyperparameters searched? Is this process counted in the tunability measurement?

[1] Smith and Le, A Bayesian Perspective on Generalization and Stochastic Gradient Descent, https://arxiv.org/abs/1710.06451

[2] Smith et al, Don't Decay the Learning Rate, Increase the Batch Size, https://arxiv.org/abs/1711.00489

[3] van Laarhoven et al, L2 Regularization versus Batch and Weight Normalization, https://arxiv.org/abs/1706.05350

[4] Rethinking the Hyperparameters for Fine-tuning
 https://openreview.net/forum?id=B1g8VkHFPH

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The main contributions of the submission are:

1. A comprehensive empirical comparison of deep learning optimizers, with their performance compared under different amount of hyper-parameter tuning (they perform hyper-parameter tuning using random search).
2. The introduction of a novel metric that tries to capture the "tunability" of an optimizer. This metric attempts to trade off the performance of an optimizer when tuned only with a small number of hyper-parameter trials, and its performance when carefully tuned. The metric is defined as a weighted average of the performance after tuning with i random trials, with i that goes from 1 to K. The weights of this weighted average and K are "hyper-parameters" of the metric itself. They use K=100 and suggest 3 possible choices of weights.

The paper appears to treat 2. as the main contribution. However, I do not think the metric they introduce is good enough to be recommended in future work, when comparing tunability of optimizers (or other algorithms with hyperparameters). The reason is that simpler methods provide just as much information, and do not rely on the need of interpreting the choice of the weights and K. This point is proven in the paper itself, where for example Figure 2 provides a more concrete and easier to interpret information than the tunability metric, similar graphs could be easily provided per dataset. Similarly, figure 3 as well as figures 5-7 and 8 in the appendix provide very good information about the tunability of the various optimizers without using the introduced metrics. Information similar (although not identical) to that summarized in table 5 could be captured by substituting the 3 metrics with the best performance after tuning for 4, 16 and 64 iterations respectively (just as examples).

A stronger contribution is 1., which however is somewhat incremental compared to similar comparisons made in the past. Comparisons which, while mentioned, should perhaps have been discussed and compared more in detail in this work. Overall, I do not feel the comparisons dramatically change the qualitative understanding the field has of the different optimizers and their tunability.

They also suggest that when the tuning budget is low, using Adam but tuning only the learning rate is beneficial, which could be a valuable and practical suggestion.

I enjoyed reading the submission, which is very clearly written, but due to the relatively limited value of the contributions, and excessive focus on the tunability metric which I do not feel is giustified, I slightly lean against acceptance here at ICLR. I do think, however, that it would make a great submission to a smaller venue or workshop.

Other comments/notes:
* One aspects that is mostly left out of the discussion (except from one side comment) is the wallclock time, as some optimizers might be on average quicker to train (for example due to quicker convergence), this can easily lead it to be quicker to tune even though it requires a higher budget of trials. I think it would be worth discussing this more.
* minor: in figure 8 in the appendix, the results after 100 iterations is, as far as I understand, over a single replication, so is not particularly reliable (and will always be 100% of a single optimizer)
* similarly to the above, if the configurations are always sampled from the same 100, confidence intervals in the graphs become less reliable as the budget increases.
review_title: Official Blind Review #1

Meta-review: The paper proposed a new metric to define the quality of optimizers as a weighted average of the scores reached after a certain number of hyperparameters have been tested.

While reviewers (and myself) understood the need to better be able to compare optimizers, they failed to be convinced by the proposed solutions. In particular (setting aside several complaints of the reviewers with which I disagree), by defining a very versatile metric, this paper lacks a strong conclusion as the ranking of optimizers would clearly depend on the instantiation of that metric.

Although that is to be expected, by the very behaviour of these optimizers, it makes it unclear what the added value of the metric is. As one reviewer pointed out,  all the points made could have been similarly made with other, more common plots.

Ultimately, it wasn't clear to me what the paper was trying to achieve beyond defining a mathematical formula encompassing all "standard" evaluation metric, which I unfortunately see of limited value.
Decision: Reject