Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper presents an algorithm (AutoLRS) to tune learning rate schedules based on Bayesian Optimization with Gaussian processes and loss curve forecasting with exponential functions. The authors conduct experiments on three large scales settings in computer vision and natural language processing.

Strengths:
- This is probably one of the first attempts to use BO to fit LR schedules on the fly.
- The exponential forecast model seems a simple but effective idea. 
- Experiments are very large scale and show that particular instantiations of the proposed algorithm achieve noticeable speed-ups and can also improve final performances in some cases.

Weaknesses:
- The method has several hyperparameters, which potentially limits its significance and applicability.
- Unclear contribution of the BO part of the method.
- Comparisons with other methods and experimental protocol could be improved.

Comments:
- AutoLRS has several hyperparameters.  The authors mention four in the relative paragraph, but considering also Sec 4.4 I personally count at least another three. They argue that some standard choices work well across the three experimented settings, but some of them (e.g. the number of iterations) are clearly task-dependent. Besides, there are also several choices related to the BO part of the method that feel somewhat arbitrary to me (choice of kernel function for the GP/acquisition function). Would they work also in another domain/learning setting, or should they also be considered as hyperparameters of AutoLRS? The presence of all these configuration parameters may render the application of the proposed algorithm impractical (lowering the significance of this work).
- The paper reads overall fairly well, but I think it could have been organized better. Some information (e.g. description of training details and in-depth reports of the numerical results) could have been postponed to the appendix and, vice-versa, some content of the appendix e.g. comparison with other methods should have been included in the main text.
- There is no ablation of the BO part of the method.  I find this quite surprising and I would like to see this in the rebuttal. Specifically, looking at Figure 6 (last column), it seems that after 10 samples the points are more or less evenly spaced (in log scale). I wonder if the BO part of the method adds that much. One could simply pick 10 values on a grid (or sample randomly) and only run the exponential forecasting part of the method. This reminds me of [1]; which could be related. 
- I have doubts about the comparisons with other methods. A) It is not clear to me how the authors searched for the parameters of the schedulers (CLR/SGDR), besides that they did >= 10 trials. I would argue that manual search is not a fair method in this setting  B) Within the hypergradient descent class, the authors only reported some results with the method of Baydin et al. but none with Donini et al. (which is a generalization of HD that also considers optimizing with a validation set). C) Runs with Hyperband use exponential decay. Depending on which functional form they use, sampling the decay rate uniformly in (0,1) may be unnecessarily punitive. E.g. if the schedule is $\eta_0 \delta^{t}$ then clearly I would expect that only values of $\delta$ close to 1 would yield decent results. These points can be clarified in the rebuttal phase.
- The authors should be very careful when using the word "optimal" and related. As they say, the method that they propose is a greedy one that performs a (very) partial stagewise optimization. Therefore it is not clear to me concerning what they consider the various quantities as `"optimal". I strongly suggest replacing the occurrences with words such as "tuned" or similar. 

Minor comments:
- The experimental results do not report any kind of measure of uncertainty, for instance, related to the random seed. While this is understandable for the large scale experiments, I would have appreciated a set of smaller-scale experiments to consider this.
- Algorithm 1 does not use explicitly $\eta_{min}$ and $\eta_{max}$. To improve clarity, authors should specify that line 5 solves a *constrained* optimization problem over $\eta$, in the range $[\eta_{min}, \eta_{max}]$
- A small explanation of the “LR range test” would be useful, due to the fact it is used to calibrate 2 of the 6 hyperparameters of the proposed method.


References
[1]Zhang, Michael, et al. "Lookahead optimizer: k steps forward, 1 step back." Advances in Neural Information Processing Systems. 2019.


review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: The auto method presented in the paper is based on Bayesian optimization, but with an modification to make it less expensive. The method should be interesting to machine learning community. 

The is clearly written and relatively easy to understand. The idea proposed in new and the experimental results demonstrate its effectiveness. 

Pros: 
1. The method is new and seems to be effective in speedup  the training several types of models. Using a forecast model to reduce running time is an interesting idea, thought the forecast model is an exponential time-series forecasting model. 
2. The paper demonstrates the effectiveness of this method by pretty solid experiments. The three models used are popular and important ones. 
3. The paper also describe two practical improvements( Gradually increase $\tau$ over the course of training, and Minimizing training loss in early stage) 

Cons:
1. The method itself introduced a few hyperparameters including k, $\tau$, etc. Tuning them seems a manual process. The paper can say a bit more how and why choose values used in the experiment
2. The paper focuses on learning rate. In practice, there are many other hyperparameters that have impact on performance. How to make sure the results reported in paper is independent of the setting of other hyperparameters?  
review_title: An automatic learning rate adjustment method based on Bayesian Optimization

final_score: rating: 7: Good paper, accept
review_content: 
## Overview
---
Training deep neural networks is typically done using gradient-based methods with either pre-defined learning-rate schedules or off-the-shelf adaptive optimizers (such as Adam). The former can not reliably align with the non-linear loss landscape, while the latter add additional hyperparameters to tune. This paper proposes an algorithm for automatically tuning a learning-rate schedule. The method works by modelling the training dynamics and adapting the learning-rate to optimize performance on the validation set. The method does end up introducing additional hyperparameters,
but the empirical results suggest some degree of robustness to them. Empirical performance is investigated on a number of widely-used models which are expensive to tune.

**Overall, I recommend this paper for acceptance.** I have some minor concerns which are detailed below, but generally the clarity was excellent throughout and the approach seems novel.

It seems that there are various "hidden" hyperparameters in addition to the ones stated. The algorithm seems to perform well in a variety of problems with them fixed, but there is a concern that the default settings suggested in the paper could just happen to fit well in these particular problems. For example,
  - **Page 4**: Don't you need to know (or guess) the variance of the noise term $\varepsilon$ to fit a GP? If so, how was this noise variance chosen? what's the sensitivity to it?
  - **Page 14, Appendix A.2**: *"Hence, we apply a simple gradient descent method that starts from an initial b, computes the linear least squares w.r.t. a, c under b, search for the next b by the gradient descent method, and repeats these two steps."*
    The problem in a,c can be solved in closed form but how is the step-size for descent in b chosen? Is this a tuned hyper-parameter? How was it chosen?
  - **Page 15, Appendix A.3**: *"At the end of each iteration, we remove the loss values that are among the farthest 3%"* How was 3\% chosen? This one in particular seems like it would depend a lot on how noisy the particular data is

My other concern is that there's no indication of statistical spread in any of results (neither tables nor figures). It  would be nice to see at least standard error bars so we know the variance in performance of the proposed method, and how it compares to the baselines. I assume this is because the experiments presented are too expensive to perform enough trials to get statistically significant results, but it would be worth-while to add an appendix showing results on smaller tasks on which variance can be assessed. 

## Clarifications
---
- **Page 2**: *"Moreover, since it directly minimizes the validation loss, it does not only accelerate the convergence but also improves the generalization."* This isn't true necessarily, one could just be overfitting to the validation set. This is why we tune on a validation set and test performance on a separate test set --- to ensure that the performance truly does generalize to unseen data

## Minor Comments (which did not affect my score)
---
- **Page 4**: "loss decreases exponentially" vs. "linear convergence" (iteration complexity) is an unfortunate quirk in the optimization literature, as the equivalence is easy to miss. It would be useful for many readers if the equivalence was clarified --- in a footnote, perhaps --- or if the same convention was used in both phrases.
- I think an interesting addition to the empirical results would be something like "computation to performance" for each of the methods. For example, how much compute is it going to cost me to reach a certain level of performance using each method when we account for time spent tuning hyperparameters. I am unsure what the best measure would be here, but tuning non-trivial models is often so expensive overall that it would really showcase the benefits of the proposed method. (To be clear, *I am not requesting such a result*)

review_title: The paper is well-written and has an interesting approach. Minor concerns regarding hidden hyperparameters and statistical significance of empirical results.

final_score: rating: 7: Good paper, accept
review_content: Summary:
This paper uses Bayesian optimization (BO) to dynamically tune the learning rate during the course of training of DNNs.  In every stage of training, the algorithm firstly uses BO to explore different learning rates with the help of a parametric exponential model for learning rate extrapolation, and then applies the selected learning rate at the current stage. The algorithm is applied to the training of state-of-the-art DNN models, and is shown to outperform the original learning rate schedules, as well as other methods for learning rate scheduling.

Strong points:
- I think this is a very interesting and practical application of Bayesian optimization (BO).
- The way BO is used for learning rate selection in every stage is very clever, especially the trick of reverting back to the checkpoint model parameter before the evaluation of every LR, which resolves the non-stationarity issue due to evolving model parameters.
- The experiments are conducted using state-of-the-art DNN models such as transformers and BERT.
- The paper is in general vey well written, although some places describing the algorithm could be improved as specified below.

Weak points/questions:
- I think the brief descriptions of the algorithm especially in the Abstract and Introduction are not very accurate and sometimes confusing (especially the 3rd paragraph of the Introduction). I couldn't really tell how the algorithm works until I've read Algorithm 1. I feel it would be easier to understand if these descriptions clearly say that: in every stage, BO is firstly used to select the learning rate, where each learning rate evaluation during BO starts at the same model parameter checkpoint and is early-stopped using an exponential model; after BO, the selected optimal learning rate is used for throughout the current stage.
- Since your algorithm incurs additional computational cost due to BO, I think it would be a more fair comparison if you compare the runtime (instead of training epochs/steps as in Figures 1 and 2) of different algorithms. I wonder whether the proposed algorithm still has an advantage after taking into account the additional computational cost due to BO.
- In the experiments, the number of BO iterations is selected as k=10. I think this is a small number of iterations for BO. Given that BO needs to perform exploration at the initial iterations, I wonder is how your algorithm will perform if you replace BO by pure exploration (i.e., simple random search)?
- The proposed Algorithm 1 seems extremely similar to Refs [1] and [2] given below, and hence they should be referenced. Both [1] and [2] also use an exponentially decaying kernel to model the evolution of the learning curve, and hence early-stop some evaluations of hyperparameters (learning rate).

[1] Bayesian optimization meets Bayesian optimal stopping, ICML 2019.
[2] Freeze-thaw Bayesian optimization, Arxiv, 2014.
review_title: Very Interesting Application of Bayesian Optimization

Meta-review: The paper has been actively discussed, both during and after the rebuttal phase. I am thankful for the active communication that took place between the authors and some of the reviewers.

The paper was, overall, found quite clear, with an interesting methodology (especially the introduction of a forecasting step) and a solid large-scale experimental evaluation. As a result, it is recommended for acceptance.

However, several concerns remained after the rebuttal phase and we strongly encourage the authors to try to improve the following aspects of their submission:
* Clarify as much as possible (notably in the light of the ablation studies further added in the paper) the importance & impact of the BO component (which cast some doubts among the some reviewers on its necessity to get good performance)
* Transparently discuss the choice of, _and the robustness with respect to_, the “hyper-hyperparameters” of the proposed method (e.g., k, tau, tau’, kappa, tau_max, mini-batch size of validation set,...). Such an in-depth discussion is essential to fully demonstrate the practical value of the method. 

Decision: Accept (Poster)