Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This work proposes to learn an adaptive LR schedule, which can adjust LR based on current
training loss and the information delivered from past training histories. It adopts a (parameterized) LSTM as the schedule, which is meta-learned following the existing work meta-weight-net. Extensive experiments are conducted to show its effectiveness on image classification, compared to pre-defined policies. 

In general, this paper addresses an important problem of LR schedule and the suggested method is easy to follow and the experimental results are promising. However, there are several issues that need to be addressed to improve the work further.

1.	The novelty seems quite limited – it appears to be a straightforward application of meta-weight-net to the problem of LR scheduling.
2.	Analysis on the learned optimizers is lacking, such as the convergence and speed.
3.	It is not clear why the learned optimizer can be transferred to new heterogeneous tasks. Some deeper insights might be helpful. For example, can a LR scheduler learned from an image classification task be transferrable to a task of object detection?
4.	Other forms of parameterized LR scheduler should be discussed and compared, such as MLP. It is not obviously clear why LSTM is a good choice -- the structure of LSTM seems quite complicated.
5.	Missing relevant work: Meta-SGD (https://arxiv.org/abs/1707.09835) which also meta-learns a LR scheduler.

review_title: Interesting work on important problem. Novelty and transferability need more justifications.

final_score: rating: 4: Ok but not good enough - rejection
review_content: ## Summary

This paper proposes a learning-to-learn type approach for step size schedules. An LSTM is used to predict step sizes based on observed values of the training loss. It is meta-trained with the goal of achieving maximal validation loss. The proposed method is evaluated empirically on image and text classification problems.


## Rating

The basic idea of the paper is solid. Fully meta-learned optimization methods have shown promising results, but have been brittle when transferred to settings that deviate substantially from those they have been meta-trained on. Using a standard optimizer update direction and only meta-learning the step size schedule seems like a reasonable idea to tackle the major pain point (step size tuning) while potentially providing more robustness. But, in my opinion, the idea is not executed and evaluated up to the standard of this conference. I have several issues with the quality and transparency of the experimental evaluation and the proposed method shows only limited success in the presented experiments. I am detailing my concerns below. Moreover, the paper is not really well-written. **In its current state, I recommend rejecting this paper.**


## Major Comments

1) The quality of the empirical evaluation is questionable in my opinion.
    a) The settings and competing methods are chosen somewhat erratically. For example, why is there no comparison to a simple multi-step schedule and/or SGDM for the text datasets?
    b) There is no explanation whatsoever of the used protocol for tuning the hyperparameters of the methods involved in the experimental comparison.
    c) Why is there no comparison to other automatic adaptation techniques (L4, HD) in the transfer experiments? The paper says that “Since the methods […] are not able to generalize, we do not compare them here”. I don’t understand what that is supposed to mean. Since they aren’t meta-learned, there is no notion of generalization for this method. But they could still be applied in the transfer setting as baselines.

2) This is a purely empirical paper proposing a practical method for learning rate scheduling. The impact of such a paper comes down, to a large extent, to the success of the proposed method. Unfortunately, the results are really a mixed bag. The proposed method outperforms baselines only in a small subset of the experiments: mainly when meta-training the MLR-SNet on the identical dataset/architecture (Section 4.1), and even there it gets outperformed by a multi-step schedule when using SGD with momentum. In the transfer experiments (Section 4.2), the proposed method is matched or outperformed by one of the simple baselines in almost all cases. Of course, there might be an argument to be made in terms of the effort of manual tuning that goes into the different baselines. But that point is not really driven home in the paper and I am skeptical for two reasons: (i) the absence of a clear hyperparameter tuning protocol (see previous point) and (ii) the fact that the method still relies on a manually-chosen scaling factor $\gamma$ for the step size. Overall, I don’t think the experimental results are convincing.

3) I am missing a discussion of the computational and memory cost of the proposed method. The meta-updates of the LSTM weights (line 6 in Algorithm 1) requires backpropagating through an interval of $T_\text{val}$ past iterations. Such back propagation through unrolled training trajectories are usually very memory-intensive (the entire state of the model has to be kept in memory for the unrolled iterates)  and adds considerable computational cost. In Section 4.1, this supposedly very costly method is compared to baselines that are essentially for free without any consideration given to that discrepancy in computational cost. Of course, this is a one-off cost when using a pre-trained MLR-SNet as a plug-and-play step size scheduler, but the results for this transfer setting are not really convincing (see above).

4) The paper proposes to adapt the learning rate based on the observed loss values. Fully meta-learned optimizers use gradient observations in addition to the loss. Why did you choose not to use the gradient information. To keep the method light-weight? Because a good step size can be chosen exclusively based on loss values? I think this modelling choice should be discussed and justified in the paper.


## Minor Comments

5) The quality of the writing is subpar. I am trying not to get hung up on linguistic mistakes, but occasionally it is really hard to decipher the meaning of certain sentences. There are also plenty of typos and stylistic errors. The bib-file could need a thorough review: you are citing arXiv versions of several papers that have been published in peer-reviewed venues, conferences are cited only by their acronym, et cetera.

6) It would have been nice to investigate the versatility of the proposed method by applying it to other base update directions than SGD(M). For example, could the MLR-SNet be applied on top of Adam as well?
review_title: Solid idea, but execution not up to the standard of ICLR

final_score: rating: 6: Marginally above acceptance threshold
review_content: In this work, the authors use an LSTM to meta-learn learning rate schedules. This LSTM depends only on the validation loss at time t. They train this LSTM for some tasks and they show that it gives good performance when compared with baselines. Then they transfer one of these trained LSTMs to different tasks and gives good results. 

Understanding learning rate schedules is an interesting problem in machine learning and the authors seem to provide a useful blackbox learning rate scheduler.

-In the SM, they discuss that they have to choose 3 MLR-SNET for transferability. This part is very confusing and should be written more clearly. This should also be mentioned in the main text because the impression there is that one just transfered the MLR-SNET as is. 
-How many runs are needed to meta train the LSTM? Is the transferable LSTM coming from only one run?

-How robust is the performance of MLR-SNET with respect to different initial learning rates?

-It would be nice if the authors could add tables reporting the accuracies, it is not clear from the plots how good/bad the performance of MLR-SNET is. The Adam baseline is pretty weak, it should be supplemented with learning rate schedules, one should be able to get acc > 73% with Adam+schedules on imagenet.
review_title: Interesting paper proposing black box generated learning rate schedules. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: The paper proposes to parameterize learning rate (LR) schedule with an explicit mapping formulation. This learnable structure allowed the proposed meta-trained MLR-SNet to achieve good LR schedules. For validation, the proposed method is evaluated on both image and text classification benchmark with various network architectures and datasets, as well as transfer the learned network for new task or architectures. 

Justification of rating: The paper solve a practical problem that is not handled in the existing literature. Despite the straightforwardness of the proposed approach and the methodology, this work has the potential to bring high impact to the research community. 

Strengths:
+ This work proposed to parameterize the LR schedule with a MLR-SNet.  The results shows it is more flexible and general than the hand-picked LR schedule.
+ The meta-learned approach allow the learned model to be applied to unseen data.
+ The paper provide comprehensive experiment to validate the efficacy of the proposed model. 

Comments:
- Experiment on Penn Treebank shows the convergence of the proposed MLR-SNet is slower than SGD and Adam. The paper argue that it predicts LR according to training dynamics by minimizing the validation loss. Please provide more details why is this a more intelligent way to employ validation set. Is this also observed in any other dataset?
- This work transfer the learned LR schedules on CIFAR-10 with ResNet-18 to several other datasets. Has the author  try to transfer MLR-SNet learned with other source dataset? How will the model trained with different model/dataset behave when transferred to new datasets or networks. It might be interesting to explore if the certain type of network architecture (more complex or simple) would learned a more generic model.
- In the "Formulation of MLR-SNet", it states that the input $h_{t-1}$ and the training loss are preprocessed by a fully-connected layer $W_1$ with ReLu activation function. Please describe the purpose of this layer. 
review_title: A practical and potentially high impact work for SGN based DNN training.

Meta-review: The paper proposed a meta-learning method for tuning the learning rate. In the discussion, reviewers agreed that the key issue is that the empirical evaluation is not yet sufficient to demonstrate the efficacy of the method. In particular, this is an especially pressing issue given that there are now many meta-learning methods for tuning the learning rate (none popular in practice though), and the paper does not compare to any of them. Relatedly, most reviewers found that the novelty of the method is not clearly established and discussed in the paper. 

Based on the above, I have to recommend rejecting the paper. I would like to thank the authors for submitting the work for consideration to ICLR. I hope the feedback will be useful for improving the work.

Decision: Reject