Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: In this paper, the authors introduce a hypergradient optimization algorithm for finding learning rate schedules that maximize test set accuracy. The proposed algorithm adaptively interpolates between two recently proposed hyperparameter optimization algorithms and performs comparably in terms of convergence and generalization with these baselines.

Overall the paper is interesting, although I found it a bit dense and hard to read. I frequently found myself having to scroll to different parts of the paper to remind myself of the notation used and the definition of the different matrices. This makes it harder to evaluate the paper properly. The proposed algorithm seems interesting however, and the experimental results look quite impressive.

I have a few concerns regarding the experiments however, which explains my score:

1. In figure 2, does MARTHE diverge for values of beta greater than 1e-4? This seems to indicate that MARTHE is somehow more sensitive to beta than the other variations used. Do the authors have any intuition about what might be causing this behavior?

2. The initial learning rate for SGDM and Adam was fixed at certain values for all experiments. Why is this a reasonable thing to do? It feels like MARTHE should be compared to SGDM and Adam at least when the initial learning rate is tuned for these properly. Otherwise, it doesn't feel like a fair evaluation? To the best of my knowledge, the final achieved accuracies achieved with MARTHE however seem quite competitive with the best results typically reached with tuned SGDM on the convolutional nets used in the paper.

3. The learning rate schedules found by MARTHE seem to be somewhat counterintuitive. While an initial increase matches the heuristic of warmup learning rates frequently used when training convnets, the algorithms seems to decrease down the learning rate after that even quicker than what the greedy algorithm HD does. Do the authors have any intuition why this can lead to such a big improvement in performance over HD?

4. Is it possible to provide some sort of estimate of how much computation MARTHE requires compared to a single SGDM run? How feasible is to test this algorithm on a bigger classification model on ImageNet?

I think this paper is borderline, although I am leaning towards accepting it given the impressive empirical results. It would really improve the paper if the readability was improved, as well as if larger experimental results were included.

====================================

Edit after rebuttal:
I thank the authors for their response. I am happy with their response and am sticking to my score.
review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: The results are given only for the CIFAR datasets. Even for these two datasets the authors use very outdated networks, e.g., the best error rate for CIFAR-10 is in order of 6 percent. One should use contemporary/bigger networks, e.g., WRNs published in 2016 would give you about 4 percent. 
1) The initial learning rate for CIFAR-10 and CIFAR-100 are different, respectively 0.1 and 0.0003. The use of such small initial learning rate for CIFAR-100 is not motivated especially given that it is usually in order of 0.05 or 0.1 when resnets are considered. 
2) The authors don't compare to cosine annealing without restarts which is a pretty strong baseline. 
3) The authors compare to SGDR but don't set its initial number of epochs in a way that its last restart convergences at around 200 epochs. 
4) The proposed method has its own hyperparameters which greatly influence the results as shown in the appendix. I suspect that setting these hyperparameters is exactly what controls the slope of the learning schedule. 

Overall, the results are not convincing. The authors show that the previous adaptive approaches don't work well on the CIFAR datasets (despite the fact that their authors claimed the oppositve) and I don't think that the paper contains enough material to avoid the situation that futures approaches will claim similar things about the current study. 
review_title: Official Blind Review #1

Meta-review: First, I'd like to apologize once again for failing to secure a third reviewer for this paper. To compensate, I checked the paper more thoroughly than standard.

The area of online adaptation of the learning rate is of great importance and I appreciate the authors' effort in that direction. The authors carefully abundantly cite the research on gradient-based hyperparameter optimization but I would have appreciated to also see past works on stochastic line search (for instance  "A stochastic line-search method with convergence rate") or statistical methods ("Using Statistics to Automate Stochastic Optimization").

The issue with these methods is that, despite usually very positive claims in the paper, they are not that competitive against a carefully tuned fixed schedule and end up not being used in practice. Hence, it is critical to develop a convincing experimental section to assuage doubts. Unfortunately, the experimental section of this work is a bit lacking, as pointed by both reviewers. I would like to comment on two points specifically:
- First, no plot uses wall-clock time as the x-axis. Since the authors state that it can be up to 4 times as slow per iteration, the gains compared to a carefully tuned schedule are unclear.
- Second, the use of a single (albeit two variants) dataset also leads to skepticism. Datasets have vastly different optimization properties and, by not using a wide range of them, one can miss the true sensitivity of the proposed algorithm.

While I do not think that the paper is ready for publication, I feel like there is a clear path to an improved version that could be submitted to a later conference.
Decision: Reject