Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: Why did you only take the median of two runs in the last rows of Table 1 and is that the superior or inferior run?

Other remarks:

- Missing full stop at bottom of page 2 before "Our empirical results".
- The black font on the dark blue in Fig 3. is hard to read on printouts

review_title: CIFAR evaluation

final_score: rating: 7: Good paper, accept
review_content: This heuristic to improve gradient descent in image classification is simple and effective, but this looks to me more like a workshop track paper. Demonstration of the algorithm is limited to one task (CIFAR) and there is no theory to support it, so we do not know how it will generalize on other tasks

Working on DNNs for NLP, I find some observations in the paper opposite to my own experience. In particular, with architectures that combine a wide variety of layer types (embedding, RNN, CNN, gating), I found that ADAM-type techniques far outperform simple SGD with momentum, as they save searching for the right learning rate for each type of layer. But ADAM only works well combined with Poliak averaging, as it fluctuates a lot from one batch to another.

Revision:
-  the authors substantially improved the contents of the paper, including experiments on another set than Cifar
-  the workshop track has been modified to breakthrough work, so my recommendation for it is not longer appropriate
I have therefore improved my rating
review_title: Great trick worth publishing, but is there enough material for a full paper?

final_score: rating: 7: Good paper, accept
review_content: This paper describes a way to speed up convergence through sudden increases of otherwise monotonically decreasing learning rates. Several techniques are presented in a clear way and parameterized method is proposed and evaluated on the CIFAR task. The concept is easy to understand and the authors chose state-of-the-art models to show the performance of their algorithm. The relevance of these results goes beyond image classification.


Pros:

- Simple and effective method to improve convergence
- Good evaluation on well known database


Cons:

- Connection of introduction and topic of the paper is a bit unclear
- Fig 2, 4 and 5 are hard to read. Lines are out of bounds and maybe only the best setting for T_0 and T_mult would be clearer. The baseline also doesn't seem to converge

Remarks:
An loss surface for T_0 against T_mult would be very helpful. Also understanding the relationship of network depth and the performance of this method would add value to this analysis.

review_title: An effective method to improve convergence of neural network training

final_score: rating: 7: Good paper, accept
review_content: This an interesting investigation into learning rate schedules, bringing in the idea of restarts, often overlooked in deep learning. The paper does a thorough study on non-trivial datasets, and while the outcomes are not fully conclusive, the results are very good and the approach is novel enough to warrant publication. 

I thank the authors for revising the paper based on my concerns.

Typos:
- “flesh” -> “flush”
review_title: null

final_score: rating: null
review_content: A quick clarification about appendix 7.2: does this mean that in the main paper, your method always uses twice the samples/compute per epoch compared to the baselines? Then all the learning curves (with epochs as x-axis) would be misleading?

My main question is about annealing versus restarts -- numerous papers have used various (linear/geometric/stepwise) learning-rate annealing schedules for deep learning (going back to at least Ciresan et al '12). I see the key innovation in your paper in its anytime performance, achieved via restarts+annealing. However, you mostly focus on final results, and on these the conclusions are less clear, also because I'm missing comparable annealing-no-restart baselines (e.g. T_0=200), and because some of the best results did very few restarts...

Also, Fig 1 is really difficult to read: the legend should not hide the data, and there are plotting tools that help disambiguate curves in better ways... in general the plotting quality is disappointing: it might be worth rethinking what you want to demonstrate and plot jsut the right information without the clutter (fig 4 with its actual anytime-test-errors seems much clearer, although there the y-axis is odd.)
review_title: restarts versus annealing

Meta-review: All reviewers viewed the paper favourably, with the only criticism being that seeing how the method complements other approaches (momentum, Adam) would make the paper more complete. We encourage the authors to include such a comparison in the camera ready version of their paper.
Decision: Accept (Poster)