Title: SGDR: Stochastic Gradient Descent with Warm Restarts
Authors: Ilya Loshchilov
Frank Hutter
Venue: ICLR 2017
Keywords: Deep learning
Optimization

Abstract: Restart techniques are common in gradient-free optimization to deal with multimodal functions. Partial warm restarts are also gaining popularity in gradient-based optimization to improve the rate of convergence in accelerated gradient schemes to deal with ill-conditioned functions. In this paper, we propose a simple warm restart technique for stochastic gradient descent to improve its anytime performance when training deep neural networks. We empirically study its performance on the CIFAR-10 and CIFAR-100 datasets,   
where we demonstrate new state-of-the-art results at 3.14\% and 16.21\%, respectively. We also demonstrate its advantages on a dataset of EEG recordings and on a downsampled version of the ImageNet dataset. Our source code is available at \\ \url{https://github.com/loshchil/SGDR}

TL;DR: We propose a simple warm restart technique for stochastic gradient descent to improve its anytime performance.

Review discussion:
A quick clarification about appendix 7.2: does this mean that in the main paper, your method always uses twice the samples/compute per epoch compared to the baselines?
Then all the learning curves (with epochs as x-axis) would be misleading?
My main question is about annealing versus restarts -- numerous papers have used various (linear/geometric/stepwise) learning-rate annealing schedules for deep learning (going back to at least Ciresan et al '12). I see the key innovation in your paper in its anytime performance, achieved via restarts+annealing.
However, you mostly focus on final results, and on these the conclusions are less clear, also because I'm missing comparable annealing-no-restart baselines (e.g. T_0=200), and because some of the best results did very few restarts...
Also, Fig 1 is really difficult to read: the legend should not hide the data, and there are plotting tools that help disambiguate curves in better ways... in general the plotting quality is disappointing: it might be worth rethinking what you want to demonstrate and plot jsut the right information without the clutter (fig 4 with its actual anytime-test-errors seems much clearer, although there the y-axis is odd.)
All methods (SGDR and baselines) use the same number of samples/compute per epoch
You are right that when only the final results are considered, the annealing alone can do a great job.
We tried to emphasize it in our original discussion section:
"Our results suggest that \textit{even without any restarts} the proposed aggressive learning rate schedule given by eq. (\ref{eq:t}) is competitive w.r.t. the default schedule when training WRNs on the CIFAR-10 and CIFAR-100 datasets."
We are running experiments with T_0=200. We didn't do that before because our primary variant of SGDR with T_0=10, T_mult=2 (respectively, T0=1, T_mult=2) delivers its last (within 200 epochs) recommendation at epoch #150 (respectively, epoch #127).
Still, it is interesting to see the results with T_0=200.
We modified Fig. 1 to make it more readable. We also removed yellow color from our plots. We fixed the y-axis of Fig. 4.
Why did you only take the median of two runs in the last rows of Table 1 and is that the superior or inferior run?
Other remarks:
- Missing full stop at bottom of page 2 before "Our empirical results".
- The black font on the dark blue in Fig 3. is hard to read on printouts
Please note that if there are an even number of observations, we follow the convention to compute the median as the mean of the middle two numbers. The only reason we used only 2 runs of WRN-28-20 was the total cost of training for all listed scenarios.
Thank you for the remarks!
- The missing full stop is fixed.
- We modified Fig. 3 to make it lighter and hopefully more readable with a greater font size.
This paper describes a way to speed up convergence through sudden increases of otherwise monotonically decreasing learning rates. Several techniques are presented in a clear way and parameterized method is proposed and evaluated on the CIFAR task. The concept is easy to understand and the authors chose state-of-the-art models to show the performance of their algorithm. The relevance of these results goes beyond image classification.
Pros:
- Simple and effective method to improve convergence
- Good evaluation on well known database
Cons:
- Connection of introduction and topic of the paper is a bit unclear
- Fig 2, 4 and 5 are hard to read. Lines are out of bounds and maybe only the best setting for T_0 and T_mult would be clearer. The baseline also doesn't seem to converge
Remarks:
An loss surface for T_0 against T_mult would be very helpful.
Also understanding the relationship of network depth and the performance of this method would add value to this analysis.
This an interesting investigation into learning rate schedules, bringing in the idea of restarts, often overlooked in deep learning. The paper does a thorough study on non-trivial datasets, and while the outcomes are not fully conclusive, the results are very good and the approach is novel enough to warrant publication.
I thank the authors for revising the paper based on my concerns.
Typos:
- “flesh” -> “flush”
This heuristic to improve gradient descent in image classification is simple and effective, but this looks to me more like a workshop track paper. Demonstration of the algorithm is limited to one task (CIFAR) and there is no theory to support it, so we do not know how it will generalize on other tasks
Working on DNNs for NLP, I find some observations in the paper opposite to my own experience.
In particular, with architectures that combine a wide variety of layer types (embedding, RNN, CNN, gating), I found that ADAM-type techniques far outperform simple SGD with momentum, as they save searching for the right learning rate for each type of layer.
But ADAM only works well combined with Poliak averaging, as it fluctuates a lot from one batch to another.
Revision:
-  the authors substantially improved the contents of the paper, including experiments on another set than Cifar
-  the workshop track has been modified to breakthrough work, so my recommendation for it is not longer appropriate
I have therefore improved my rating
Thank you for your review. Please consider this quick reply:
i) We also show that SGDR works very well on a completely different dataset of EEG recordings (see section 7.1 in the Supplementary Material). We realize it was not the best decision to only mention this additional result in the supplementary material, and we'll pull it into the main part of the paper in the next revision.
ii) We do not counterpose ADAM to SGDR because the latter (warm restarts) could also be applied to the former. We focused on SGD with momentum since it dominates for optimizing state-of-the-art residual networks for image classification with large DNNs (e.g., on CIFAR, SVHN and ImageNet).
iii) Regarding the theoretical results, please consider our note given in Section 2.2:
"The authors showed that fixed warm restarts of the algorithm with a period proportional to the conditional number achieves the optimal linear convergence rate of the original accelerated gradient scheme. Since the condition number is
an unknown parameter and its value may vary during the search, they proposed two adaptive warm restart techniques (O'Donoghue & Candes, 2012)". Unfortunately, the results of (O'Donoghue & Candes, 2012) are limited to strongly convex functions and their extension to non-convex stochastic functions does not seem trivial."
In response to your review, and to demonstrate the generality of SGDR we will also try to get some compute resources to evaluate it on additional datasets over the holidays.
We thank all reviewers for their positive evaluation and their valuable comments. We've uploaded a revision to address the issues raised and briefly reply to the reviewers' concerns here.
Experiments on additional benchmarks
In order to address the concerns of AnonReviewer4 about our method's generality, we added two more benchmarks to the main paper.
1. We had already included additional results for another domain in the appendix of the original submission. That domain is motor-control decoding based on EEG data, which, due to the large noise present in brain signals, is very different from visual object recognition in general, and CIFAR in particular. We realize that it was not the best decision to only mention those results in the supplementary material, and we have now moved them to the main paper (new Section 4.4).
In fact, our work is in part motivated and funded by a project aimed at accelerating the processing of this sort of brain data (see acknowledgments).
2. We've now added a new experiment on a new downsampled version of the ImageNet dataset (new Section 4.5), which is much harder than the CIFAR benchmark because there are 1000 classes and because the target concept often only occupies a small part of the image. Due to the computational expense of training on 1 million images, these results are very preliminary -- e.g., we didn't tune hyperparameters yet.
While the dataset is difficult for both the default learning rate schedule and SGDR, SGDR achieved much better results. An interpretation of this might be that, while the initial learning rate seems to be very important, SGDR reduces the problem of improper learning rate selection by quickly annealing from the initial learning rate to 0.
We would also like to point out that the simplicity and generality of SGDR let Huang et al use it to develop snapshot ensembles (https://openreview.net/forum?id=BJYwwY9ll) and show that the combination of SGDR and snapshot ensembles works very well on a variety of datasets.
New results for T_0=200
We added experiments for the experiment with annealing for 200 epochs (T_0=200) AnonReviewer3 suggested. The results show that longer annealing works great on CIFAR-10 but not that great on CIFAR-100.
When considering both datasets, annealing with $T_0=200, T_{mult}=1$ is comparable to $T_0=10, T_{mult}=2$.
However, the latter has a better any-time performance. Our initial text attempted to highlight the advantages of the annealing alone with "Our results suggest that \textit{even without any restarts} the proposed aggressive learning rate schedule given by eq. (5) is competitive w.r.t. the default schedule when training WRNs on the CIFAR-10 and CIFAR-100 datasets." We adjusted that sentence with "(e.g., for $T_0=200, T_{mult}=1$)" to point the reader towards the relevant result / setting. The results of $T_0=200, T_{mult}=1$ for WRN-28-20 are based on 1 run only because our computational resources were used for 5 runs of WRN-28-10 and new experiments with the downsampled ImageNet above.
Readability of figures
Following the suggestions of AnonReviewer1 and AnonReviewer3, we improved the readability of some figures. It might be the case that the introduction of the curve with $T_0=200$ suggested by AnonReviewer3 slightly degraded the readability again, and we would be happy to move results with that setting (or any other settings the reviewers suggest) to the appendix.
While the curves for SGDR tend to be lean, the ones for our baselines are very busy since they are quite noisy. If the reviewers preferred this, we would be open to the possibility of moving most plots that include many methods to the supplementary material and focussing on plots with a few select methods in the main paper.
Summary
We are glad that the reviewers agree that our method is novel, simple, and effective, and we hope to have demonstrated its generality better now as well. We would also like to point out that it was already very useful, allowing Huang et al to develop snapshot ensembles, which directly use SGDR (as discussed in our Section 4.3, where we also present results with snapshot ensembles). We expect that the combination of SGDR and snapshot ensembles will be the new default strategy for many researchers, and we believe this is an additional point in favor of accepting our paper as a conference paper at ICLR.
Thank you again for your reviews!

Meta-review:
role: PC
data: comment: All reviewers viewed the paper favourably, with the only criticism being that seeing how the method complements other approaches (momentum, Adam) would make the paper more complete. We encourage the authors to include such a comparison in the camera ready version of their paper.

Decision: Accept (Poster)