Title: Snapshot Ensembles: Train 1, Get M for Free
Authors: Gao Huang
Yixuan Li
Geoff Pleiss
Zhuang Liu
John E. Hopcroft
Kilian Q. Weinberger
Venue: ICLR 2017
Keywords: Deep learning
Computer vision

Abstract: Ensembles of neural networks are known to be much more robust and accurate than individual networks. However, training multiple deep networks for model averaging is computationally expensive. In this paper, we propose a method to obtain the seemingly contradictory goal of ensembling multiple neural networks at no additional training cost. We achieve this goal by training a single neural network, converging to several local minima along its optimization path and saving the model parameters.  To obtain repeated rapid convergence, we leverage recent work on cyclic learning rate schedules. The resulting technique, which we refer to as Snapshot Ensembling, is simple, yet surprisingly effective.  We show in a series of experiments that our approach is compatible with diverse network architectures and learning tasks. It consistently yields  lower error rates than state-of-the-art single models at no additional training cost, and compares favorably with traditional network ensembles. On CIFAR-10 and CIFAR-100 our DenseNet Snapshot Ensembles obtain error rates of 3.4% and 17.4% respectively.


Review discussion:
- A trivial way to obtain equivalent local minima is to permute the hidden units. Have you thought about ways to discourage this? (I'd be interested to what degree this happens, which might be measured by comparing activations in sorted order or something like that.)
- It seems on first glance difficult to evaluate hyperparameter choices when you're intentionally allowing the objective function to grow. Can you comment on the feasibility of hyperparameter search in this model class? One particular (perhaps obvious) improvement that as far as I can tell isn't mentioned is that, at each convergence, the new member's validation set predictions can be saved and a running sum/average maintained, enabling cheap computation and monitoring of the *ensemble* validation error.
- Is it necessary to enable re-training of the entire model with the cycled learning rates or would a few of the top layers suffice?
A small comment: please consider that your figures may be read in grayscale (or by the colorblind) and try to make them legible without relying on color.
1. Such equivalent local minima will ultimately lead to identical softmax predictions, although they are difficult to be observed from the hidden layers. According to our Snapshot Ensemble results, the models obtained from different learning rate cycles should have give very different softmax predictions.
Hence we can safely conclude that the trivial solutions did not occur in our experiments. Probably they are automatically discouraged by the randomness of the SGD algorithm and the high dimensionality of the parameter space.
2. This is an interesting point. We did not emphasize the hyperparameter selection method in our manuscript because we simply re-used the settings of training a single model. One would expect a set of particularly tuned hyperparameters will further improve the ensemble performance. In that case, we may use your suggest idea to perform hyperparameter selection.
3. We tried this in an earlier experiment, but it did not perform as well as retraining the entire model.
Moreover, it is less straightforward to implement and does not necessarily save much training time when data augmentation is used.
So we opted out this choice.
1. In Figure 6, the diversity of activations is studied with respect to different learning rate schedules. It would be interesting to see how this diversity, which is key for ensembles, compares with a different ensembling technique, e.g. (1) "true ensembles", (2) ensembles from dropout as described by Gal et. al, 2016 (Dropout as a Bayesian Approximation).
2. In Table 2 (pg. 7), is the evaluation done using just the final snapshot or an ensemble? If using an ensemble, please specify the size of the ensemble (m) for clarity.
1. Thanks for the suggestion. We will include the result for “true ensembles”, and try to compare with ensembles from dropout.
2. It is the ensemble result. The ensemble size is shown in the first column. We will make this clearer.
The work presented in this paper proposes a method to get an ensemble of neural networks at no extra training cost (i.e., at the cost of training a single network), by saving snapshots of the network during training. Network is trained using a cyclic (cosine) learning rate schedule; the snapshots are obtained when the learning rate is at the lowest points of the cycles. Using these snapshot ensembles, they show gains in performance over a single network on the image classification task on a variety of datasets.
Positives:
1. The work should be easy to adopt and re-produce, given the simple techinque and the experimental details in the paper.
2. Well written paper, with clear description of the method and thorough experiments.
Suggestions for improvement / other comments:
1.
While it is fair to compare against other techniques assuming a fixed computational budget, for a clear perspective, thorough comaprisons with "true ensembles" (i.e., ensembles of networks trained independently) should be provided.
Specificially, Table 4 should be augmented with results from "true ensembles".
2. Comparison with true ensembles is only provided for DenseNet-40 on CIFAR100 in Figure 4. The proposed snapshot ensemble achieves approximately 66% of the improvement of "true ensemble" over the single baseline model. This is not reflected accurately in the authors' claim in the abstract: "[snapshot ensembles] **almost match[es]** the results of far more expensive independently trained [true ensembles]."
3. As mentioned before: to understand the diversity of snapshot ensembles, it would help to the diversity against different ensembling technique, e.g. (1) "true ensembles", (2) ensembles from dropout as described by Gal et. al, 2016 (Dropout as a Bayesian Approximation).
Thanks for the detailed suggestions.
1. We agree that “true ensemble” results are important as they can be viewed as a lower bound of the test error given by Snapshot Ensemble. We’ll include more of those in the final version.
2. Thanks for pointing out this. We will fix it in the revision.
3. Thanks, we will add those comparisons.
I don't have much to add to my pre-review questions. The main thing I'd like to see that would strengthen my review further is a larger scale evaluation, more discussion of the hyperparameters, etc. Where test error are reported for snapshot ensembles it would be useful to report statistics about the performance of individual ensemble members for comparison (mean and standard deviation, maybe best single member's error rate).
Thanks for the suggestions. We will include statistics about the individual ensemble members, and evaluate our method on a larger data set, e.g., ImageNet.
This work develops a method to quickly produce an ensemble of deep networks that outperform a single network trained for an equivalent amount of time. The basis of this approach is to use a cyclic learning rate to quickly settle the model into a local minima and saving a model snapshot at this time before quickly raising the learning rate to escape towards a different minima's well of attraction. The resulting snapshots can be collected throughout a single training run and achieve reasonable performance compared to baselines and have some of the gains of traditional ensembles (at a much lower cost).
This paper is well written, has clear and informative figures/tables, and provides convincing results across a broad range of models and datasets. I especially liked the analysis in Section 4.4.  The publicly available code to ensure reproducibility is also greatly appreciated.
I would like to see more discussion of the accuracy and variability of each snapshot and further comparison with true ensembles.
Preliminary rating:
This is an interesting work with convincing experiments and clear writing.
Minor note:
Why is the axis for lambda from -1 to 2 in Figure 5 where lambda is naturally between 0 and 1.
Thanks for the positive comments. We will provide more detailed analysis about the individual model snapshots, and more comparisons with true ensembles in the final version.
We agree that for lambda the most interesting values are between 0 and 1. We will update the figures in the final version.

Meta-review:
role: PC
data: comment: Significant problem, interesting and simple solution, broad evaluation, authors highly responsive in incorporating feedback, all reviewers recommend acceptance. I agree.

Decision: Accept (Poster)