Venue: ICLR 2017 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: I don't have much to add to my pre-review questions. The main thing I'd like to see that would strengthen my review further is a larger scale evaluation, more discussion of the hyperparameters, etc. Where test error are reported for snapshot ensembles it would be useful to report statistics about the performance of individual ensemble members for comparison (mean and standard deviation, maybe best single member's error rate).
review_title: A good paper.

final_score: rating: null
review_content: .
review_title: .

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The work presented in this paper proposes a method to get an ensemble of neural networks at no extra training cost (i.e., at the cost of training a single network), by saving snapshots of the network during training. Network is trained using a cyclic (cosine) learning rate schedule; the snapshots are obtained when the learning rate is at the lowest points of the cycles. Using these snapshot ensembles, they show gains in performance over a single network on the image classification task on a variety of datasets.


Positives:

1. The work should be easy to adopt and re-produce, given the simple techinque and the experimental details in the paper.
2. Well written paper, with clear description of the method and thorough experiments.


Suggestions for improvement / other comments:

1. While it is fair to compare against other techniques assuming a fixed computational budget, for a clear perspective, thorough comaprisons with "true ensembles" (i.e., ensembles of networks trained independently) should be provided.
Specificially, Table 4 should be augmented with results from "true ensembles".

2. Comparison with true ensembles is only provided for DenseNet-40 on CIFAR100 in Figure 4. The proposed snapshot ensemble achieves approximately 66% of the improvement of "true ensemble" over the single baseline model. This is not reflected accurately in the authors' claim in the abstract: "[snapshot ensembles] **almost match[es]** the results of far more expensive independently trained [true ensembles]."

3. As mentioned before: to understand the diversity of snapshot ensembles, it would help to the diversity against different ensembling technique, e.g. (1) "true ensembles", (2) ensembles from dropout as described by Gal et. al, 2016 (Dropout as a Bayesian Approximation).
review_title: Review

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: This work develops a method to quickly produce an ensemble of deep networks that outperform a single network trained for an equivalent amount of time. The basis of this approach is to use a cyclic learning rate to quickly settle the model into a local minima and saving a model snapshot at this time before quickly raising the learning rate to escape towards a different minima's well of attraction. The resulting snapshots can be collected throughout a single training run and achieve reasonable performance compared to baselines and have some of the gains of traditional ensembles (at a much lower cost). 

This paper is well written, has clear and informative figures/tables, and provides convincing results across a broad range of models and datasets. I especially liked the analysis in Section 4.4.  The publicly available code to ensure reproducibility is also greatly appreciated.

I would like to see more discussion of the accuracy and variability of each snapshot and further comparison with true ensembles.

Preliminary rating:
This is an interesting work with convincing experiments and clear writing. 

Minor note:
Why is the axis for lambda from -1 to 2 in Figure 5 where lambda is naturally between 0 and 1.
review_title: Review

final_score: rating: null
review_content: 1. In Figure 6, the diversity of activations is studied with respect to different learning rate schedules. It would be interesting to see how this diversity, which is key for ensembles, compares with a different ensembling technique, e.g. (1) "true ensembles", (2) ensembles from dropout as described by Gal et. al, 2016 (Dropout as a Bayesian Approximation).

2. In Table 2 (pg. 7), is the evaluation done using just the final snapshot or an ensemble? If using an ensemble, please specify the size of the ensemble (m) for clarity.

review_title: Activation Diversity and a clarification

final_score: rating: null
review_content: - A trivial way to obtain equivalent local minima is to permute the hidden units. Have you thought about ways to discourage this? (I'd be interested to what degree this happens, which might be measured by comparing activations in sorted order or something like that.)
- It seems on first glance difficult to evaluate hyperparameter choices when you're intentionally allowing the objective function to grow. Can you comment on the feasibility of hyperparameter search in this model class? One particular (perhaps obvious) improvement that as far as I can tell isn't mentioned is that, at each convergence, the new member's validation set predictions can be saved and a running sum/average maintained, enabling cheap computation and monitoring of the *ensemble* validation error.
- Is it necessary to enable re-training of the entire model with the cycled learning rates or would a few of the top layers suffice?

A small comment: please consider that your figures may be read in grayscale (or by the colorblind) and try to make them legible without relying on color.

review_title: Symmetry & hyperparameter selection

Meta-review: Significant problem, interesting and simple solution, broad evaluation, authors highly responsive in incorporating feedback, all reviewers recommend acceptance. I agree.
Decision: Accept (Poster)