Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: *Summary*
Extending the observations of Frankle et al. (2019, "The Lottery Ticket Hypothesis"), this paper examines "rewinding" as an alternative to fine-tuning in a typical network pruning process. After training to convergence for T iterations, the k% of weights with the smallest magnitude are pruned (set to zero). Typically, in fine-tuning, the remaining weights are continued to be trained for several more iterations at a small learning rate. With "rewinding", the remaining weights are reset to their values in iteration 0<=t<T and are trained for T-t iterations with the original learning rate schedule.

Experiments with this method (and an iterative variant) show that (a) rewinding achieves 2-5x smaller networks matching the unpruned accuracy (with comprehensive tuning of the rewinding parameter t); (b) selection of rewinding iteration is important, but is flexible within a large range for higher sparsity pruning.

*Rating*
The paper is very well written with good exposition, thorough notes and citations for all methodological choices,
and an explicit statement of the limitations of the work.

A few considerations relevant to the rating:
(1) Novelty: The idea for rewinding is not novel, as acknowledged clearly in the paper. Frankle et al. (2019, "Stabilizing the Lottery Ticket Hypothesis") showed that rewinding to an early iteration of training (0<t<T) yielded better accuracy than rewinding to iteration t=0 for VGG-19 and ResNet-18 on CIFAR-10. However, Frankle et al. did not consider fine-tuning as it was not relevant to lottery ticket discovery. This submission studies the tradeoffs of rewinding vs. fine-tuning.

(2) Thoroughness: The paper considers ResNet-20 and VGG-16 with CIFAR-10 and ResNet-50 with ImageNet. Conclusions would be strengthened with additional combinations of networks and datasets.

(3) Acknowledged limitations: As noted, the paper doesn't consider any pruning criteria other than weight magnitude, nor does it consider structured pruning. The latter in particular is important for applications where prediction speed on commodity hardware is a limiting factor.

As it is, I think this paper a worthy (if limited) contribution to the understanding of network pruning.

*Notes*
Table 1/Figures *: note which dataset is used for each architecture
Figures 2-3: It seems that many values are clipped by the legend range of +/- 0.5%. Consider showing the figure with a larger range or adding such a figure to the appendix.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: Building on the results of [Frankle et al, 2019], this paper seeks to utilize rewinding as a core procedure in pruning neural networks, in combination with the usual fine-tuning procedures.  Specifically, [Frankle et al, 2019] demonstrate that it is possible to find sparse subnetworks, such that rewinding weights to their initial values and retraining from that initialization, yields test accuracy similar to the original network.  While this is already a form of pruning, the submitted paper explores a wider space of pruning procedures that utilize rewinding as a subroutine.

This wider framework includes the choice of rewind point (e.g. rewinding to partway through training rather than to initialization) and how to balance computation budget between rewinding (and retraining for an equivalent number of epochs) vs continuing to fine-tune a network.  Experiments cover this hyperparameter space, as well as the range of desired sparsity level (pruning amount).  Results show rewinding (to a point 30% - 60% into training) dominates any amount of fine-tuning, if moderate to high sparsity is desired.

The empirical study conducted by this paper is useful and complements the results previously reported in [Frankle et al, 2019].  However, the paper itself is light on novelty, as the core ideas were already established by [Frankle et al], and the application of them here is relatively straightforward.  The extensive experiments here add value to the conversation about the lottery ticket hypothesis, but are not otherwise ground-breaking.

review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: This paper does an in-depth evaluation of the notion of rewinding pruned networks to the weights at a previous point in training, and then re-training the pruned network from then on. This is in comparison with fine-tuning a pruned network, where the retraining continues from the network's current weights. The authors focused on vision networks and unstructured magnitude pruning in their evaluation.

The paper is well-written and easy to follow, and the authors have done a good job of empirically comparing rewinding against fine-tuning in a number of different scenarios.

My main concern with the paper is that it seems too incremental to stand on its own. Rewinding is a notion that was already explored in the Lotter Ticket Hypothesis (Frankle et al., 2019), so this paper seems to be more of an extension of that work. The take-away message from this paper (stated by authors in their conclusion) is that practicioners "should explore rewinding as an alternative to fine-tuning for neural network pruning", but that's a case that was already made by the LTH paper. The current work certainly gives more weight to that claim, but I don't feel the contributions are strong enough on their own to justify a full conference publication.

I encourage the authors to continue working on this, as it is very interesting and can be very useful. Some ideas to make the paper stronger:
- Given that this is a purely empirical paper, it'd be better to not limit the experiments as much. Can you run on non-vision networks? What about mobile-net? Can you try different pruning techniques? etc.
- How do the different methods compare in terms of accuracy/sparsity versus FLOPs?

Finally, two minor comments to improve the writing:
- First sentence of 3.1: s/that meet that the accuracy/that match the accuracy/
- First sentence of 3.1: s/than fine-tuning can/compared to fine tuning
review_title: Official Blind Review #1

Meta-review: Reviewers unanimously accepted this paper. 
Decision: Accept (Talk)