Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper carefully observes the behavior of weight magnitudes during training, finding the is a stage of saturation that is closely related to the winning lottery tickets drawing. Based on this observation the authors hypothesize that we can draw lottery tickets early but too early pruning can irreversibly hurt the learning capability for complex pattern. To remedy this and draw the tickets as soon as possible, the authors propose to adopt gradual pruning, which 1) can start early without hurting the learning capability too much; 2) avoid computation-heavy iterative pruning in previous works.

Questions:

1. Overall I am very happy with the interesting observations and analysis of the dynamics of weight magnitudes and how it can be related to the early winning lottery tickets drawing. But how valuable is it for practical use? In practice, we cannot know in advance when to start (gradual) pruning.

2. In Fig.1, what does it mean if we perform weight-magnitude based pruning at 10th epoch but rewind the weight to the 20th epoch? Is there a baseline network that is normally trained straight to the end and to which we rewind all pruned models?

3. I am not quite convinced by the experiment of Fig. 4 and argument at the bottom half of page 5. I buy the intuition that pruning too early might irreversibly hurt the capability of learning complex pattern. But I have trouble understanding how the experiment of Fig. 4 supports this intuition. The curve of retraining with smaller LR (0.01) has the save trend as the baseline and retraining with larger LR (0.1). Retraining only one epoch can hardly convince me of its relationship with learning capability. Also, for the experiment in Fig. 5, to validate the proposed hypothesis, it's more valuable to provide results around the claimed turning point, i.e. around 100 epoch instead of suddenly jumping from 20 epoch to 120 epoch.

4. In Tab. 2, the results of ResNet56 with gradual pruning is not presented. In Tab. 3, the results of ResNet50 with one-shot pruning is not presented. It would be better to have these results for clear comparison.

Overall, I love the empirical observation of weight magnitudes and think it would help the community to understand lottery tickets and training process of deep models.

Update:
The response from the authors addressed some of my questions and more experiments were added per my suggestions. However, also considering the authors' response to R#1 and R#3, I don't think it's strong enough for me to raise my score. Therefore I will keep my current score.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: Overview:

The paper is dedicated to conducting an in-depth investigation of the structure of winning lottery tickets. The author provides supporting evidence for the structure of the early winning tickets: 1) lottery tickets emerge when the weight magnitude of a model saturates with SGD optimization. 2) pruning before model saturation may result in accuracy degradation. In the experiment part, they employ the memorization capacity analysis and discover the early wining tickets without expensive iterative pruning. The author also conducts extensive experiments with various ResNet architectures on both CIFAR 10 and ImageNet, achieving state-of-the-art results with only 1/5 of the total epochs for iterative pruning.

Strength Bullets:

1. The experiment organization is complete and convincing. Especially for the figure, it not only clearly shows that lottery tickets emerge much earlier before full training ends, but also shows the effect of rewinding. 
2. The author not reveal interesting observations, but also provides useful guidance. It is a complete logic chain that is also aligned with my intuition. For example, first, the author discusses the memorization capacity of different pruned models at different epochs. Then, they introduce a reasonable gradual pruning technique. Finally, they conduct experiments to confirm it.
3.  The early winning tickets in this paper achieve state-of-the-art results with only 1/5 of the total epochs for iterative pruning.

Weakness Bullets:

1. For lottery tickets, especially for early winning tickets, I think there is a lot of randomnesses. Thus, for the plot like figure 2, figure 5, they need to contain an error bar and the curve should be the average of tens of experiments. It will be more convincing if it decouples the randomness from the real patterns.
2. The description and organization of section 4 need to be more clear. For example, an algorithm pseudo code will definitely give readers a much more clear understanding of the early winning tickets finding strategy.



Question the authors don't answer which confuses me more:
Need more convincing analysis about the indicator - Hamming Distance 

Just as the comment I posted after Review 1, we would like to see more analysis about Hamming Distance between different winning tickets. The author mentioned in the following way:

''However, as we demonstrate in Fig. 3 of our paper, the mask-distance does not well characterize the winning tickets. E.g., the lottery tickets drawn at Epoch 120 and 200 have a mask distance of 0.082 in Fig. 3, which is much larger than the mask distance between Epoch 180 and 200. Whereas, all three tickets achieve comparably high accuracy as shown in Fig. 1, implying a shallow correlation between the accuracy and the mask distance''

As far as I know, this observation only tells us that the structure of the lottery tickets changes, which are drawn from 120 epochs to 180 epochs (although the maintain a similar accuracy). However, we can not conclude that the mask distance is not a reliable measure. As mention by the [EB] paper provided by the authors, you can use the rate of the distance change to indicate the early winning tickets. In this way, we can find winning tickets much earlier than authors' work.

To better address this question, I suppose the authors need to provide more analysis of the mask distance indicator. I think all three reviewers would like to see the results. A good indicator for early winning tickets is very important, otherwise authors' notion of early still involves quite a lot of training.

Recommendation:

Due to the unsolved important question, here is a weak reject.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper attempts an in depth study of the lottery ticket hypothesis. The lottery ticket hypothesis holds that sparse sub-networks exist inside dense large models and that the sparse sub-networks achieve at least as good an accuracy as the underlying large model. These sub-networks are discovered by training and iteratively pruning the dense model. This paper investigates the epoch at which pruning should occur as well as the epoch at which weights should be rewound when retraining. Then, the authors conduct experiments with different pruning strategies (one-shot vs. gradual) in an attempt to find such sparse models (or "winning tickets") earlier than they otherwise would have been found.

The experiments conducted by the authors seem to be very extensive, and I think the paper contains useful data to have for researchers interested in better understanding the lottery ticket hypothesis. However, my main issue is with both the originality and significance of this work. This paper gives evidence that winning tickets may be found "early," although their notion of early still involves quite a lot of training. 

Although the paper is interested in addressing the structure of the winning tickets, I really didn't find any of the discussion of structure to give much insight into the lottery ticket hypothesis. Most of the section focuses on analyzing weight magnitude, though I was hoping for something more about the actual structure of the sparse subnetwork -- especially given the title of the paper. Figure 3 is notable, showing that different winning tickets (parameterized by different prune and rewind epochs) can have a large Hamming distance between them. This is very interesting, and I wish the authors had more to say. How is this affected by different initializations? Are these solutions connected on a loss landscape? Is there something invariant about the sparse architecture after symmetries are taken into account? It's not clear to me that Hamming distance alone is enough.

In conclusion, the paper presents a set of nice experiments, but doesn't really shed too much additional light on the scientific nature of the lottery ticket hypothesis.
review_title: Official Blind Review #1

Meta-review: This paper does extensive experiments to understand the lottery ticket hypothesis. The lottery ticket hypothesis is that there exist sparse sub-networks inside dense large models that achieve as good accuracy as the original model. The reviewers have issues with the novelty and significance of these experiments. They felt that it didn't shed new scientific light. They felt that epochs needed to do early detection was still expensive. I recommend doing further studies and submitting it to another venue.
Decision: Reject