Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: The paper empirically analyzed the wide existence of "early-bird tickets", e.g., the "lottery tickets" emerging and stabilizing in very early training stage. The potential connection to (Achille et al., 2019; Li et al.,2019) reads interesting. 

The authors made several contributions in addition to the observation: (1) the EB tickets stay robust under large learning rates (while early stopping) and low-precision training; (2) the EB tickets can be detected using epoch-wise consecutive comparison (mask distance), rather than comparing with some oracle ticket; (3) the application of EB ticket towards energy efficient training, which is interesting as this is perhaps the first practical application demonstrated of lottery ticket. 

While I like how the paper connects theory hypothesis to real applications, the experiments need to be solidified in a few aspects:

1) Figures 1 and 2, why a few plunges of curves (say Fig 2.a, p = 70%)? Does it imply the training might not be stable?

2) Table 1, the authors test two lr schedules to show "large learning rates favor the emergence of EB Tickets". Yet the choice of lr matters a lot and can be tricky. Why the authors pick the two specific learning rate schedules? Why are they "comparable"? What if being more aggressive in choosing larger lr, say starting from lr= 0.5?

3) The low precision EB ticket is not actually applied or evaluated in Section 4. It would have been interesting to see.

4) I fail to find the 4.7 times energy saving as claimed in abstract from Table 2?

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The authors further study the lottery ticket hypothesis formulated by Frankle and Carbin. They demonstrate that the sparsity pattern corresponding to a lottery ticket for a given initialization can be uncovered via low-cost training. By doing so, they propose a method to: 1) first identify the lottery ticket efficiently and 2) exploit the sparsity of the resulting network to train it at a lower cost.

The contribution is however a bit incremental in my opinion. The original LT paper was not focused on efficiency, and it is not a far stretch to try to find the tickets sooner during the training. On the other hand, the experiments are well conducted (especially 4.3) and even if the original idea is simple, it is of interest to see it tested as clearly. All in all, I found this paper convincing and worth reading, and I think it should be accepted.

Positive points:
- The literature review is sufficient and present with great clarity the latest results.
- The problem tackled is of great interest and has potentially impactful applications.
- The authors focus on hardware friendly types of pruning.
- The paper is well written and enjoyable.
- The algorithm used to compute the EB tickets seems a bit ad hoc, but in my opinion sufficient as a first approach.


Nitpicking:
- Not sure why Max(Q) > eps is in the while condition (return if Max(Q) < eps should be sufficient)
- The treatment of the mask distance in figure 3 is confusing. It is not obvious why the authors are plotting 1-distance, and the legend of the figure suggest that a mask has a distance of 1 with itself. Recommend plotting d instead of 1-d and invert the color bar instead if they feel so inclined (yellow=0).
- Abstract: “consistently exist across models and datasets” a bit of a strong claim as only cifar is used.

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper presents a method to speed up training of deep neural networks. The main contribution is a method to quickly identify winning lottery tickets (denoted early-bird, or EB by the authors), without running the model to convergence. The authors present interesting preliminary experiments that motivate their method, and show that it works on two image recognition datasets using two models.

This paper addresses an under-explored, but very important problem in AI: the increasing cost of training models. The authors present interesting evidence about the potential to detect EBs early on. The experiments presented in Figures 1 and 3 are convincing and will be of interest to the community. The proposed method seems to work, at least on the setups explored by the authors. I am leaning towards acceptance, but am concerned with the following: 

1. The authors experiment with a limited set of datasets (CIFAR-10 is a relatively easy task), and with a set of non-competitive baselines (SOTA for CIFAR-10/100 is 99%/91.3%, see https://benchmarks.ai/cifar-10{,0}). I would have liked to see whether the proposed method translates to harder datasets and stronger models.

2. I might be missing something here, but to the best of my understanding the large learning rate part (page 4) does not demonstrate the benefits of increasing the learning rate, but the problems with *decreasing* it. The two might seem like the same thing, but in fact they're not: the authors claim the [80,120] policy is standard, and use it when training the subnetwork, so showing that [0,100] is inferior does not present a way to improve over the current approach, but evidence that the other approaches are inferior.

Other questions: 
1. In Figure 1, it seems that the extracted subnetworks are doing very well even after 0 epochs. Does this mean that a trained version of a random subnetwork could reach within 1-2 points of the unpruned model? or is it pruned after training for 1 epoch?

2. If I understand correctly, Figure 5 should be illustrating the proposed method, which automatically identifies the early stopping point. In that case, I am not sure why the plot is a function of the epoch.

3. Do the authors have any intuition as to the sharp decrease in the 70% graph in Figures 1 and 2 around epoch 50?

Writing: 

1. The language used by the authors is sometimes exaggerated. Expressions such as "bold guess" (section 3.2), "innovative ... scheme" (section 4) and comparisons to Winston Churchill would be better left out of the manuscript. 

2. Typos and such: 
-- several across the paper. For instance: 
- Intro: After *bring* identified (should be "being")
- Related work: when training *it* isolation (in)

-- Missing venue for Frankle and Corbin (2019)

review_title: Official Blind Review #2

Meta-review: This work studies small but critical subnetworks, called winning tickets, that have very similar performance to an entire network, even with much less training. They show how to identify these early in the training of the entire network, saving computation and time in identifying them and then overall for the prediction task as a whole. 

The reviewers agree this paper is well-presented and of general interest to the community. Therefore, we recommend that the paper be accepted.
Decision: Accept (Spotlight)