Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper goes deep into the investigation of the lottery ticket hypothesis, and found out that the training recipe that the original findings were based on had been insufficient. With a revised recipe and through extensive experiments, the author empirically studied the patterns of whether and when a "jackpot ticket" and "secondary tickets" can be identified. The major factor turns out to be the learning rate with which to train the subnetworks. The paper then ends with guidelines on how to find winning tickets.
main_review: The paper challenges the established LTH, reproduced and improved the original training recipe, and comes up with a revised definition, and recommended guidelines on obtaining winning tickets based on an extensive set of empirical study. Overall I think such a sanity check endeavor is of great value to the community. 

limitations_and_societal_impact: The author claims that they found a better training recipe than used in original LTH. And Figure 1 is the only figure that showed such superiority. However it is on three hand-picked sparsity values. One plot I feel like is missing and could really drive this point home is one that plots accuracy vs. sparsity like in the original LTH and many follow-up works. So instead of picking three sparsity levels, the three subplots would be the three LRs, and each subplot would have x axis being the sparsity level or "percentage of weight remaining" like in all the other LTH papers.

Another limitation that invites questions is the ImageNet result. According to paper, "Tiny-ImageNet and ImageNet-1K, no clear secondary prize tickets are identified using ResNet-18 or ResNet-50." That is not quite believable. One possibility is that the author adopted the original rewinding recipe used for CIFAR-10, where the weights are rewound to initial values. However a follow-up work on LTH on larger models [1] demonstrated that it is necessary to rewind to a later point during training for IMP to succeed. I suggest the author study this paper and many other papers on finding winning tickets large-scale models and datasets, which almost all proved the existence of winning tickets in ImageNet and ResNets.

[1] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Stabilizing the
lottery ticket hypothesis. arXiv preprint arXiv:1903.01611, 2019.

Errors, modifications or improvements needed:
 - In 2.1 when defining a network function f (·), I don't think input x should be included, that is, it should just be f (\theta) as opposed to f(x, \theta). Both because the function shouldn't depend on input, but also throughout the section x is always the same and hence repetitive. 
 - Figure 2 the difference between (b) and (c) is unclear. According to the definition, the highlighted difference between "jackpot" and "secondary" should be whether the original network is well trained (with appropriate LR and length of training) or not. The plots though seem to only highlight whether the crossing point is at a lower or higher LR. Therefore it doesn't seem consistent with the definition. 

Minor:
Each figure and table should be self-sufficient, in that the caption should convey what the reader is supposed to get out of. Sometimes that requires repeating what's said in the main text but it should be very much encouraged. For example Figure 6 is confusing to me as what observation or conclusion we are supposed to draw.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper critically evaluates the lottery ticket hypothesis under different hyper parameter settings. They note that the learning rate used has a substantial impact on the quality of the unpruned model as well as the lottery ticket networks. They propose a stronger form of LTH by requiring the lottery ticket subnetworks to be able to match the best performing unpruned network. They find that using a larger learning rate results in a better unpruned model, but also results in an inability to find equally performant pruned networks.

They show that IMP is better than OMP only for residual networks, and residual networks prefer a smaller learning rate on small datasets, and when larger learning rates are preferred, random reinitialization performs no worse than lottery tickets.
main_review: Originality: 
- There are some novel observations, including a systematic look at the impact of learning rate on LTH, comparing iterative magnitude pruning (IMP) and one-shot pruning (OMP), as well as the investigation into the loss contours for residual connections. However, the overall originality of the contribution is limited. Some results already exist in prior work. For example:
    - The definition of secondary prize tickets seem very similar to what is used in prior work, and observations regarding them have already been well established in those papers. The failure to train LT from initialization has also been studied in [2].
    - It's well known that initial weights and final weights are correlated, and that LT train to the same basin as the original network, which makes the analysis regarding correlation indicator unsurprising.

Quality:
- The experimental settings in this paper are comprehensive, which I highly appreciate. 

- The stricter definition proposed in this paper requires LTs to be found at the best hyper parameter settings for the unpruned network. However, it doesn’t consider the possibility that training the subnetworks would require different optimal hyper parameters. In Lan et al [1], they show that it is optimal to use different learning rates during pre-training (for finding the mask) and during LT training (for training the sparse model), and that doing so leads to well-performing lottery tickets. Though the authors cannot be faulted for not citing non-conference papers, the observations in Lan et al suggest that different learning rates between pruned and unpruned models should also be considered in this work. Intuitively, if a jackpot lottery ticket satisfying the specified conditions can be found by using different learning rates, it should be considered as a successful lottery ticket. Without running those experiments, it raises doubt regarding the soundness of this paper’s main message.

- In section “Which pruning method is better, IMP, OMP , or it does not matter?”, how are the IMP/OMP trained? Is there an unpruned model trained without residual connections, and new lottery ticket masks are derived from that model? If so, how does the loss contours compare for the unpruned models with or without residual connections? I wonder if the smoothing effect is inherent to ResNets, and IMP simply preserves it, or is it specific to sparse models? Do you see the same non-smooth behavior in other non-ResNet model architectures (both unpruned and pruned)? Do these other model architectures prefer larger learning rates (unlike ResNets)?

- No justification is provided for the thresholds of 0.5%, 1%, and 1.5%, other than prior work - please cite the relevant prior work there. 

- Missing citations for Stabilizing the Lottery Ticket Hypothesis [2], as many similar experiments have been conducted there. Any differences in results should be noted.

Clarity:
- The paper is overall well written. 
- Regarding secondary prize ticket, is it identical to the definition proposed in the original LTH? 
- In the section "When does theta_0 benefit subnetwork training?" Could the same conclusions be drawn by simply looking at the correlation of the weights between theta_0 and theta_T?

Significance:
- I think the position of this paper is significant and an important contribution to lottery ticket work. I believe that the stricter form of lottery tickets is necessary in future work and can help remove inconsistencies in this research direction. However, this is contingent on resolving the issues I raised above, especially regarding the hyperparameter optimization performed for lottery ticket networks.

[1] https://openreview.net/forum?id=XkI_ggnfLZ4

[2] https://arxiv.org/pdf/1903.01611v3.pdf

===================================================
Update post rebuttal:

I want to thank the authors for the very thorough response, which helped to address my original concerns. I hope the authors can incorporate many of these discussion points in the final manuscript. On the original reading it seemed like the conclusion was that no jackpot tickets can be found, but the rebuttal provides a more nuanced stance, which I hope can be better reflected in the final paper. I also would encourage the authors to provide a more thorough study of decoupled learning rates (as the authors already indicated that they will do), as it is an important factor in determining final performance and should really be explored in a paper that aims to provide general guidelines for LT training.

Assuming these changes will be incorporated in the final paper, I will raise my score to 7.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper tries to reconcile the controversies around lottery ticket hypothesis. Their study reveals that small learning rate, insufficient training epochs etc. are the main reasons for these controversies. They define a rigorous version of LTH and show that under different training recipes, network architectures and train data size, no rigorous winning tickets are found by the current methods. Their study highlights that using small learning rate on DNNs with residual connections is more likely to find winning ticket and in conditions where small learning rate is not favourable then any special initialisation has no impact on chances of finding a winning ticket. 
main_review: The paper studies a very important problem and tries to find answers to some of the inconsistencies around LTH. The findings are novel, very insightful and valuable to the LTH and pruning research. Finding a winning ticket is an expensive process and guidance provided by this paper on several hyper-parameters is quite useful. Most part of the paper is well written and has sufficient clarity. I feel the definition of rigorous LTH can be made more clear, possibly using some mathematical constructs. The experiments are done over different datasets( CIFAR-10/100, Tiny-Imagenet) and different architectures (ResNet, VGG, MBNet etc.), which makes their findings more credible. I think the claims provide important hints about LTH however, since the claims are only based on experimental results, so we need to be open to situations where some of their claims may not hold as well. 
limitations_and_societal_impact: I think the definition of rigorous LTH can be made more clear. In addition to the empirical explanations, it would be great if these claims can be explained using theory.
review_title: null

Meta-review: This paper investigates the impact of hyperparameters on lottery ticket performance and uses these analyses to provide guidance as to how to best select these hyperparameters. All 3 reviewers praised the paper for its comprehensive and compelling experiments as well as its clarity. There was some concern regarding the stringent definitions used in this work, but clarifications in the discussion were sufficient to resolve these concerns so long as the authors add additional discussion to the manuscript. Given the rigor of this work and its relevance to the lottery ticket community, I recommend this paper is accepted.
Decision: Accept (Poster)