Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper provides the interpretation of IMP method in terms of the loss geometry, to better understand why and how IMP finds winning tickets. It provides various interesting empirical results. The main finding is that whether we find a matching network or not is related with whether the subnetworks at successive rounds are linearly connected in the loss landscape. 

strength_and_weaknesses: * Strengths
    * It provides extensive empirical results
    * The interpretation provided in this paper makes sense.

* Weaknesses
    * It is mostly empirical, and is not providing theoretical insight.
    * It compares with random pruning, but I'm curious the result for other pruning algorithms applied in an iterative manner. 
    * Although the result is interesting, we still don't know why simple "magnitude-based" pruning can find such flatter directions in the error landscape. 
 

clarity,_quality,_novelty_and_reproducibility: * It is well written and the claim is clear
* It is novel approach and novel results

summary_of_the_review: This is an interesting empirical work. Although we do not have sufficient theoretical explanation, it is good to share with the research community, given a proper comparison with existing iterative pruning methods.
review_title: null

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: The paper investigates iterative magnitude pruning (IMP) from the perspective of loss landscapes to answer questions such as why is it necessary to rewind the search iteratively rather than perform the pruning in one-shot fashion. The authors show that the IMP mask encodes the location of linearly connected modes (i.e. modes with low error barrier) containing matching sparse solutions. Further, the authors show that rewinding is necessary to “equalize” the weights to identify the next set of low-magnitude weights to prune. Finally, a link to the Hessian eigenvalue spectrum is established. The insights can lead to improved (faster) versions of IMP, one such version is proposed by the authors in the appendices.
strength_and_weaknesses: Strengths:
* The paper studies IMP from the loss landscape perspective, the results are insightful and open up many future research directions.
* The obtained insights have a very practical application (e.g. improving IMP).
* The paper is very well written, the experiments are realistic.

Weaknesses:
* The paper does not link the observations to the attraction basins in the loss landscape, which to me feels like an important omission.
* The appendices for the paper are extensive, and include such sections as related work, which seem better suited in the main body of the paper. In general, I think this paper would be more suited for a journal rather than a conference publication. However, I acknowledge that the ML community strongly favors top-tier conferences, and understand authors’ decision to compress their work into 9 pages. I commend the impressive effort, but honestly believe that the narrative would have benefited from a less stringent page limit.
clarity,_quality,_novelty_and_reproducibility: This is a brilliant paper, clear and novel, reviewing it is an honor and a pleasure. The results should be easily reproducible.

I do, however, have a few questions to the authors, as well as some suggested corrections. Please see below.

“flatter error landscapes allow more aggressive pruning” — isn’t this an obvious consequence of flatter landscapes having more irrelevant parameters (hence the flatness, i.e. no or little influence on the loss)? 

Page 6: “But how is SGD able to extract this information from the mask at the rewind point.” - Should end with a question mark.

The authors speak of the “robustness of SGD training to perturbations at late enough rewind steps”. Would it be valid to say that rewinding late enough does not actually cause the algorithm to leave the attraction basin? I think the attraction basin perspective is necessary here, otherwise the said robustness seems arbitrary. From the loss landscape perspective, what causes the robustness? 

“This happens because the error landscape is getting sharper at higher sparsity levels” - why is the landscape getting sharper at higher sparsity levels? And can it thus be concluded that self-regularized solutions (“lottery tickets”) would typically be found in sharp minima? The latter would be a very interesting observation, since current theory favors flat rather than sharp minima (at least in terms of the expected generalization performance).
summary_of_the_review: To conclude, I would be very sad if this paper is rejected, since it is definitely one of the best ones that I have recently reviewed or read. I think this work is potentially very impactful, and I believe that the loss landscape perspective provides an excellent lense to deeper our understanding of neural networks training and behavior.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper investigates why iterative magnitude pruning (IMP) is successful.  The authors hypothesize that the masks discovered during iterations of IMP encode subspaces that intersect the linearly connected loss sublevel sets associated with the previous iterate.  They present multiple experiments to support this hypothesis.
strength_and_weaknesses: **Strengths:** 

The paper investigates the relevant problem of identifying the conditions for the success of IMP.  The hypothesis is appealing and is consistent with their data.  The figures are clear and readable.

**Weaknesses:**

This paper lacks an adequate discussion of related work in the main body.  While they have a related work section in the Appendix, it is standard to have this section in the main body.  It is important for readers of the work to have these comparisons made clear.  Also note that reviewers for ICLR are not required to read the supplementary material for better or for worse, which makes it even more important for this section be included in the main body.

The paper makes many key claims (e.g. the six bullet points spanning the end of page 3 and the start of page 4).  While there are data to support each of these claims, these are big proclamations and some could be a separate paper on their own.  For example when investigating robustness of SGD you would need to take many perturbations and then train to really see if all the solutions are linearly connected due to the high dimensionality of the space.  I think the paper could potentially be improved if it slightly reduced its scope and focused the experiments on its most important claims.

clarity,_quality,_novelty_and_reproducibility: **Quality**

Aside from my concerns in the "Strengths and Weaknesses" about the amount of experiments relative to the number of claims made in the paper, I believe the experiments are of high quality.

**Clarity**

The work is clear and is well written.

**Originality**

As far as I can tell the work is original and places itself well among the literature.

**Comments/Additional feedback:**

On page 4 when referencing Figure 2 the “red” curve as you describe it appears to be more orange colored in the figure.  Perhaps you could recolor the figure or change the description.

Second paragraph of the discussion in Section 4 references multiple results in the Appendix, e.g.
“we show in Fig. 10 that it is possible to achieve the same performance as IMP but with fewer levels of pruning by dynamically choosing the per-level pruning ratio so as to stay within the LCS-set after projection”.  It may be more clear to the reader to emphasize that these results appear in the Appendix but not the main body.
summary_of_the_review: I would be inclined to accept this paper after revisions are made to properly address the literature in the main body of the paper and to make proper comparisons to the present work.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper shed some light on the underlying principles that allow IMP to find winning tickets. In particular, the authors focus on four fundamental questions regarding the information provided by the mask, the need of an iterative procedure for pruning, and the differences between retraining, learning rate rewinding and finetuning.
strength_and_weaknesses: I think that this paper is very interesting because it tries to answer to some fundamental questions related to the lottery ticket hypothesis, providing a significant contribution to the research in this field. The analysis presented in this paper is based on solid theoretical assumptions and validated by extensive empirical experiments. 
clarity,_quality,_novelty_and_reproducibility: The paper is well written and the experiments presented to validate the authors' claims are well described. As said in the previous section, I think that it can be a significant contribution to the research in this field.
summary_of_the_review: I think that this is a solid paper and it provides a significant contribution to explain the underlying principles of the lottery ticket hypothesis.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper studies lotter tickets, and connects them to many other aspects of the training process, such as SGD and flat minima.  Reviewers have a few misgivings, but may were corrected during the discussion phase, and I am glad to mark this as a clear accept.
justification_for_why_not_higher_score: Reviewers identified a few weaknesses, and the bar for oral is extremely high. 
justification_for_why_not_lower_score: Most of the weaknesses identified by reviewers were fixed in the revision phase; overall an impressive effort by the authors!
Decision: Accept: notable-top-25%