Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper studies a fundamental and important research approach in network pruning: iterative magnitude pruning (IMP). Previously IMP is criticized to be time-consuming, layer-independent and sub-optimal in performance. In this paper, extensive empirical studies are conducted to show that under proper learning rates, IMP can have close performance with more advanced pruning approaches, with little training time increased. 
main_review: 
Strength:

1. The paper revisits IMP, a basic yet important pruning approach. The empirical findings are promising: IMP can be trained on par with stable pruning approaches using reasonable training time under proper learning rate schemes.

2. Extensive studies with detailed experimental setups are provided. Code is provided for reproducibility.


Weakness:

1. It is not explained why SLR is combined. I wonder if IMP can be improved when combined with other training tricks (e.g., CLR) w.r.t. the discussed aspects. And will there be any improvement if other baselines (e.g., Uniform/ERK/LAMP in Figure 2) are combined with SLR?

2. The writing is not very focused, making it kind of hard to follow by the general audience, especially for Section 2. More technical details can be introduced for IMP and SLR, both of which are studied throughout the experiments.

3. While the paper provides empirical studies for IMP, the technical contribution can be minor, as most components are existing methodologies.


Detailed comments:

- The title can be misleading? The empirical results show that only under proper learning rate schemes (SLR) can IMP achieve better performance with less training time. However, SLR is not a basic technique.

- While the authors draw the conclusions based on the improved accuracy with SLR, the reasons for improvement are still unclear nor well explained. It would be more inspiring to explicitly study why previous IMP are sub-optimal, and how SLR can improve IMP.

- Are the rest baselines in Figure 1-3 trained with the same learning rate scheme (FT or SLR)? Otherwise, it can be unfair to compare with IMP+SLR.

- Reference formats are not formal and inconsistent.

- It can be restrictive to study unstructured pruning as well as theoretical speed-up with FLOPs reduction in practice.
summary_of_the_review: The empirical findings are promising. However, it is still not clear in what way SLR can improve IMP, and the reasons behind the improved accuracy / reduced training time.


=== Post rebuttal ===
Thanks for the authors' response and paper revision, which addresses some of my concerns. While the paper introduces inspiring findings on how SLR (or CLR) help IMP, most components are from existing techniques. I choose to keep my score, and encourage the authors to provide more in-depth analysis behind the improvement by SLR, via either new methodologies or perspectives.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper investigated several recently proposed pruning stable approaches and compared them with the basic iterative magnitude based pruning (IMP), and observe that IMP actually performs on-par or even better in the experiments. The authors investigated the retraining approaches, the computation cost, the importance of sparsity allocation (including layer-collapse) and compared IMP (+retraining) to different pruning methods.
main_review: This paper is motivated by a simple question: whether the basic IMP method is enough for weights pruning? Through the whole paper, the authors conducted lots of experiments to show that IMP with a good retraining is as good as many recently proposed pruning methods. The authors also show that using IMP we can also directly determine the layerwise sparsity through global ranking, instead of using more complex method to determine the sparsity for each layer.

Although the authors did lots of experiments validating the claim, the experiments are kind of limited. All the experiments are in image classification, with larger size models. I am wondering how is the result on compact models like mobilenet. Besides CNNs, weights pruning are actually used for other architectures too, e.g., LSTMs and transformers. How are the comparison on those cases? I admit that this paper already conducted a comprehensive comparison, while it's hard to say IMP is enough for pruning with only a few models on Image classifications showing that. In addition, many new pruning algorithms are actually built for NLP and speech tasks.
summary_of_the_review: I like the author's view and the motivation to make a realizable and unified baseline for weights pruning. However, I think it's better to have more evidence on different models and tasks to validate this conclusion. If the proposed claim can be further validated, I think it will be a significant contribution to the community.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work focuses on highlighting the strengths of Iterative Magnitude Pruning (IMP). Specifically, that it is capable of achieving strong performance when compared to more complex pruning approaches. The work explores the common arguments against IMP like, a) it reaches sub-optimal states since training doesn't compensate for sparse structures, b) it fails to identify optimal layer-wise pruning ratios and c) it is expensive, slow and non-competitive. The critical outcome shown is that IMP, with a global selection criterion and extremely small overhead, remains highly competitive with common state-of-the-art pruning approaches, both in sparsity, performance and theoretical speedup. 
main_review: Strengths
- The clarity in writing throughout most of the paper helps the reader quickly assimilate the contents and understand the intent of the paper.
- The level of detail, across the experiments as well as the figures, provided is of high quality and much appreciated.
- Explanation of the different approaches to pruning, their background and clearly advocating for the missing link is commendable.

Weaknesses
- The first paragraph on the arguments against IMP (Pg. 2, bullet point 1), reads slightly incoherently, like a mix of multiple ideas. A more clear and point by point description of sub-optimal states, re-training epochs and others would be extremely helpful.
- Additionally, "Many proposed improvements..." is followed by a single citation in the same bullet point 1. Adding multiple citations would be more appropriate with that statement.
- Pg. 4 Paragraph of Pruning Approaches: The nomenclature of "pruning approaches" is slightly counterintuitive when discussing ideas related to how pruning methods limit the overall compression of different layers. An alternative heading would summarize the contents more appropriately.
- While the intention of Section 3.1 is to highlight the low computational overhead of IMP in matching baseline performance, the intermediate outcome of SLR being better than FT traces the takeaways from Renda et. al (2020) and slightly weakens the contribution. Further, it isn't exactly clear what is the overhead when comparing the final set of results for IMP vs. SOTA pruning approaches. Comparison to a potentially weak baseline isn't quite enough and I encourage the authors to provide those results in a bid to strengthen their argument.
- Some of the key takeaways from Section 3.2 are very similar to that of Tanak et al. (2020) and Lee et al.(2020). Specifically, that of layer-collapse and the SLR vs.FT performance comparison. Leaning on these outcomes, in text and across image captions, weakens the novelty of the content. I encourage the authors to emphasize on IMP-specific behavior and insights. 
- In Section 3.3, the last paragraph, the statement of IMP can be applied to already trained models while other methods in the experiment cannot isn't fair since the experiment uses pruning-during training approaches and not the general set of pruning methods.

After Rebuttal

I would like to thank the authors for their timely and pertinent discussions and revisions to the manuscript. 
As I described previously the consistent references and experimental structure borrowed from existing work hinder the novelty of the work. 
The authors have consistently clarified the difference in setup and mentioned that the exact outcomes differ from already existing work. 
While the implementation and proposed work are focused on resurrecting IMP as a valid SOTA baseline, more in-depth work in alternative settings (unseen in existing literature) would be one possible way to help further highlight the novelty aspect of IMP. 

summary_of_the_review: The proposed work does an extremely good job of explaining the landscape of pruning and setting up the missing link of IMP not being explored sufficiently. However, the experimental takeaways follow similar patterns to Tanaka et al.(2020) and Renda et al.(2020), which weaken the novelty of the work. The work sounds more like an extension of IMP into existing experimental frameworks and results than a purely novel instance.
review_title: null

Meta-review: The paper provides an analysis of the well known method of Iterative Magnitude Pruning (IMP) for DNN compression. The problem tackled is undoubtably an important one, and IMP is likely one of the most known solutions for DNN compression. As such, there is no doubt that the paper is well motivated. In addition to the motivated task, the reviews indicate that the paper is well written and provides a thorough review of the related literature, making the paper easy to read and follow.
The main weakness of the paper seems to its novelty, as it seems that similar analyses have been done in the past. This issue was raised by the reviews and remained after the correspondence with the authors:

 WMeJ: “As I described previously the consistent references and experimental structure borrowed from existing work hinder the novelty of the work”, dL1d: “While the paper introduces inspiring findings on how SLR (or CLR) help IMP, most components are from existing techniques”.

Given the discussion and concerns related to the novelty of the paper, I feel that the paper requires too major of a revision to be accepted, either improving its core analysis, or presenting it in a better way that clearly distinguishes it from previous art.
Decision: Reject