Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper revisits a traditional model compression method, global magnitude pruning (GP), and shows that GP can achieves state-of-the-art. The paper further improves GP by introducing minimum threshold (MT). Experiments on ResNet show that GP+MT achieves better accuracy with the same sparsity ratio compared to GP.


main_review: 1) My major concern is that the paper does not clearly explain differences between GP in this paper and traditional GP methods. Based on Equation (1), I think GP in this paper is the same as GP adopted in Han et al.(2015a). If GP in this paper can achieve SOTA, Han's method can also achieve SOTA. My suggestion is that the authors provide qualitative or quantitative comparison with previous GP methods.

2) The proposed MT is not a sufficient improvement. First, GP+MT increases accuracy on MobileNet, while slightly decreasing accuracy on ResNet. Second, I am not sure whether MT is sensitive to selected threshold. 

3) Since CIFAR-10 is a tiny-scale dataset, I think experiments on CIFAR-10 cannot sufficiently validate that for WideResNet-22-8, GP+MT can further improve accuracy compared to GP with the same sparsity ratio.
summary_of_the_review: The paper does not clearly explain differences between GP in this paper and traditional GP methods. The proposed MT is not a sufficient improvement. So my rating is "5: marginally below the acceptance threshold"
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: A very simple and effective pruning method is proposed. Instead of layer-wise pruning, a global threshold is used to prune weights according to their magnitude. In addition, to avoid over-pruning, a minimum number of parameters is preserved for each layer after pruning. Experiments on CIFAR-10 and ImageNet validate the effectiveness of proposed method. 
main_review: * Layer-wise sparsity ratios used to be tricky hyper-parameters. A global threshold and a minimum number of preserved weights eliminate layer-wise sparsity ratios. In this context, the proposed method is practical. 

* Although the empirical results seem promising despite of the simplicity. it is not straightforward to understand why we should use global threshold across layers and why MT is crucial to ensure superior performance for certain models. For example, how does over-pruning impact the performance? 

* A related study [1] argues that layer-wise sparsity is of importance. A comparison (both theoretical and empirical) should make the result stronger. 


[1] Lee, Jaeho, et al. "Layer-adaptive Sparsity for the Magnitude-based Pruning." International Conference on Learning Representations. 2020. 
summary_of_the_review: The simplicity and effectiveness of proposed method are important and interesting. The findings will inspire more studies on the nature of network pruning. However, it is not easy for readers to understand where the effectiveness of proposed method comes.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper revisits Global Pruning (GP) and makes a case to consider it seriously as part of the pruning literature. The authors also present an addition to GP, that claims, to further increase accuracy and reliability called Minimum Threshold (MT). MT is the constraint put on every layer of the neural network to ensure they are not pruned beyond a certain limit which might result in catastrophic accuracy drops. 

The paper presents experiments on CIFAR-10 with a few CNN architectures along with current pruning baselines (layer-wise mostly. They also show results on ImageNet using ResNet50 and MovileNetV1. Following these experiments, the authors claim that GP or GPMT are SOTA for pruning despite their simplicity. 

The paper also includes some discussion on how output patterns of each layer change with different pruning schemes and a note about how to set the optimal MT values. 

While I agree with the sentiment of the paper, I do not think the paper is presenting anything new (even from a benchmarking perspective) but rather is missing the point about what makes GP vs layer-wise pruning a worthy trade-off often. I will list my concerns in the main review.
main_review: I will go sequentially in the paper but will start with the strengths follow it up with the weaknesses: 

Overall, the writing quality is OK and is easy to follow. The information has been well put together for a new reader to understand without having to worry about the background. The authors should be appreciated for that.  

Strengths:
1) The fact that authors revisited GP to show its power is a very nice step. GP has been (according to me) the strongest baselines for pruning for a given parameter budget even now (has been acknowledged in multiple papers). 
2) The idea of MT makes sense and is a valid fix for the overly pruned layers in general.
3) The experiments on ImageNet are commended because the insights from there a more valuable than other things presented in this paper. 
4) The discussion on network architectures, outputs and settings of MT are interesting in general. 
5) I really liked the related work, it is thorough from a literature review standpoint and would recommend new readers to go through it for a nice picture of the field 

Weaknesses:
This might sound a bit critical, but I would appreciate it if the authors understand the merit of whatever I am trying to put forward here. 

1) No one ever claimed GP is not sufficient to achieve SOTA performance on pruning. GP is extremely powerful no matter what gives the highest accuracy (often) for the same #params even at the highest sparsity levels. GP is not forgotten or overlooked by any means. The reason why people work on layer-wise pruning is slightly different and I will come to it later. 
2) While the authors mentioned people of LTH community probably used GP as one of the potential pruning techniques, GP was the only one that resulted in strong tickets at higher sparsities, making IMP explicitly work on GP no matter what other factors are. 
3) MT comes into the picture only when there is one-shot pruning and GP has been shown to work great even with gradual schedules (IMP is an example) and will not suffer from pruning out entire layers just because of the gradual processing. I would even argue setting the small pruning schedule is data-independent and has a similar cost as finding the optimal MT as done in this paper. MT makes sense for one-shot pruning, but one-shot pruning itself is problematic (will explain in a bit).
4) Table 1 is a poor representation, I would say, STR and GMP are extremely easy to implement. GMP could be done in One-shot and GMP is data-independent no matter what. I would like to hear authors' thoughts on this. 
5) Coming to the setup, the biggest problem I have with the way GP or GPMT is being used is the application on top of a pre-trained network. Eg. On imagenet the fine-tuning of GP or GPMT takes 20 extra epochs, which isn't the case with all the baselines in the discussion. All of them are pruning while training baseline that only takes 100 epochs at best. I personally have adapted GP to be pruning while training like with IMP and GP does amazingly well, so the aspect of one-shot is overshadowed by the extra cost involved in fine-tuning. I don't know how we can make a case for these trade-offs to make the comparison fair.
6) The big one, the paper completely disregards the compute cost during inference (FLOPs) for these pruned models. The biggest difference GP and layer-wise pruning methods have is the inference costs involved. Please see STR or RigL papers to see about that axis of comparison. GP often has 2x more inference FLOPs than uniform-sparsity with very little gain in accuracy. If uniform sparsity was adjusted for that compute costs, the accuracy will be much higher. This paper, while being a study on GP, completely overlooks this aspect of GP or GPMT.
7) While the results on CIFAR-10 are good for debugging and understanding, the insights aren't particularly transferable. I would also not read too much into 40% sparsity results or even 95% sparsity because the networks involved are so heavily overparameterized that even at that sparsity level CIFAR-10 seems like a simple task. However, I would like to say Table 5 is a real thing for one-shot pruning, but will not happen for gradual GP. I would recommend authors implement gradual GP to see what it brings to the table, along with reduced training costs (debatable because one can think of pre-trained models as free). 
8) Figures 2 and 3 just reinforce what I am saying above. 
9) The gains in Tables 6 and 7 are not significant enough to make sweeping claims on SOTA while also not measuring FLOPs.
10) The use of "huge margin" is highly discouraged in Table 8 because the gains are very insignificant and trust me the FLOPs are going to be so exorbitantly high that the accuracy vs # params doesn't matter in that case. 
11) As mentioned in the paper, MT doesn't seem to bring anything to the table for ImageNet experiments, just because of how the weight norms across layers change. MT is probably not a valuable addition in this case. 
12) I would not get into the discussion of output preservation because, it doesn't necessarily mean that the model is doing well if its outputs are closer to some other model, the yardstick itself is flaky here. 
13) Lastly, the discussion of MT tries to make it feel like it is not complex or data-dependent. But the same paper says GMP schedule is complex. I would say searching for MT is as good as coming up with a cubic pruning pattern which will even help GP when done in a gradual fashion.

Overall, I do not think the paper brings any new observations into the limelight and also ignores the issues present with whatever has been presented. The paper is not ready for publication, but I am happy to chat with the authors to help them understand and (myself) gain perspective if I am missing something. 


summary_of_the_review: The paper does not propose anything novel (which is fine) but the observations are also not new while the paper ignores most of the issues and tries to underplay other baselines. 

The paper is not ready to be published but is an excellent starting point for new readers. 


=---------------------------------------------------------------------------------------

After an extremely long discussion with the authors and providing them with things I felt were necessary for fixing the experiments, claims etc., I think the paper is not ready to be published. The authors should revisit most aspects of the paper if they want to make it a benchmark paper for GP and claim so. None of the insights and contributions are novel given my discussion and experience. I  hope the authors understand that it is better to publish a ready paper than a paper changing around a lot at this point. 

I vote for rejection.
review_title: null

Meta-review: The paper claims that one of the most common (and obvious) pruning methods in the literature today (global magnitude pruning) is "overlooked" and "seen as a mediocre baseline by the community." As an active member of the pruning research community myself, I can attest that this is simply not true. I am in strong agreement with reviewer MHY2 and - after reading the discussion around that review and the paper itself in detail - I confidently recommend rejection.

Magnitude pruning itself dates back decades, at least to the work of Janowski (Pruning vs. Clipping in Neural Networks, 1988). The paper is correct that *global* magnitude pruning (in which all weights are compared in a layer-agnostic manner) was largely ignored in favor of layer-wise magnitude pruning (i.e., pruning all layers by the same amount) in much of the work that popularized magnitude pruning (e.g., Han et al., 2015). However, global magnitude pruning has become much more popular since that time. In work establishing the lottery ticket hypothesis, Frankle and Carbin (The Lottery Ticket Hypothesis) use it in certain cases and - later - in all cases (Frankle et al., Linear Mode Connectivity and the Lottery Ticket Hypothesis). In the past several years, global pruning in general has become the de facto way to use all new pruning heuristics (e.g., SNIP: Single-Shot Network Pruning based on Connection Sensitivity; Picking Winning Tickets Before Training by Preserving Gradient Flow; Pruning Neural Networks without Any Data by Iteratively Conserving Synaptic Flow). Moreover, other papers have specifically advocated that global magnitude pruning is state-of-the-art within recent years at this very conference: Comparing Rewinding and Fine-Tuning in Neural Network Pruning (Renda et al., ICLR 2020 oral): "We propose a pruning algorithm...that matches state-of-the-art tradeoffs between Accuracy and Parameter-Efficiency across networks and datasets:...globally prune the 20% of weights with the lowest magnitudes." (This paper does not cite Renda et al. despite the fact that it is a prominent paper that directly contradicts the purported problem that the paper relies on to support the significance of the findings.)

In short, in the pruning literature, the idea that global pruning, magnitude pruning, or global magnitude pruning is overlooked or is not recognized as a strong baseline is simply preposterous. The reason that global magnitude pruning has "largely been ignored in recent years, generally being relegated to the position of a baseline for comparison" is because it is a simple technique whose efficacy has long been known and established - exactly what a good "baseline for comparison" should be.

The paper has narrowed its claims somewhat during the discussion and revision period, advocating for a one-shot global magnitude pruning strategy that "does not require any complex pruning frameworks like RL or sparsification schedules [or]...iterative procedure." To do so, however, the proposed method replaces each of these "complex" hyperparameters with another set: whether or not to use a minimum threshold (MT) and where to set it. Even if the approach isn't iterative, the hyperparameter search necessary to set it almost certainly is, and it is unclear whether searching for the MT value is any more efficient than the other approaches. The costs of this hyperparameter search need to be measured. And iterative pruning's costs can often be mitigated by making pruning gradual, something the paper considers superficially in the revisions.

Finally, as reviewer MHY2 observes, one of the primary reason papers *don't* use global magnitude pruning is that, although it leads to higher sparsities than layerwise magnitude pruning, it also often leads to higher FLOP counts. Although FLOP counts are a terrible indicator of real-world speedup, they are a much higher-fidelity indicator than parameter-count, which neglects the fact that - in convolutional networks - a small number of parameters can lead to vastly more FLOPs if they operate on larger activation maps (i.e., before the activation maps have been downsampled). In the revisions, the paper gives a token nod (and a superficial dismissal) to this fact in Sections 4.3 and 6, but the paper needs to fully acknowledge this point by measuring and discussing its consequences. "Look[ing] at this in future work" is not enough.

Due to these many concerns, I strongly recommend rejection.
Decision: Reject