Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors study how pruning network parameters affects relative recall across classes. In particular, they argue that high levels of pruning increases the asymmetry in recall already present in the dense network. If recall for a class is below accuracy before pruning, it tends to decrease relative to accuracy after pruning, and if it is larger than accuracy before pruning, it tends to improve relative to accuracy after pruning. 

They evaluate this hypothesis by constructing a model-level metric for “intensification” that aggregates information across classes and conducting statistical tests to verify this intensification effect. Using similar statistical tests, they observe that intensification is higher at high pruning ratios, and for high model and data complexity, but lower for low pruning ratios. They also study how the intensification effect varies across 3 existing pruning methods and introduce a new pruning method which they claim decreases this intensification effect.

**EDIT:** The authors have sufficiently addressed my main concerns and the additional experiments are quite promising and make this a stronger paper. The methodology in this paper is quite thorough and I think that the findings will be interesting to the community. I am raising my score from a 6 to 7.
strengths_and_weaknesses: **Strengths:**

Though previous work has demonstrated that the impact of pruning is uneven, as far as I know, the specific hypothesis that pruning intensifies a pre-existing recall distortion is novel. This is an interesting hypothesis and has potential impact as it enables one to predict, pre-pruning i.e. based on the dense model, which classes are likely to be more affected by pruning. The authors perform a thorough and systematic statistical analysis to verify their hypotheses. The authors also study under what conditions (variations in model/dataset complexity and pruning ratio) the hypothesis holds. The methodology is sound and commendable. Additionally, the finding that recall distortion at low pruning ratios decreases intensification is potentially very interesting and impactful. The paper is overall well written and clear, though a few minor typos and grammatical errors are a little distracting—I would recommend a careful proofread.


**Weaknesses:**

Despite the thorough experimentation, this work has a few weaknesses that I think needs to be further addressed. I will outline them here but please see the Questions section for a more thorough discussion.
1. The primary metric $\alpha$ used in this paper has some weaknesses.
2. The direct utility of this work is a little unclear. A more explicit discussion about how this work is different from previous work and what the authors think the marginal benefit of this work is would also be helpful.
   (a) The motivation for and benefit of UP is unclear. 
   (b) Reduction in recall distortion from small amounts of pruning is under-explored.
   (c) Lacks a demonstration of how this can be operationalized.

questions: **First, minor points on clarity:**

The contributions section is not well worded and rather unclear. The paper would benefit from more proofreading of that section and a clearer, more explicit statement of the contributions. In particular:
1. Contribution (i) has grammatical errors that make it a little unclear. I also am not sure what exactly the contribution is here: the method of statistical analysis is not novel.
2. Contribution (ii) should be more explicit: what exactly is meant by "less severely affected" and "more severely affected"?

**Questions about $\alpha$:**

I think that there are some issues about $\alpha$ that need to be addressed since this is the basis for most of the experiments. I am not sure $\alpha$ is the best way to aggregate information across classes. As mentioned in the paper, it down-weights classes with small $\|R^c - A\|$ _before_ pruning. This seems problematic as it would down-weight the examples where there is small negative recall balance before pruning but large negative recall balance after pruning, which is a problematic case. I think that it is also important to include an ablation study of just using the simplest obvious metric: $I^c_t$ averaged across classes without weightings to make sure that the the results still hold.

**Significance:**

I would encourage the authors to be more explicit about what additional information this paper provides compared to previous work and how that is useful to the community. 

Right now, the benefit of the newly introduced pruning method is unclear. The method does not seem very well motivated. Why is doing GP with this additional L2 term a useful thing to do? Why does it reduce intensification of the recall distortion? Also though UP does seem to do better than MP, the gains seem to be marginal. A more explicit and clearly presented head to head comparison and statistical significance of the gains would be helpful here. 

Another route to impact would be to further explore the reduction of recall balance from small amounts of pruning. How significant is this finding and how sensitive is it to the fraction pruned? How does the overall accuracy change? Would the authors recommend this as a good practice to improve all models post training, and if so, why?

Another direction to explore would be: how can this finding of the connection between recall balance pre and post pruning be operationalized? Are there ways we can mitigate this effect? Can this be used for model selection between models that perform with similar accuracy, i.e. if two models have similar accuracy before pruning but one has less recall distortion, will both models have similar accuracy after pruning? If so, this could be used as a metric for selecting models with lower post pruning recall distortion. Is there a tradeoff between lower recall distortion and overall accuracy after pruning? Or perhaps, this new metric should be another axes along which pruning methods should be evaluated. In that case, a well documented and easy to use code base that would enable people to very easily and quickly adopt this metric into their analysis would be helpful. 

I think that improving this work in 1 or more of these directions to really operationalize this would take it beyond an interesting observation and turn it into a strong and impactful work for the community.
limitations: The authors have sufficiently addressed the limitations of the work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Paper studies the problem of recall distortion in neural network pruning. Particularly, authors observe that pruning makes recall relatively worse for a class with recall below accuracy  and makes recall relatively better for a class with recall above accuracy. Authors proposed a new pruning algorithm namely undecayed pruning which can attenuate such effect. Observations made in the paper will be helpful to the future work on network pruning.
strengths_and_weaknesses: **Strengths:**
1. Paper studies an important problem of recall distortion in neural network pruning. 
2. Selected number of criteria is quite representative and contains magnitude based criteria, gradient based criteria and random pruning. 
3. Proposed properties of the pruning are quite interesting and novel. For example, intensification is less severe with the proposed undecayed pruning algorithm but nevertheless more pronounced with relatively more difficult tasks, less complex models, and higher pruning ratios
4. Many pruning papers consider only one setting, but authors use the same framework (datasets, models, pruning ratios) and provide insights by rigorous statistical tests.

**Weaknesses:**
1. Paper lacks some conclusions about how to interpret intensification. In other words, how intensification helps design better pruning algorithms. 
2. The proposed undecayed pruning is affected by the regularization techniques applied during training. For example, weight decay is often used in normal training. Batch Normalization and Dropout have implicit regularization. Could authors comment on how those settings will affect the undecayed pruning?
3. Other pruning methods: Significant progresses have been made recently in developing advanced network pruning techniques, such as [1, 2]. Do these methods have the same effect of recall distortion? Or are these methods already a stepping stone in the right direction towards reducing intensification? 
[1] Minsoo Kang and Bohyung Han. Operation-aware soft channel pruning using differentiable masks. In ICML, 2020.
[2] Yang Sui et al. CHIP: CHannel independence-based pruning for compact neural networks. In NeurIPS 2021.

**Post rebuttal**

Thank you for your detailed rebuttal. All my concerns are addressed, it includes a comparison with other pruning methods. I decide to increase my rating from 5 to 6. 
questions: Please see 1-3 in Weaknesses.
limitations: Yes
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper claims that network pruning inherently makes the model's class-wise accuracy (i.e., recall) imbalanced across classes, which the authors call intensification effect. In order to prove the hypothesis, some ratios are introduced, which models relative performance of each class with respect to the overall accuracy, and thereby intensification ratio is finally defined and experimentally measured. In the experiments, intensification effect is examined in terms of model complexity and dataset complexity, and the corresponding results show that the more complex the dataset, the simpler the model, the stronger the intensification effect.
strengths_and_weaknesses: (Strengths)
1. The paper is well written and motivated, providing novel insights on network pruning.
2. The hypotheses are adequately designed and analyzed in a statistical manner.
3. Undecayed pruning is devised via a novel finding about the relationship between magnitude pruning and gradient pruning.

(Weaknesses)
1. As the authors also mentioned, the tackled pruning schemes are somewhat naive and conventional, compared to the SOTA pruning methods. This weakness makes this work less interesting in practice.
2. Some new metrics should be more rigorously justified. For example, recall balance and intensification ratio can be defined in many other ways rather than computing the relative figure with respect to the overall accuracy.
3. In Sections 7.2 and 7.3, the expected results on intensification effect are not well explained. Thus, why the smaller models are prone to intensification effect? and why the more complex datasets can make the effect stronger?
questions: 1. Please refer to Weakness 3 above.
2. How is the variance of class-wise recalls? Does it become larger with a higher pruning ratio?

limitations: Yes.
review_title: null

Meta-review: metareview: This paper studies the disparate effect of model pruning across classes and proposes a new method to reduce the "recall distortion" across classes. This is a critically important problem, and one which has just begun to be carefully studied in the literature, so this work is timely and relevant. All reviewers recognized the relevance of the problem and the novelty of the authors' approach, both with respect to the new approach presented here, as well as the detailed analysis of the various factors which impact recall distortion. There were some concerns regarding the complexity of the pruning algorithms studied, but the authors provided a number of additional experiments on other pruning approaches in their response, finding qualitatively similar effects (as might be expected given the reliance of many of these approaches on some form of magnitude pruning). I think this paper will be a valuable addition to a poorly understood and important research area, and should be accepted.
confidence: Certain
Decision: Accept