Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper investigates what makes network pruning improve accuracy despite the removal of parameters from the model. The authors contend that the folkloric claim about smaller models implying better generalization is not sufficient given the recent studies about the overparameterized regime, in which increasing model size may also improve generalization. As an alternative, the authors investigate the effect of pruning directly on the samples used for training by using a metric from the literature to identify how relevant each sample is for training the neural network. Consequently, they find out that pruning leads to a greater reduction in the loss function for samples that are more challenging to classify correctly, whereas pruning may also lead to an increase in the loss function for noisy samples that actually hurt generalization.
strengths_and_weaknesses: This paper presents a well organized and very compelling analysis on the effects of pruning on how the neural network learns from the training set. However, I was disappointed at how this paper, like many others in the LTH bandwagon, simply ignored the rest of the work currently being done in network pruning. Whereas the conclusions are very interesting, it makes me wonder what would happen if the same analysis was done with other pruning algorithms as well, or at least with random pruning for contrast. 

I find the first two contributions very interesting, although I have questions below about how to distinguish more clearly between them. On the other hand, the third contribution is a bit of a stretch in my opinion: your work shows two factors that may partly explain what makes network pruning so useful. In that sense, they remain in equal footing with model reduction in terms of explaining something but not everything. I believe the wording is lines 92-94 is unnecessarily drastic. For example, the results in Section 5.2 make it clear to me that we actually cannot ignore the effect of model reduction in itself.

### Ignored work

The first paragraph cites modern papers that are exclusively based on magnitude-based pruning, but cites older work based on gradient-based pruning. The latter literature has grown into something that is often denoted as impact-based pruning and has remarkably good results as well. For example, the survey by Blalock et al. (2020) [1] outline these two kinds of approaches as predominant in the literature. With exception of  Wang et al. (2020), I see none of this second line of work acknowledged in the paper. I believe the authors should not ignore it, since it is a very active branch of the network pruning literature [2-14].

[1] https://arxiv.org/abs/2003.03033

[2] https://arxiv.org/abs/1506.02515

[3] https://arxiv.org/abs/1611.06440

[4] https://papers.nips.cc/paper/2017/hash/c5dc3e08849bec07e33ca353de62ea04-Abstract.html

[5] https://arxiv.org/abs/1711.05908

[6] https://openreview.net/forum?id=HJfwJ2A5KX

[7] https://openreview.net/forum?id=B1VZqjAcYX

[8] https://arxiv.org/abs/1905.05934

[9] https://arxiv.org/abs/2001.00218

[10] https://arxiv.org/abs/1906.06307

[11] https://arxiv.org/abs/1911.07412

[12] https://arxiv.org/abs/2004.14340

[13] https://arxiv.org/abs/2006.05467

[14] https://arxiv.org/abs/2102.07804

[15] https://arxiv.org/abs/2107.07467

[16] https://arxiv.org/abs/2203.04466

### Other comments about the writing of the paper

Abstract:
- I could not understand the last sentence of the abstract before reading the paper. I would recommend rewriting it, possibly breaking it down into multiple sentences.

Introduction:
- Line 38: Add space between word "generalization" and following citations
- Line 67: Remove first "but"
- Line 82: Remove "count reduction" 

Section 2: 
- Line 116: Remove second period

Section 3:
- Line 122: "are affected": add "more" between those words?
- Line 129: "he" -> "they" (that work has multiple authors)

Section 5:
- The title of this section is confusing, and I honestly was expecting something else when I read it: that you would try to isolate the effects of optimization and regularization from each other. Can you please change it to something more direct, such as "Isolating training time and model size effects"?

References:
- Han et al. (2015) is cited twice: remove one of those

Appendix B:
- Line 514: "interests" -> "interest"
questions: 0) Can you please give more consideration to the other modern lines of work on network pruning? 

1) Your work heavily implies that Renda et al. (2020) is the state-of-the-art on network pruning. Can the authors cite other papers acknowledging that status? What about the extensive recent work on pruning at initialization, which spans way more than Frankle et al. (2021) (see Wang et al. (2020) + [7,10,13])?

2) The fact that a single algorithm is used for pruning is a noticeable limitation of the work. Have you tried other algorithms, even if something as simple as one-shot magnitude-based / gradient-based pruning, or - better yet - random pruning just to make a contrast?

3) I am not entirely convinced of the results in Section 5.2: the plot seems to indicate that there is a largely negative distinction for higher p when compared to dense models, and that the results are otherwise very similar. In line 269, how can you do better by -4 to 0.4%? Please explain.

4) The expression "better optimization" is very imprecise and in fact confusing (in line with people saying "more optimal"). The same about wording such as "the change in optimization due to pruning". Can the authors please consider a less ambiguous and more specific expression, such as "better training"?

5) Figure 1 (b): Have you measure these changes on the test set as well? 

6) Your paper implies that negative changes due to the loss function are due to relevant but challenging examples, whereas positive changes are due to noisy examples. Whereas you have done some ablation of the noisy part with artificially inserted noise, have you looked into whether the samples for which the loss function change the most are indeed not the same as the pruned percentage goes higher and the changes become predominantly positive?

7) I am a bit curious about your thoughts on the rightmost plot of Figure 8 (a): Does that imply that MNIST is such an uniform dataset that the noisy discussion does not apply as much there?

8) In Section 4 and Figures 3-5, how did you choose the percentage p of random label noise? Why is it different across models/datasets? Can you add p=0% as another curve in each plot of Figures 3-5?

9) In the conclusion of Section 4, can you please describe in more detail how your computational results imply that low pruning levels are dominated by the optimization and that high pruning levels are more influenced by regularization?

10) What do you mean by pruning "stability" in line 297?

11) What "novel analytical technique" are you claiming in line 331?

12) How do you think your results relate to what happens in the case exact compression methods [14,15]?
limitations: The authors did a good job describing the limitations, except for using a single pruning algorithm to derive their conclusions.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper finds that the size-reduction hypothesis can not fully explain the pruning's effects on generalization increase. It proposes the decomposition of such increase to better optimization with lower training loss and stronger regularization by avoiding hard training samples. Furthermore, the paper shows that neither extended training time nor size-reduction alone can not fully explain the generalization improvement. Thorough experiments are conducted to support the claim with the help of the EL2N score.
strengths_and_weaknesses: Strengths:
1. Most of the claims in the papers are backed by extensive experiments.
2. The paper is easy to follow.
3. It is a very interesting phenomenon to me that pruning actually improves optimization at a low sparsity ratio. Since most pruning papers mainly focus on test accuracy alone.
4. The decomposition of pruning's effect into better optimization and stronger regularization through the EL2N score is original and important.

Weaknesses:

1. For section 5.2, the comparison actually seems a bit unfair to me. First, the down-scaling width method, though having the same number of parameters compared to the weight pruning method, may have much less representation power (model capacity) due to lower feature map size. Second, the granularity of down-scaling width methods is not fine enough. For example, only 5 sparsity ratios are tested for the C100-ResNet32 combination while there are 10 for weight pruning methods. We may see a matching/better performance on datasets with random label noise.

2. For Figure 2, I think it is also helpful to plot average CE loss before and after pruning. It may give a better visualization if the neural networks avoid fitting difficult examples.
questions: I think it is important to show the training accuracy of width down-scaling methods in Figure 7. If the training accuracy of the two methods is far apart, the validity of the conclusion may be undermined. Since if there is a great training accuracy drop after using width down-scaling, there is no point talking about generalization. 

Also see my concerns in weaknesses.

[Minor issues:]
1. It would be better to also include the optimal sparsity level for Figure 3.
2. Maybe it is better to give a further explanation of the augmented training method for Figure 1. If I understand correctly, % weights remaining is always 100% for the augmented training method. However, augmented training is also plotted using changing % weights remaining as the x-axis.
limitations: Yes, the authors have addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper provides emperical study on the effect of pruning on the generalization of neural network model. The discussion focus on state of the art iterative pruning pipeline, and discuss the effect of pruning on achieving better optimization and better regualrization for the model training process.
strengths_and_weaknesses: Strengths:
1. Understanding the effect of pruning on model generalization is an interesting and important topic, this paper makes insightful contribution along the line.
2. The paper is overall well written and easy to follow.
3. The paper make novel contribution by subgrouping the data with the difficulty of learning, and make seperated observation for data of different EL2N percentile. This leads to new insight on pruned model's optimization outcome and can inspire future investigation on other neural network optimization techniques. 

Weaknesses:
1. The paper introduces two effects: better optimization and stronger regularization, and two possible reasons: longer training and reduced model size, but doesn't make clear discussion on what's the relationship between the reasons and effects. It would be better to discuss if (following common belief) longer training mainly leads to better optimization, and smaller model size mainly leads to stronger regularization; or the effect of the two potential reasons cannot be distinguished
2. The experiment setting on model size reduction may be improved: Since the work is mainly based on the pruning method used in lottery ticket hypothesis I believe the discussion should be focusing on element-wise pruning, yet the model size reduction experiment is set for an channel-wise pruning scenario. A better experiment may be reinitialize the nonzero weight element of a sparse model and retrain the model for the same epochs, this will give a fairer comparison with previous iterative pruning observations
questions: 1. Does longer training mainly contributes to better optimization, and smaller model mainly for stronger regularization?
2. What would be the effect in Figure 7 be if the weight is pruned in a elementwise fashion with the same zero positions as the fully trained sparse model from iterative pruning?
limitations: The limitation and social impact is adequately discussed.
review_title: null

Meta-review: metareview: All the reviewers, including the AC, agree that the paper makes a significant contribution to the field that deserves publication at NeurIPS. The AC refers the readers to the reviews for the discussions on the pros and cons of the paper. 
confidence: Certain
Decision: Accept