Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The paper attempts to study and compare different pruning "criteria".
The paper addresses an interesting question, trying to evaluate if there is a need for a more complex pruning criteria. However the paper can be better organized. And some of the concerts are hard to follow. 
main_review: Originality: The work is original and provides an attempt to evaluate the validity of new and more complex pruning criteria
Quality: The paper attempts to prove empirical study along with their theoretical/mathematical proof which adds more significance to work. However, the paper organization needs additional work. Especially the introduction part, where the main concepts and problems are introduced. 
Clarity: The paper is hard to follow. The main concepts are purely introduced and paper is hard to follow. "Convolution Weight Distribution Assumption" were not well introduced and hard to follow. It would be helpful if more details be provided on the existing pruning criteria. Provided grouping is too generic and not well organized.
Significance: The work attempts to provide a tool for more unified evaluation of the pruning methods, which was missing in the current methodology. 
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Paper studies the problem of selecting a criterion for ranking based structured pruning. Particularly, authors observe that norm based criteria show quite similar behavior. Authors proposed a similarity and an applicability properties of 8 considered criteria. Additionally paper proposes a Convolutional Weight Distribution Assumption. Observations made in the paper will be helpful to the future work on structured pruning.
main_review: Strengths:

1) Paper studies an important problem of comparing and understanding various pruning criteria. 
2) Selected number of criteria is quite inclusive and contains norm based criteria, gradient based criteria, and activation based. 
3) Proposed properties of the pruning criterion is quite interesting and novel. For example Similarity is a good starting point for developing new pruning criterion in the future. Applicability attempts to estimate certainty of different criteria.
4) Comparison between layer-wise and global pruning is on point and provides different observations. Many pruning papers consider only one setting and it makes hard to compare them, but authors used the same framework and provide insights that same criteria behave differently. 

Weaknesses:
1) Paper lacks some conclusions about how to interpret Similarity, Applicability and CWDA can help to understand what criteria is the best out of all considered. It might also seem that a criterion with random score assignment might score the highest in similarity and applicability (can authors comment on the Random pruning as it is an important baseline for any pruning method?). Would be interesting to see Fig 5 for that and CWDA. 
2) CWDA depends on regularization techniques applied during training. For example weight decay will push weights to be Gaussian, Batch Normalization will de-correlate filters, Drop-out will correlate them more. Could authors comment on how those settings will affect the study and if their settings were varied during the study.
3) Applicability assumes that ideal criterion will clearly distribute filters with ranks (deviation of the scores will be high). However, the hypothesis might be wrong as regularization will significantly affect it. For example if drop-out is used then it is known that the layer will be more robust to pruning (doesn't matter what we remove because it prevents co-adaptation). In this case the ideal criterion will have a low Applicability but will be still relevant. Because of this fact, the Applicability property might be not correct even for the graph in Figure 5. 

For rebuttal I am looking in understanding the following:
Answer to points 1-3. Analysis of random pruning will be very interesting.
Can authors finetune some of the model they get from pruning to verify that provided analysis doesn't break after the model is trained (for CWDA at least). Also we will see if similarity is important, as a different choice for only 1 neuron might make a huge difference. 
Comment on the limitations (what authors see as a limitation for this work) as well.

Originality - Novel, didn't see the same study in other works
Quality - well written
Clarity - great
Significance - will be interesting to benchmark new applications

limitations_and_societal_impact: The main limitation seem to be in the criteria during pruning and do not reflect (or not shown) any outcome of the fine-tuning these models. For example out of 3 metrics we want to see what should be maximized/min to develop new criteria.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose and verify an assumption about the distribution of convolutional weight called CWDA, which reveals that the well-trained convolutional filters in each layer approximately follow a Gaussian-alike distribution. Based on this assumption, the authors analyze the similarity and the applicability problem among different pruning criteria. The point of this analysis is that norm-based metrics, particularly L1 and L2, behave quite similarly. Experimental results on various network structures, tasks, and datasets demonstrate the proposed analysis. The resulting conclusions are interesting. However, there are some issues in the paper. Detailed comments are as follows.
main_review: **Contributions**:
1.	The authors propose and verify an assumption called CWDA, which reveals that the well-trained convolutional filters approximately follow a Gaussian-alike distribution. 

2.	Based on the proposed CWDA, the authors analyze the similarity and the applicability problem of different types of pruning criteria, which is an interesting perspective. These two blind spots are able to guide the researchers to design more reasonable criteria. 

**Questions and points needed to be improved**:
1.	The title “Rethinking the pruning criteria…” is over-claim. In fact, the main point of the paper is to show that norm-based metrics behave quite similarly. The authors also analyze the similarity and applicability of other pruning criteria. However, the conclusion here is not clear. 

2.	In Figure 3 and Table 4, the authors only show the performance of Norm-based pruning criteria. It would be better for the authors to compare the performance between Activation-based [1-2], Importance-based [3-7], and BN-based criteria [8] on layer-wised pruning settings. These results would help the community to better understand different pruning criteria.

3.	There are some repetitions in this paper. (1) References 11 and 26 are repeated. (2) References 12 and 34 are repeated. (3) Sections I and P are repeated in the supplemental material.

4.	Some papers in the references list have been published in a conference/journal. For example, “Wide residual networks” have been published in BMVC 2014. Please cite the conference/journal version instead of the arXiv one.

5.	The authors do not reference the correct Lamma or Equation in Line 740, 762, 780, and 782 of the supplemental material.

**References**

[1]	Network trimming: A data-driven neuron pruning approach towards efficient deep architectures. arXiv 2016.

[2]	An entropy-based pruning method for cnn compression. arXiv 2017.

[3]	Pruning convolutional neural networks for resource efficient inference. ICLR 2017.

[4]	Importance Estimation for Neural Network Pruning. CVPR 2019.

[5]	SNIP: Single-shot Network Pruning based on Connection Sensitivity. ICLR 2019.

[6]	HRank: Filter Pruning using High-Rank Feature Map. CVPR 2020.

[7]	Discrimination-aware Network Pruning for Deep Model Compression. TPAMI 2021.

[8]	Learning Efficient Convolutional Networks through Network Slimming. ICCV 2017.

limitations_and_societal_impact: Please discuss the limitations of the proposed method.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper studies 1) the similarity between different pruning criteria for channel pruning, 2) the applicability of different pruning criteria, where applicability refers to the confidence in the importance ranking of filters.  The authors empirically show that for some common pruning strategies such as l1 and l2 pruning and other norm-based pruning, the importance ranking of the filters is almost identical. Moreover, they claim with theoretical and experimental support that the importance score of different filters is very close to each other under certain conditions.  The main contribution of this paper is to analyze these two phenomena for better pruning criteria in the future. 
main_review: I like the broad topic of the paper -- understanding the pruning criteria beter. However, the current version of the paper leaves many questions unanswered and needs substantial improvement. I list my concerns below:

- Writing: I think the writing can be improved significantly in terms of clarity, flow, and grammar. Here are some of the easy-to-fix typos I noticed:  
    - line 62: unnecessary column at the end of the sentence.
    - line 93: 'Taking a convolutional layers'. 
    - line 101: missing space after the comma.
    - line 145: missing space after the comma.
    - line 146: unclear sentence?
    - line 151: consider about.
    - line 162: n-dimensional.
    - line 191: 'Since Proposition 1,'.
    - line 194: 'closed to 0'.
    - line 208: 'in more types of pruning criteria'.

- Overall, I am not convinced with the importance of the problem at hand. It is not surprising that norm-based pruning criteria give almost the same ranking (Similarity). It is also not surprising that for some layers, the importance scores are very similar (Applicability). It would be interesting to compare the effect of pruning filters with similar importance scores on the performance. Perhaps they all perform well. In that case, the pruning criterion is reliable. If some of them do not perform well, then the pruning criterion does not produce a reliable ranking of filters' importance.  However, no such result is provided. Another question to ask is whether very similar importance scores relate to the incompressibility of that layer. Maybe the conclusion we should get from this observation is if importance scores are very similar, it is better not to prune any filters. However, none of these interesting points are discussed in the paper. So, I am having difficulty in understanding the authors' goal in analyzing 'similarity' and 'applicability' phenomena. Can the authors provide a stronger motivation for their study?

- I am confused about the setup in CWDA (lines 84-98). In line 85, $F_{ij}$ are assumed to be iid. However, in line 96, it is said that parameters in the same channel of $F_{ij}$ are observed to be linearly correlated. According to this observation, iid assumption is unrealistic. Did the authors make this iid assumption to simplify the problem?

- I also have concerns about the use of expectation and variance operators. As far as I understand, $\mathbb{E}[X]$ refers to the average of the importance scores in one layer. Do the authors think of the importance scores as different realizations of the same random variable X? If so, 1) why is $X=(x_1,...,x_k)$ defined as a deterministic sequence? 2) the authors should make it clear that they are using an empirical approximation to $\mathbb{E}[X]$. I feel like the authors only wanted to find the average of $X$ and misused expectation operator. Can the authors clarify their notation here?

- There is a recent paper on layer-wise sparsity [1], where a new pruning criterion and strategy are proposed to avoid layer collapse.  The authors may want to have a discussion on that in the paper.

- Minor: I would avoid abbreviations like 'can't' in an academic paper.

[1] Lee, Jaeho, et al. "Layer-adaptive Sparsity for the Magnitude-based Pruning." International Conference on Learning Representations. 2020.
limitations_and_societal_impact: The authors discuss some limitations in Section 6. I suggest the authors discuss also their iid assumption and explain why they had to make that assumption although it is unrealistic. The authors do mention how their findings would impact the research community. They do not address the potential negative societal impact of their work.
review_title: null

Meta-review: The paper provides a study of pruning criteria by evaluating different methods and defining metrics that help understand the different properties of the various criteria out there. The topic of DNN pruning is quite rich today and I personally find it refreshing (as do some of the reviewers) to see a paper that tries to sort out the different methods. The paper is far from a complete solution to the problem, but it provides a good start. Given the importance and magnitude of the problem, I believe the paper is worthy of being published at NeurIPS and will likely motivate follow up papers that will improve the way we currently measure pruning criteria.
Decision: Accept (Poster)