Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Inspired by the integrated gradient method in attribution methods, the authors propose a new importance metric for pruning: an integral of the product between the norm of the parameter weight and its attribution along a path between this weight value and a baseline (zero) value. This metric captures more global view than previous magnitude-based and gradient-based pruning methods. Then, the authors devise the so-called SinGE method to intertwine the pruning with the network fine-tuning. Empirical studies are conducted to compare SinGE with both structured and unstructured pruning methods on Cifar 10 and ImageNet.
strengths_and_weaknesses: ## Strength
1. Motivation is very clear and the idea of using the approximated integration as the criterion of pruning is a good way to measure the influence of forcing a parameter to 0.
2. This paper has clear writing and easy to follow.

## Weaknesses

1. Figure 1 is not effective and clear enough to facilitate the readers’ understanding on the arguments made in Sec 3.1. One reason is that the notation $\mu$ is not introduced before its appearance. Also, the claims at line 129 – 133 about the effects of the change in values are not clearly depicted in Figure 1.

2. The claim that SInGE significantly outperforms existing state-of-the-art DNN pruning methods seems to be a bit inappropriate as it’s sometimes inconsistent with what demonstrates in the empirical studies. For example, in Table 5, the accuracy of SinGE is nearly the same as those of Adapt-DCP, ManiDP-A and MDP. However, the latter remove significantly more parameters or FLOPS than SinGE does.
questions: 1. As far as I undertand, the integration evaluation will take much time, so do the authors think that adding an experiment of the time efficiency necessary?
2. There are a few typos, for example, at the range of $\mu$ at line 138, the spellings of trade-offs at line 167, and the incompleteness of parentheses at line 181.
3. As I suppose the bolded numbers in the tables stand for the best results, some of those in Table 3 and 4 are not correctly labeled, which is a bit misleading.
4. The top-1 accuracy of integrated magnitude method without fine-tuning and with 1000-step entwined fine-tuning seems to be the same (both reported as 85.38). I’m wondering why there is no increase in the accuracy.
5. May I ask what does the notation “1<” stand for in Table 1?
6. As the performance of SinGE is very close to its counterpart in many cases, could the authors please provide the error bars for the results reported in Table 2-6?
7. May I ask whether the results in Table 3 use fine-tuning or not? If yes, I’m wondering whether the same fine-tuning strategy (i.e., entwined) is adopted for all the candidates?
8. In Table 6, only the accuracy of SInGE when 90% of parameters removed is provided but not the other methods. It would be appreciated if the authors could provide the performance of other methods at this setting to support the superiority of SInGE.
limitations: The authors have included adequate limitations.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a new neuron pruning  method based on the Integrated Gradients (IG) attribution algorithm. The paper claims that IG-based pruning is more optimal than magnitude and local gradient-based methods since it accounts for the gradient effects globally along the path from 0-valued baseline weight to given input weight in each layer l. The paper performs impressive number of experiments for a number of structured and unstructured pruning techniques from the state of the art literature and shows that IG-based pruning outperforms those methods based on two evaluation metrics (% removed parameters and % removed floating point operators).
In addition to that the paper also proposes entwined pruning and fine-tuning procedure and shows that it help to obtain better accuracy while performing the pruning. 
The experiments in the paper are conducted for ImageNet and Mobilenet V2 backbone for ImageNet and  CIFAR10 dataset.

strengths_and_weaknesses: **Strengths**

+ Overall the idea of the paper is clearly described, related work is properly cited, experimental setup and results are properly discussed
+ The paper shows strong empirical evidence that IG-based pruning outperforms magnitude, gradient-based approaches, as well as a number of other state of the art approaches from the neuron pruning literature. 
+ The paper also shows that entwined fine-tuning helps to improve model accuracy regardless of pruning technique using over 80% pruning target.

**Weaknesses**

+ I think that in general the readability (fluency) of the paper can be further improved. Specifically, I felt that in the notation and description of the approaches, there is some level of expectations that the readers know the notations of structured pruning techniques and integrated gradients. For example, neither Figure 1 nor the context talking about Figure 1 describes what $\mu$ and $\mu^2$ are.
+ Figure 1 and its descriptions are a bit confusing. Originally, I was thinking that a, b and c are weight matrices but later the paper refers to a and b as two different weights. It would be good to improve the description of Figure 1. 
+ Given that entwined fine-tuning is one of the important contributions of the paper it would be good to describe it in detail. It would be good to make it clear what fine-tuning over O steps mean and what we mean by fine-tuning the network F over a batch from D. It seems that the authors make an assumption that the readers are familiar with the specific fine-tuning approach that they are describing.
At this point from the given description it is hard to tell the amount of novelty in the entwined fine-tuning.
+ Table 1: It would be great if the authors explain relatively large std in integrated magnitude x grad C_{IG}^2 method

**Minor Comments**

+ On page 9 a notation, $N_n$ but I don’t see a description of notation $N$.
+ Table 6 has 2 SInGE rows but it is not clear what are the differences between those two SInGEs


questions: 
+ From Table 1 it is unclear what the authors mean by integrated gradients C_{SG}^2 and integrated magnitude x grad C_{IG}^2. It would be good if they could elaborate it further and also describe which of those two they used for the results in tables 3, 4, 5 and 6.
+ I wonder if Noisegrad paper which introduces stochasticity to model weights could also be relevant to this paper.
https://arxiv.org/pdf/2106.10185.pdf

limitations: 
I do not see any potential negative societal impact in this work
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a novel pruning method based on an integrated gradient pruning criterion, which combines a magnitude and gradient based criterion and integrates the product over the path of the neuron removal. 

In addition, the work proposes a fine-tuning flowchart to improve the performance of the fine-tuned network.  This strategy is an iterative layer-wise approach which sets the pruning ratios for each block separately. 

Lastly, a comparison of SInGE on ResNet-56 on Cifar10 and ResNet-50 and MobileNet-V2 on ImageNet shows state-of-the-art performance. 

strengths_and_weaknesses: Strengths:

* I like the idea of an integration criterion and according to Table 1 it seams that this is the main driver for the superior performance of SInGE. To the best of my knowledge this is the first method to use an integration criterion for (structured) pruning. 

* The Integration criterion is well motivated, and the ablations support the design choices made by the authors.

* SInGE achieves state-of-the-art performance on CIFAR10 and ImageNet

Weaknesses:

* The main weakness that I see is that the work does not explain how the FLOPS and parameters of the pruned network are calculated. This is a major issue because the work does not directly compare to other state-of-the-art methods but merely quotes the numbers. Since the calculation of the FLOPS and parameters of a pruned network is non-trivial (see Appendix D of SOSP [43]) This can have a huge impact on the performance and could reduce the performance significantly.

* The second main weakness is that the work applies an iterative fine-tuning scheme but compares to single-shot pruning methods such as HAP, SOSP, OTO and SRR-GR. Since the paper shows in Table 2 that an iterative pruning scheme seems to improve the overall performance significantly, this would give SInGE an advantage and makes a direct comparison of the pruning criterions harder. 

* Further, the authors do not mention the computational effort to evaluate the criterion. Due to the Riemannian Integral this should be significantly higher compared to other gradient, magnitude or first-order based methods.

* Lastly a direct comparison to at least one competing method within the same pruning pipeline would provide the possibility to better disentangle the performance gains from the fine-tuning pipeline and criterion. 

questions: I quite like the idea of the paper and would be willing to increase my score if the authors can address my concerns. In addition to the concerns mentioned above, I have some more minor concerns:

Why do you not compare to the same methods on all datasets? It seems that some of the methods compared to on ImageNet also provide results for CIFAR10. Further, some recent structured pruning works such as Group Fisher Pruning [A] were not considered for comparison

Why did you choose to use the product of the norms of the gradient of the output of the NN and the weights? One could imagine other gradients i.e., the gradient of the loss.


[A] Liu, Liyang, et al. "Group fisher pruning for practical network compression." International Conference on Machine Learning. PMLR, 2021.

limitations: Yes
review_title: null

Meta-review: metareview: Novel pruning method based on integrated gradients. Reviewers agreed that the method is well-motivated and that the comparisons showcase the potential of this method. There are some concerns regarding fairness of the comparisons in terms of flops and parameter count. I believe some of the rebuttal answers from the authors address those concerns. I think this work is novel and interesting enough to be accepted at NeurIPS.
confidence: Certain
Decision: Accept