SInGE: Sparsity via Integrated Gradients Estimation of Neuron Relevance
Authors: Edouard YVINEC
Arnaud Dapogny
Matthieu Cord
Kevin Bailly
Venue: NeurIPS 2022 Accept (2022)
Keywords: pruning
compression
acceleration
deep learning
efficient inference

Abstract: The leap in performance in state-of-the-art computer vision methods is attributed to the development of deep neural networks. However it often comes at a computational price which may hinder their deployment. To alleviate this limitation, structured pruning is a well known technique which consists in removing channels, neurons or filters, and is commonly applied in order to produce more compact models. In most cases, the computations to remove are selected based on a relative importance criterion. At the same time, the need for explainable predictive models has risen tremendously and motivated the development of robust attribution methods that highlight the relative importance of pixels of an input image or feature map. In this work, we discuss the limitations of existing pruning heuristics, among which magnitude and gradient-based methods. We draw inspiration from attribution methods to design a novel integrated gradient pruning criterion, in which the relevance of each neuron is defined as the integral of the gradient variation on a path towards this neuron removal. Furthermore, We propose an entwined DNN pruning and fine-tuning flowchart to better preserve DNN accuracy while removing parameters. We show through extensive validation on several datasets, architectures as well as pruning scenarios that the proposed method, dubbed SInGE, significantly outperforms existing state-of-the-art DNN pruning methods.

SInGE: Sparsity via Integrated Gradients Estimation of Neuron Relevance
Edouard Yvinec Arnaud Dapogny², Matthieu Cord', Kevin Bailly Sorbonne Universitél, CNRS, ISIR, f-75005, 4 Place Jussieu 75005 Paris, France Datakalab², 114 boulevard Malesherbes, 75017 Paris, France ey@datakalab.com
Abstract
The lepp in performance in state-of-the-art computer vision methods is attributed the development of deep neural networks. However it often comes at a com- putational price which may hinder their deployment. To alleviate this limitation, structured pruning is a well known technique which consists removing channels, neurons or filters, and is commonly applied in order to produce more compact models. In most cases, the computations remove are selected based on a relative importance criterion. At the same time, the need for explainable predictive models has risen tremendously and motivated the development of robust attribution meth- ods that highlight the relative importance of pixels of an input image or feature map, this work, we discuss the limitations of existing pruning heuristics, among which magnitude and gradient-based methods. We draw inspiration from attribu- tion methods to design a novel integrated gradient pruning criterion, in which the relevance of each neuron is defined as the integral of the gradient variation path towards this neuron removal. Furthermore, we propose an entwined DNN pruning and fine-tuning flowchart to better preserve DNN accuracy while removing parameters. We show through extensive validation on several datasets, architectures well as pruning scenarios that the proposed method, dubbed SInGE, significantly outperforms existing state-of-the-ar DNN pruning methods.
Introduction
Deep neural networks (DNNs) ubiquitous in modern solutions for most computer vision problems such as image classification (T). object detection (2] and semantic segmentation (3). However, this performance was achieved at the price of high computational requirements and memory foot-print. As such, over-parameterization 4 is a common trait of well performing DNNs that may hinder their deployment on mobile and embedded devices. Furthermore, the case of deployment on cloud environment, latency and energy consumption are of paramount importance.
Consequently, compression and acceleration techniques aim at tackling the issue of DNN deployment. Among these methods, pruning approaches consist in removing individual weights (unstructured pruning) or entire computational blocks, such neurons channels or filters (stractured pruning) The sparsity induced by pruning reduces both the computational cost and the memory foot-print of neural networks. To do so, there exists wide variety of heuristies behind such pruning techniques. A few examples are: pruning at initialization (9. grid search [10] [I]. magnitude-baser
5
8
12 or redundancy based [3] approaches. Among such heuristics, magnitude-based pruning remains the favoured one [14 16]. It consists in defining metric to assess the relevance of
each neuron in the network, with the goal remove the least important ones while still preserving the predictive function as much as possible. An important limitation of these methods lies in the choice of this importance criterion: magnitude-bass criteria [17] do not take into account the whole computations performed in the network (e.g. within the other layers) and gradient-based [88 criteria
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

are intrinsically local within the neighborhood of a current value or set thereof: from this perspective,
setting value abruptly zero might break this locality property.
To craft a better criterion, we borrow ideas from the field of DNN attribution [9] These methods aim at understanding the behavior of neural network, L.e., in the case of computer vision model, by providing visual cues of the most relevant regions in image for the prediction of a network.
Tools developed to explain individual predictions are also often called visual explanation techniques
One example of such model is the Integrated Gradient method [24) that consists in defining the contribution of each input by the influence of marginal local changes in the input on
(20 21
23 24
the final prediction. This provides fine-gre fine-granned evaluation of the importance of each pixel of the
image (alternatively. of an intermediate feature map) in the final decision.
Our work is based on the idea that DNN pruning and attribution methods share an important notion, namely that they both rely on the definition of an importance metric to compare several variables of multidimensional prediction system: for pruning, to remove the least important DNN parameters, and, for attribution, to highlight the most important pixels. With this in mind, we propose to adapt the integrated gradient method for pruning purposes. Specifically, for each parameter (or set thereof, if consider structured sparsity), we define its importance as an integral of the product between the norm of this weight and its attribution along a path between this weight value and a baseline (zero) value. By doing so, we avoid pathological cases which less sophisticated gradient- based methods are subjected to such as weights that can be reduced but not zeroed-out without harming the accuracy of the model. Furthermore, we embed the proposed integrated gradient method within re-training framework to maximize the accuracy of the pruned DNN. We name our method SInGE, standing for Sparsity via Integrated Gradients Estimation of neuron relevance. short, the contributions of this paper are the following:
* We discuss the limitations of existing pruning heuristics, among which magnitude and gradi- ent based methods. We draw inspiration from attribution methods to design an integrated gradient criterion for estimating the importance of each DNN weight.
We entwine the updates of the importance measurement within the fine-tuning flowchart to preserve the better DNN accuracy while pruning.
The proposed approach, dubbed SInGE, achieves superior accuracy v.s. pruning ratio on every tested dataset and architecture, compared with recent state-of -ee--offtthtrrr approaches.
2
Related Work
2.1 Pruning
Pruning methods are often classified as either structured [25 26 27 10 28 (filters, channels or
neurons are removed) or unstructured [13] 29 30 31] (single scalar weight values are set to zero).
In practice, the former offers straightforward implementation for inference and immediate runtime benefits but at the price of lower number of parameters removed. For instance, in GDP (32), weights pruned with learned gate that zeroes-out some channels for easier pruning post-training. In CCP [33, sparsity is achieved by evaluating the inter-channel dependency and the joint impact of pruned and preserved channels on the final loss function. In HAP (34), authors replace less sensitive channels based on the trace of the Hessian of predictive function with respect to the weights. Generally speaking, these methods rely on defining a criterion to estimate and compare the importance of weights in the networks, and remove the least important such candidates. A limitation of these methods is that the proposed criteria are usually only relevant within the neighborhood of the current value for a considered weight, which can be problematic since abruptly setting this weight value might violate this locality principle. In this work, we address this limitation by borrowing ideas from the DNN attribution field.
2.2 Attribution
Autribution methods, also referred to as visual explanation methods (20 21 22 23 24 measure the importance of each input feature on the prediction. Their use was motivated by the need for
explainable models [19] as well as constrained learning (35). We can classify attribution as either
occlusion-based or gradient-based. The latter usually offers satisfactory results at a much lower

---

1n" w1
la
wi
Nww ww
V a wi
IV
IV powli
JV
V Pawl M w/
w
(a)
(b)
(c)
Figure I: Illustration of possible limitations of traditional pruning criteria for distinct cases and neurons (a.b.)). For a neuron n at layer we plot the weights norm ||μ="/") "u/] and corresponding gradients norm ||V wff' of different neurons (a, and c) for different powers of f e|0; 1 [orrespoodinn to a path towards zeroing out this neuron. Magnitude-basedd approaches (a) remove low magnitude neurons regardless of the sensitivity (gradient norm) of the predictive function by the intrinsic locality of gradient, and abruptly setting a neuron weights to zero may break this locality principle. Conversely, our integrated gradient-basee approach (c) will prune neuron although it initially has a high magnitude and gradient, integrating its gradient variations along a path down to zero magnitude.
these neurons. Gradient- based approaches (b) limited
computational cost. Considering that most DNNs for classification derivable, Grad-CAM (36] computes the gradients of the predictive function with respect to feature maps and weights these gradients by the features. The resulting importance maps then processed by ReLU function to extract the features that should be increased in order to increase the value the target class. Another gradient-based attribution of interest is Integrated-Gradients (24). In this work, Sundararajan al. propose to sum the gradients of the predictive function with respect to the feature maps over uniform path in the feature space between feature at hand and a reference point. The resulting attribution maps are usually sharper than maps obtained by Grad-CAM. In the proposed method, we draw inspiration from these methods as we propose to integrate the (local) evolution of the pruning criteria throughout a path going from the current weight value down to a baseline (zero) value. This way, we can smoothly bring the most irrelevant weights down zero even using intrinsically local criteria such as gradients or gradients per weight norm products.
3 Methodology
Let F : D + In be feed forward neural network defined over domain C. RT (c.g. the training dataset in most instances) and output space R" The operation performed by layer fi, for E (1....,L) L}, is defined by the corresponding weight tensor Wi E A 1xx where AA is simply convolutional layer. For the
in the case of fully-connected layers and R in the case of a sake of simplicity, we assume in what follows that A = IR, i.e. we remove neurons as represented by their weight vectors.
3.1 Simple baseline pruning criteria
One major component of pruning methods lies in the definition of an importance measurement for each neuron. The most straightforward such criterion based on the magnitude of the weight vectors. In such case, the importance criterion Czp based on the L norm || Ilp. is defined as:
CL. F,D) (I[WT) (p))s((.....))
(1)
where W is the th column of Wi. corresponding to the weight values associated with the nh neuron of layer ft. The transformation CLP operates layer per layer and independently of the rest of network F and the domain D. Intuitively, Cze assumes that the least important neurons are the smallest in norm because such neurons have lower impact on the predictions of F. Such a simple criterion however face limitations: consider for instance two first neurons (a) and (b) depicted in Figure [ by the purple stars in two-dimensional spaces as function of their magnitude and gradient norm respectively denoted ||W?'| || and VWW' for simplicity. However, we can clearly see how these local measurements provide wrong evaluation of the cost of pruning these neurons. In such a case,

---

pruning according to CL. will remove neuron (a) regardless of the fact that the predictive function F will be very sensitive to small modification of this neuron, as indicated by the large value of its gradients. This is however not the case with the gradient-based pruning criterion Cr. defined as:
C (W..,D)) H (1V F E D)ill) nE(I,.....
where Vwr E D) is the gradient of F with respect to W,, evaluated on X' sample from D. Intuitively, the latter measurement puts more emphasis on neurons that can be modified without directly altering the predictive function F. However, neuron may have low gradient norm and still strongly contribute to the predictive function, e.g. in the case where the weight large as in the case of neuron (b) on Figure To handle this, the norm gradient criterion Cyexpe straightforwardly combines the best of both worlds:
CLEXV= : (W, (1W7)l, V F(X E D))l)) nE(L.....))
(3)
3.2 Integrating gradients towards neuron removal
The importance criterion CLeXVE in Equation B faces another kind of limitation. due to the local nature of gradient information: if we consider neuron (b) on Figure T this neuron may initially (i.e. within a neighborhood of the purple star) have a low gradient norm or even low magnitude per gradient norm product. However, the gradient becomes larger This is due to the fact that Vwr F(X E 'D) only holds within a neighborhood of W? current value, and abruptly setting this neuron weights inspired from attribution methods, we propose a more global integrated gradient criterion. Formally, for neuron of layer, I, define Z, as the following integral:
bring this value down O.
may very well violate this locality principle. Thus,
I
V Nwwp F(X D)
(4)
Intuitively, measure the cost of progressively decaying the weights of neuron n and integrating the gradient norm throughout. In practice, we approximate I/' with the following Riemann integral:
S Σ
Cio+: (W,,FDD
I WT x P(X EDI1,
(5)
nE(1I....I
where eJ0; 1[ denotes an update rate parameter. Cig approximates (IT) Ine (........ up multiplicatiiee constant. Practically, this criterion measures the cost (as expressed by its gradients) of progressively bringing WT" down to 0 by S successive multiplication with the update rate parameter u: the higher μa, the more precise the integration at the expanse of increasing number of computations S. Also note that, similarly to Equation?? 2 can get rid of the weight magnitude term in Equation to obtain criterion CSGP, based on the sum of gradient norms. Explicitly, we get CSG (W,, F, D)
(E:. V F E D)I1,) ne(......) In the case depicted on Figure we will prune neuron
(c) as its gradient quickly diminishes as its magnitude becomes lower, despite high initial values for both magnitude and gradient. Thus, the proposed integrated gradients criterion Cigz allows to take into account both the magnitude of neuron's 's weights and the sensitivity of the predictive function w.r.t. small (local) variations of these weights. Furthermore, it measures the cost of removing this neuron by smoothly decaying it, re-estimating the gradient value at each step, hence preserving the local nature of gradients.
3.3 Entwining neuron pruning and fine-tuning
In order to preserve the accuracy of the network F, we alternate between removing the neurons and fine-tuning the pruned network using classical stochastic optimization updates. More specifically, given p global pruning target for the whole network F, we define layer-wise pruning objectives

---

(I....) (.......) such that Ei= x (Q(W)) ()) where ((W)) and (2() denote the number of parameters W and F, respetivelyy. Similarly [13]. tested several strategies for the per-layer pruning rates and kept their per-block strategy. Then, we sequentially prune each layer, starting from the first one, by first evaluating the relevance of each neuron (Cigr (Wi, F. DE parameter (4) in layer 1. We then rank the neurons by ascending numbers of importance and select the first, least important one. Notice at this point that if we remove neuron n we have to recompute the criterion CiG for all other neurons: in fact, during the first pass, the gradients we F(X E D) were computed with W o O and are bound to be altered with the removal of neuron ni, thus affecting the order of the n4 1 remaining neuron importance. Last but not least, once layer is pruned, perform O finetuning steps (which corresponds to O gradient descent optimization steps) to retain the network accuracy. This method, dubbed SInGE for Sparsity via Integrated Gradients Estimation of neuron importance, is summarized Algorithm
(with
Algorithm 1 SInGE Algorithm
Require: neural network F, hyper-parameters: O, and (Pr)EE().....)) and dataset D
for 2 E (1.... I.] do while pruning_rate(WW) (W)) Pt do evaluate +- CiG> (, F, D) find n arg MM] set W +
wait until reach the goal D magnitude estimation find the neuron to prune D- the pruning is performed here
for E (1, , 0} do
finetune the whole network F over batch from D end for end while end for
Empirically, as we show through a variety of experiments that the proposed integrated gradients- -base neuron pruning, along with efficient entwined fine-tuning allows to achieve superior accuracy vs. pruning rate trade-offs, as compared to existing methods.
4 Experiments
First, we introduce experimental setup, including the datasets and architectures as well as the implementation details to ensure reproducibility of the results. Second, we validate our approach on Cifar10 dataset by showing the interest of the proposed integrated gradient criterion, as well as the entwined pruning and fine-tuning scheme. We also compare our results with existing approaches on Cifarl0. Last but not least, demonstrate the superior performance of our SInGE method on several architectures on ImageNet compared with state-of-the-ee approaches for both structured and unstructured pruning.
4.1 Experimental setup
Datasets and Architectures: we evaluate our models on the two de facto standard datasets for architecture compression, i.e. Cifar10 (37) and ImageNet (38). We use the standard evaluation metrics for pruning, í.e. the % of removed parameters as well as the % of removed Floating Poottngg-point operations (FLOPs). apply our approach on ResNet 56 ([] with 852K parameters and accuracies 93.46%) on Ci and ResNet 50 ([I with 25M parameters and 76.17 accuracy on ImageNet), as well as MobileNet
(39 backbone on ImageNet with 71.80 accuracy and 3.4M parameters.
Implementation Details: our implementations based on tensorflow and numpy python libraries. We measured the different pruning criteria using random batches X' of 64 training images for both Cifar10 and ImageNet and fine-tuned the pruned models with batches of size 128 and 64 for Cifar10 and ImageNet, respectively. The number of optimization steps varies from 1k to 5k on Cifar10 and from 5k to 50k on ImageNet, while the original models were trained with batches size 128 and stochastic gradient descent of 78k and 2m steps Cifar 10 and ImageNet, respectively. All experiments were performed on NVidia V100 GPU. We evaluate our approach both for structured and unstructured pruning: for the former,
μ 0.9 and 0.95 for ImageNet and Cifar10.
5

---

Bager:
laper 1=25
136-
Figure 2: Visualization, for 5 random and different layers of ResNet 56 trained Cifarl0, of the evolution V wp F(X& E D)I, (y axis) as magnitude Wa"W"" Ipp (x axis) brought to O.
Table I: Pruning and accuracy performance of the different pruning criterion ResNet 56 trained on Cifar10. without fine-tuning. We also report the standard deviation over multiple runs.
Pruning target (% FLOPS parameters) pruning criterion
top-1 1 accuracy
0.0
baseline
93.46
magnitude Cz
42.01 0.41
magnitude Cza
42.35 0.38
gradients Cęz
77.68 0.52
73.03 75.00
magnitude x grad CLax
92.36 0.17
integrated gradients Csc
93.01 0.07
integrated magnitude x grad CIg
93.23 0.23
magnitude Cz
19.14 0.82
magnitude Cy²
19.13 0.09
gradients Cgz
28.31 1.75
86.46 85.00
magnitude x grad CLax
90.28 + 0.18
integrated gradients Osc²
91.90 + 0.15
integrated magnitude x grad CIgz
92.80 + 0.30
magnitude Cg=
10.00 + 1<
magnitude Cz²
10.00 + 1<
gradients Cea
10.00 = 1<
/0 90.00
magnitude x grad C1axv
10.00 + 1<
integrated gradients Cscz
75.38 + 1.28
integrated magnitude grad CIG2
84.54 + 0.91
respectively. For unstructured pruning, use f 0.8 for ImageNet. In all experiments we performed hatah-normalization folding from (40] and measured the pruning ratio using the same metric as SOSP [1
4.2 Empirical Validation
Pruning Criterion Validation: In Figure we illustrate the evolution of Veewp F(X' E D)
(y axis) as the magnitude [|""l (x axis) is brought This observation confirms the limitations of gradient-based criteria pinpointed in Section 3.2 as the neuron magnitude is progressively decayed, the gradient norm (e.g. yellow curves on both plots, as well as red on the left and blue one on the right plot) for these neurons rise significantly. making these neurons bad choices for removal despite low initial gradient values. This empirical observation suggests that intuition behind the proposed criterion CIgP is valid. Table [ draws comparison between the different criteria introduced in Section ( applied to prune ResNet 56 on 10 More specifically, given a percentage removed operations (or equivalently, percentage of removed parameters), compare the resulting accuracy

---

Table 2: Comparison between post-pruning and entwined pruning and fine-tuning ResNet 56 Cifar10.
% Pruning target (% FLOPS parameters)
fine-tuning
# steps
top-1 accuracy
post-pruning
1000
92.59
entwined
1000
93.18
post-pruning
2000
92.66
86.46 / /5.40
entwined 2000
93.25
post-pruning 5000
93.13
entwined
5000
93.31
post-pruning 1000
77.2
entwined 1000
85.38
post-pruning 2000
80.89
88.10/90.00
entwined
2000
87.52
post-pruning 5000
86.39
entwined
5000
90.02
without fine-tuning. We observe similar trends for the pruning targets: First, using euclidean norm performs slightly better than L1: thus, we set p = 2 in what follows. Second, using gradient instead of magnitude- bbased criterion allows to significantly improve the accuracy given a pruning target. Third, the magnitude x gradient criterion Cr.2x v2 allows to better preserve the accuracy by combining the best of both worlds: for instance, with 85% parameters removed, applying C x va increases the accuracy by 51.97 points compared with Cçz. However, those simple criteria face limitations particularly in the high pruning rate regime (90% parameters removed), where the accuracy of the pruned network falls to chance level. Conversely, the proposed integrated gradient-based criteria Csc= and, a fortiori Ciga allows to preserve high accuracies in such a case, Overall, CjOz the best criterion, allowing to preserve near full accuracy with both 85% and 85% removed parameters, and 85.38% accuracy with 90% removed parameters, outperforming the second best method by 8.18 points. For this reason, we will use this criterion (Cioz) ) in the following experiments. As the pruning rate increases, the cost removing a neuron increases and any ill-advised selection of neuron to remove has growing impact the accuracy. Consequently, the standard deviation increases the pruning rate increases this is due to the network expressivity going down.
Fine-tuning Protocol Validation: Table2validate validates our entwined pruning and fine-tuning approach with different pruning targets and number of fine-tuning steps. Specifically. for a given total number of fine-tumi aieettunngg step. we either perform all these steps at once post-pruning or alternatively spread them evenly after pruning each layer in an entwined fashion, as described in Section 3.3 First, we observe that simply increasing the number of fine-tuning steps vastly improves the accuracy of the pruned model, particularly in the high % removed parameters regime. Moreover, entwining pruning and fine-tuning performs consistently better than fine tuning after pruning. This suggests that recovering the accuracy is an easier task when performed frequently over small modifications rather than once over a significant modification.
our approach relies on removing the least important neurons, as indicated by criterion Cig². We compare with similar recent approaches such LP (42) and DPF (29 as well as other heuristics such as training neural networks in order to separate neurons for easier pruning post-training (HAP (34,, GDP (32) or similarity removal
Comparison with state-of-the-art approaches on Cifar10::
(RED [13] or LDI [T]). We report the results in Table for two accuracy set-ups: lossless pruning (accuracy identical to the baseline model) and lossy pruning (~ 2 points of accuracy drop). The proposed SInGE method significantly outperforms other existing methods by achieving 1.3% higher pruning rate in the lossy setup and considerable 8. 8.1% improvement in lossless pruning rate. As such, it bridges the gap with unstructured methods such as I.P 42 and DPF 29]. This demonstrates the quality of the proposed method.
7

---

Table State-of-the-art pruming methods performance ResNet 56 Cifar10.
topl accuracy pruning method
structured
% parameters removed
RED
13
85.0
LP
84.0
LP
42
92.6
91.5 = 0.1
LDI
88
DPF 29
X
90.0
HAP
90.0
SInGE (ours)
91.3 + 0.27
GDP
/
65.6
93.5 - 0.1
НAP
76.2
SInGE (ours)
84.3 = 0.71
Table Comparison between existing structured pruning performance on ResNet on ImageNet. In both low (< 50% parameters removed) and high (> 50%) pruning regimes, SInGE achieves remarkable results.
Method
% params rm
FLOPS rm
accuracy
baseline
0.00
0.00
76.15
Hrank (CVPR 2020)
48
36.67
43.77
74.98
RED (NeurIPS 2021)
13
39.6
42.7
76.1
HAP (WACV 2022)
34]
44.59
33.82
75.12
SRR-GR (CVPR 2021)
[28]
45
75.76
SOSP (ICLR 2021)
[1
49
45
75.21
SRR-GR (CVPR 2021)
[88]
55
75.11
SInGE
50.80 + 0.02
57.35 + 0.11
76.05 + 0.07
RED (NeurIPS 2021)
13
54.7
55.0
71.1
SOSP (ICLR 2021)
41
54
51
74.4
GDP (ICCV 2021)
32
55
73.6
HAP (WACV 2022)
34
65.26
59.56
74.0
OTO (NeurIPS 2021)
45
64.1
65.2
73.3
GFP (ICML 2021)
449
65.0
73.94
SInGE
63.78 + 0.01
65.96 + 0.21
74.7 + 0.31
4.3 Performance on ImageNet
Structured Pruning: Table Summarize results obtained by current state-of-the-art approaches in structured pruning. For clarity we divided these results the low (<50% parameters removed, where the methods are often lossless) and high pruning regime (>50% parameters removed with significant accuracy loss). In the low pruning regime, the proposed SInGE method manages to remove slightly more than 50% parameters (57.35% FLOPS) with nearly no accuracy loss, which significantly improves over existing approaches. Second, the high pruning regime, other methods such as OTO [43) and HAP (34) recently improved the pruning rates by more than 10 points over other techniques such as GDP 32 and SOSP (4I). Nonetheless, SInGE is competitive with these methods and achieve higher FLOP reduction while maintaining higher accuracy.
We also evaluated the proposed method on the more compact (thus generally harder prune) MobileNet V2 architecture. Results and comparison with existing approaches are shown in Table 5 We consider three pruning goals of 30%, 40% and z 50% parameters removed. First, with near lossless pruning, we achieve results that are comparable to ManiDP-A 44 and Adapt-DCP [45] with a marginal improvement in accuracy. Second, when targeting 40% parameters removed we improve by 0.89% the accuracy with 2.25% less parameters removed as compared to MDP (46). Finally, in the higher pruning rates, we improve by 0.25% the accuracy with marginally more parameters pruned than Accs [47.

---

Table 5: Comparison with existing structured pruning methods MobileNet V2 backbone for ImageNet.
goal
Method
% params
% FLOPS rm
accuracy
baseline
0.00
0.00
71.80
CBS (arxiv 2022) [50]
30.00
71.48
Adapt-DCP (TPAMI 202T)
45
35.01
30.67
71.4
30%
ManiDP-A (CVPR 2021)
37.2
71.6
SInGE
30.96
31.54
71.67 0.06
CBS (arxiv 2022)
[50]
40.00
69.37
40%
MDP (CVPR 2020)
46
43.15
69.58
SInGE
40.90
42.30
70.47 0.09
CBS (arxiv 2022)
[5
50.00
62.96
Adapt-DCP (TPAMI 2021)
45.0
64.13
ManiDP-A (CVPR 2021)
48.8
69.62
50%
Accs (arxiv 2021)
[7
50.00
69.76
GFP (ICML 2021)
49
50.0
69.16
SInGE
50.13
48.90
70.01 + 0.22
Table 6: Comparison with existing unstructured pruning techniques on ResNet 50 ImageNet.
Method
% params rm
% FLOPS rm
topl accuracy
DS (NeurIPS 2021)
[1]]
80.47
72.13
76.15
GMP (arxiv 2019)
52
80.08
76.15
STR (ICML 2020)
53
79.69
81.17
76.00
RigL (ICML 2020)
[54]
80.08
58.92
75.00
SInGE
80.00
82.21
75.12
SInGE
90.00
86.96
73.77
Unstructured Pruning: While being harder to leverage, unstructured pruning usually enables significantly higher pruning rates. Table several state-of-the-ard pruning methods evaluated on ResNet 50. We observe common threshold in performance around 80% parameters and FLOPS removed among state-of-the-arr techniques. However, the proposed SInGE method manages to achieve very good accuracy of 73.77% while breaking the barrier of pruning performance 90% parameters removed and almost 87% FLOPs removed. These results in addition to the previous excellent results obtained structured pruning confirm the versatility of the proposed criterion and method for both structured and unstructured pruning.
5 Conclusion
In this paper, we pinpointed some limitations of some classical pruning criteria for assessing neuron importance prior removing them. In particular, showed that magnitude- approaches did not consider the sensitivity of the predictive function w.r.t. this neuron weights, and that gradient-baser approaches were limited to the locality of the measurements. We drew inspiration on recent DNN attribution techniques to design a novel integrated gradients criterion, that consists in measuring the integral of the gradient variation a path towards removing each individual neuron. Furthermore, we proposed to entwine this criterion within the fine-tuning steps. We showed through extensive validation that the proposed method, dubbed SInGE, achieved superior accuracy pruning ratio as compared with existing approaches on a variety of benchmarks, including several datasets, architectures, and pruning scenarios.
Future work will involve introducing stochasticity in the model weights, similarly to (55), order to smooth the decision function and ultimately the neuron relevance criterion. Lasty, we will combine our approach with existing similarity- pruning methods as well with other DNN acceleration techniques, e,g. tensor decomposition quantization techniques.

---

Acknowledgments
This work has been supported by the french National Association for Research and Technology (ANRT), the company Datakalab (CIFRE convention C20/1396) and by the French National Agency (ANR) (FacIL, project ANR-I7-CE33-00022, This work was granted access the HPC resources of IDRIS under the allocation 2022-AD01 1013384 made by GENCI.
References
[1] Kaiming He, Xiangyu Zhang, et al. Deep residual learning for image recognition. CVPR, pages 770-778, 2016.
[2] Ross Girshick. Fast r-cmn. ICCV, pages 1440 1448, 2015.
[3] Liang-Chich Chen et al. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. TPAMI, pages 834-848, 2017.
[4] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-
115, 2021.
[5] Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. NeurlPS, 2, 1989.
[6] Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain surgeon. NeurlPS, 5, 1992.
[7] Suraj Srinivas and R Venkatesh Babu. Data-free parameter pruning for deep neural networks. BMVC, 2015.
[8] Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. ICLR, 2016.
[9] Huan Wang, Can Qin, Yulun Zhang, and Yun Fu. Emerging paradigms of neural network pruning. arxiv preprint arXiv:2103. 103.06600, 2021.
[10] Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep
neural network compression. ICCV, pages 5058-5066 2017.
[11] Ruichi Yu, Ang Li, Chun-Fu Chen, Jui-Hsin Lai, Vlad Morariu, Xintong Han, Mingfei Gao, Ching- Lin, and Larry S Davis. Nisp: Pruning networks using neuron importance score propagation. CVPR, pages 9194-9203, 2018.
[12] Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv: 1eee::111.6646 2016.
[13] Edouard Yvinec, Arnaud Dapogny, Matthieu Cord, and Kevin Bailly. Red: Looking for redundancies for lata-frrestruuctured compression of deep neural networks. Advances in Neural Information Processing Systems, 34, 2021.
[14] Kun Han, Yuxuan Wang, DeLiang Wang, William S Woods, Ivo Merks, and Tao Zhang Learning spectral mapping for speech dereverberation and denoising. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 23(6):92-9997 2015.
[15] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. ICLR, 2018.
[16] Pavlo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. CVPR, pages 11264-12722 2019.
[17] Guiying Li, Chao Qian, Chunhui Jiang, Xiaofen Lu, and Ke Tang. Optimization based layer-wise magnitude-basee pruning for dnn compression. In IJCAI, pages 2383-2389, 2018.
[18] Congcong Liu and Huaming Wu. Channel pruning based mean gradient for accelerating convolutional neural networks. Signal Processing. 156:84-91, 2019.
[19] Wojeiech Samek, Thomas Wiegand, and Klaus-Robert Müller. Explainable artificial intel- ligence: Understanding, visualizing and interpreting deep learning models. arXiv preprint 1rX.::1708.08296 2017.
[20] David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus-Robert Müller. How to explain individual classification decisions. JMLR, 11:1803-1831, 2010.
10

---

[21] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2013.
[22] Wojciech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Klaus- Robert Mülllr, Evaluating the visualization of what a deep neural network has learned. IEEE transactions on neural networks and learning systems, 28( (3(11))20) 1):2660-2673,, 2016.
[23] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. ICML, pages 3145-3153, 2017.
[24] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. ICML. pages 33 19-3328, 2017.
[25] Lucas Liebenwein, Cenk Baykal, et al. Provable filter pruning for efficient neural networks. ICLR, 2020.
[26] Hao Li et al. Pruning filters for efficient convnets. ICIR, 2017.
[27] Yang He, Guoliang Kang, et al. Soft filter pruning for accelerating deep convolutional neural networks. IJCAI, pages 2234 2240,, 2018.
[28] Zi Wang, Chengeheng Li, and Xiangyang Wang. Convolutional neural network pruning with structural redundancy reduction. CVPR, pages 14913-14922, 2021.
[29] Tao Lin, Sebastian U Stich, et al. Dynamic model pruning with feedback. ICLR, 2020.
[30] Sejun Park, Jacho Lee, et al. Lookahead: a far-sighted alternative of magnitudc-based pruning. ICLR, 2020.
[31] Namhoon Lee, Thalaiyasingam Ajanthan, et al. A signal propagation perspective for pruning neural networks at initialization. ICLR, 2020.
[32] Yi Guo. Huan Yuan, Jianchao Tan, Zhangyang Wang, Sen Yang, and Ji Liu. Gdp: Stabilized neural network pruning via gates with differentiable polarization. ICCV, pages 5239-52500 2021.
[33] Hanyu Peng, Jiaxiang Wu, Shifeng Chen, and Junzhou Huang. Collaborative channel pruning for deep networks. ICML, pages 5113-5122, 2019.
[34] Shixing Yu, Zhewei Yao, Amir Gholami, Zhen Dong, Sehoon Kim, Michael W Mahoney, and Kurt Keutzer. Hessian-aware pruning and optimal neural implant. WACV, pages 3880-3891, 2022.
[35] Jules Bonnard, Arnaud Dapogny, Ferdinand Dhombres, and Kévin Bailly. Privileged attribution constrained deep networks for facial expression recognition. arXiv preprint arXiv:2203.12900 12905, 2022.
[36] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. ICCV, pages 618-626, 2017.
[37] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009.
[38] J. Deng, W. Dong, et al. ImageNet: A Large-Scale Hierarchical Image Database. CVPR, 2009.
[39] Mark Sandler, Andrew Howard, et al. Mobilenetv2: Inverted residuals and linear bottlenecks. CVPR, pages 4520- 2018
[40] Edouard Yvinec, Arnaud Dapogny, and Kevin Bailly. To fold or fold: necessary and sufficient condition on batch-normalizatioon layers folding. IJCAI, 2022.
[41] Manuel Nonnenmacher, Thomas Pfeil, Ingo Steinwart, and David Reeb. Sosp: Efficiently capturing global correlations by second-order structured pruning. ICLR, 2021.
[42] Lucas Liebenwein, Cenk Baykal, Brandon Carter, David Gifford, and Daniela Rus. Lost in pruning: The effects pruming neural networks beyond test accuracy. Proceedings of Machine Learning and Systems, 3:93-138, 2021.
[43] Tianyi Chen, Bo Ji, Tianyu Ding, Biyi Fang, Guanyi Wang, Zhihui Zhu, Luming Liang, Yixin Shi, Sheng Yi, and Xiao Tu. Only train once: A one-shot neural network training and pruning framework. NearIPS, 34, 2021.

---

[44] Yehui Tang. Yunhe Wang, Yixing Xu, Yiping Deng, Chao Xu, Dacheng Tao, and Chang Xu. Manifold regularized dynamic network pruning. arXiv preprint arXiv:2103.0586,, 2021.
[45] Jing Liu, Bohan Zhuang, Zhuangwei Zhuang, Yong Guo, Junzhou Huang, Jinhui Zhu, and Mingkui Tan. Discrinination-aware network pruning for deep model compression. IEEE Transactions Pattern Analysis and Machine Intelligence, 2021.
[46] Jinyang Guo, Wanli Ouyang, and Dong Xu. Multi-dimennional pruning: A unified framework for model compression. CVPR, pages 1508-1517, 2020.
[47] Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu, and Paulius Micikevicius. Accelerating sparse deep neural networks. arXiv preprint arXiv:2104.0833 083788 2021.
[48] Mingbao Lin, Rongrong Ji, al. Hrank: Filter pruning using high-rank feature map. CVPR, pages 1529-1538, 2020.
[49] Liyang Liu, Shilong Zhang, Zhanghui Kuang, Aojun Zhou, Jing-Hao Xue, Xinjiang Wang. Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. Group fisher pruning for practical network compression. In ICML., pages 7021-7032. PMLR, 2021.
[50] Xin Yu, Thiago Serra, Shandian Zhe, and Srikumar Ramalingam. The combinatorial brain surgeon: Pruning weights that cancel one another in neural networks. arXiv preprint arXiv:2203.0444 04066,, 2022.
[51] Wei Sun, Aojun Zhou. Sander Stuijk, Rob Wijnhoven, Andrew Nelson, Henk Corporaal, et al. Dominoscarch: Find layer-wise fine-grained n: M sparse schemes from dense neural networks. NeurlPS, 34, 2021.
[52] Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv: 1902.09944 2019.
[53] Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade, and Ali Farhadi. Soft threshold weight reparameterization for learnable sparsity. In ICML, pages 5544-5555, PMLR, 2020.
[54] Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. ICML, pages 2943-2952, 2952, 2020.
[55] Kirill Bykov, Anna Hedström, Shinichi Nakajima, and Marina M-C Höhne. Noiseg- rad-ennancing explanations by introducing stochasticity model weights. In AAAI, 2022.
12


Reviews:
[{"review_id": "vYLO5Hgf7KG", "reviewer": "Reviewer_kWFQ", "summary": "This paper proposes a new neuron pruning  method based on the Integrated Gradients (IG) attribution algorithm. The paper claims that IG-based pruning is more optimal than magnitude and local gradient-based methods since it accounts for the gradient effects globally along the path from 0-valued baseline weight to given input weight in each layer l. The paper performs impressive number of experiments for a number of structured and unstructured pruning techniques from the state of the art literature and shows that IG-based pruning outperforms those methods based on two evaluation metrics (% removed parameters and % removed floating point operators).\nIn addition to that the paper also proposes entwined pruning and fine-tuning procedure and shows that it help to obtain better accuracy while performing the pruning. \nThe experiments in the paper are conducted for ImageNet and Mobilenet V2 backbone for ImageNet and  CIFAR10 dataset.\n", "questions": "\n+ From Table 1 it is unclear what the authors mean by integrated gradients C_{SG}^2 and integrated magnitude x grad C_{IG}^2. It would be good if they could elaborate it further and also describe which of those two they used for the results in tables 3, 4, 5 and 6.\n+ I wonder if Noisegrad paper which introduces stochasticity to model weights could also be relevant to this paper.\nhttps://arxiv.org/pdf/2106.10185.pdf\n", "limitations": "\nI do not see any potential negative societal impact in this work", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n+ Overall the idea of the paper is clearly described, related work is properly cited, experimental setup and results are properly discussed\n+ The paper shows strong empirical evidence that IG-based pruning outperforms magnitude, gradient-based approaches, as well as a number of other state of the art approaches from the neuron pruning literature. \n+ The paper also shows that entwined fine-tuning helps to improve model accuracy regardless of pruning technique using over 80% pruning target.\n\n**Weaknesses**\n\n+ I think that in general the readability (fluency) of the paper can be further improved. Specifically, I felt that in the notation and description of the approaches, there is some level of expectations that the readers know the notations of structured pruning techniques and integrated gradients. For example, neither Figure 1 nor the context talking about Figure 1 describes what $\\mu$ and $\\mu^2$ are.\n+ Figure 1 and its descriptions are a bit confusing. Originally, I was thinking that a, b and c are weight matrices but later the paper refers to a and b as two different weights. It would be good to improve the description of Figure 1. \n+ Given that entwined fine-tuning is one of the important contributions of the paper it would be good to describe it in detail. It would be good to make it clear what fine-tuning over O steps mean and what we mean by fine-tuning the network F over a batch from D. It seems that the authors make an assumption that the readers are familiar with the specific fine-tuning approach that they are describing.\nAt this point from the given description it is hard to tell the amount of novelty in the entwined fine-tuning.\n+ Table 1: It would be great if the authors explain relatively large std in integrated magnitude x grad C_{IG}^2 method\n\n**Minor Comments**\n\n+ On page 9 a notation, $N_n$ but I don’t see a description of notation $N$.\n+ Table 6 has 2 SInGE rows but it is not clear what are the differences between those two SInGEs\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "CYiZLyZ1bk", "reviewer": "Reviewer_BMyz", "summary": "This paper proposes a novel pruning method based on an integrated gradient pruning criterion, which combines a magnitude and gradient based criterion and integrates the product over the path of the neuron removal. \n\nIn addition, the work proposes a fine-tuning flowchart to improve the performance of the fine-tuned network.  This strategy is an iterative layer-wise approach which sets the pruning ratios for each block separately. \n\nLastly, a comparison of SInGE on ResNet-56 on Cifar10 and ResNet-50 and MobileNet-V2 on ImageNet shows state-of-the-art performance. \n", "questions": "I quite like the idea of the paper and would be willing to increase my score if the authors can address my concerns. In addition to the concerns mentioned above, I have some more minor concerns:\n\nWhy do you not compare to the same methods on all datasets? It seems that some of the methods compared to on ImageNet also provide results for CIFAR10. Further, some recent structured pruning works such as Group Fisher Pruning [A] were not considered for comparison\n\nWhy did you choose to use the product of the norms of the gradient of the output of the NN and the weights? One could imagine other gradients i.e., the gradient of the loss.\n\n\n[A] Liu, Liyang, et al. \"Group fisher pruning for practical network compression.\" International Conference on Machine Learning. PMLR, 2021.\n", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n* I like the idea of an integration criterion and according to Table 1 it seams that this is the main driver for the superior performance of SInGE. To the best of my knowledge this is the first method to use an integration criterion for (structured) pruning. \n\n* The Integration criterion is well motivated, and the ablations support the design choices made by the authors.\n\n* SInGE achieves state-of-the-art performance on CIFAR10 and ImageNet\n\nWeaknesses:\n\n* The main weakness that I see is that the work does not explain how the FLOPS and parameters of the pruned network are calculated. This is a major issue because the work does not directly compare to other state-of-the-art methods but merely quotes the numbers. Since the calculation of the FLOPS and parameters of a pruned network is non-trivial (see Appendix D of SOSP [43]) This can have a huge impact on the performance and could reduce the performance significantly.\n\n* The second main weakness is that the work applies an iterative fine-tuning scheme but compares to single-shot pruning methods such as HAP, SOSP, OTO and SRR-GR. Since the paper shows in Table 2 that an iterative pruning scheme seems to improve the overall performance significantly, this would give SInGE an advantage and makes a direct comparison of the pruning criterions harder. \n\n* Further, the authors do not mention the computational effort to evaluate the criterion. Due to the Riemannian Integral this should be significantly higher compared to other gradient, magnitude or first-order based methods.\n\n* Lastly a direct comparison to at least one competing method within the same pruning pipeline would provide the possibility to better disentangle the performance gains from the fine-tuning pipeline and criterion. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4xlLMSqj1Lm", "reviewer": "Reviewer_kiMH", "summary": "Inspired by the integrated gradient method in attribution methods, the authors propose a new importance metric for pruning: an integral of the product between the norm of the parameter weight and its attribution along a path between this weight value and a baseline (zero) value. This metric captures more global view than previous magnitude-based and gradient-based pruning methods. Then, the authors devise the so-called SinGE method to intertwine the pruning with the network fine-tuning. Empirical studies are conducted to compare SinGE with both structured and unstructured pruning methods on Cifar 10 and ImageNet.", "questions": "1. As far as I undertand, the integration evaluation will take much time, so do the authors think that adding an experiment of the time efficiency necessary?\n2. There are a few typos, for example, at the range of $\\mu$ at line 138, the spellings of trade-offs at line 167, and the incompleteness of parentheses at line 181.\n3. As I suppose the bolded numbers in the tables stand for the best results, some of those in Table 3 and 4 are not correctly labeled, which is a bit misleading.\n4. The top-1 accuracy of integrated magnitude method without fine-tuning and with 1000-step entwined fine-tuning seems to be the same (both reported as 85.38). I’m wondering why there is no increase in the accuracy.\n5. May I ask what does the notation “1<” stand for in Table 1?\n6. As the performance of SinGE is very close to its counterpart in many cases, could the authors please provide the error bars for the results reported in Table 2-6?\n7. May I ask whether the results in Table 3 use fine-tuning or not? If yes, I’m wondering whether the same fine-tuning strategy (i.e., entwined) is adopted for all the candidates?\n8. In Table 6, only the accuracy of SInGE when 90% of parameters removed is provided but not the other methods. It would be appreciated if the authors could provide the performance of other methods at this setting to support the superiority of SInGE.", "limitations": "The authors have included adequate limitations.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\n1. Motivation is very clear and the idea of using the approximated integration as the criterion of pruning is a good way to measure the influence of forcing a parameter to 0.\n2. This paper has clear writing and easy to follow.\n\n## Weaknesses\n\n1. Figure 1 is not effective and clear enough to facilitate the readers’ understanding on the arguments made in Sec 3.1. One reason is that the notation $\\mu$ is not introduced before its appearance. Also, the claims at line 129 – 133 about the effects of the change in values are not clearly depicted in Figure 1.\n\n2. The claim that SInGE significantly outperforms existing state-of-the-art DNN pruning methods seems to be a bit inappropriate as it’s sometimes inconsistent with what demonstrates in the empirical studies. For example, in Table 5, the accuracy of SinGE is nearly the same as those of Adapt-DCP, ManiDP-A and MDP. However, the latter remove significantly more parameters or FLOPS than SinGE does.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept