SOSP: Efficiently Capturing Global Correlations by Second-Order Structured Pruning
Authors: Manuel Nonnenmacher
Thomas Pfeil
Ingo Steinwart
David Reeb
Venue: NeurIPS 2021 Submitted (2021)
Keywords: Structured Pruning
Network Compression
Neural Architecture Search
Hessian Approximation
Saliency-based Pruning
Deep Learning
Computer Vision

Abstract: Pruning neural networks reduces inference time and memory cost, as well as accelerates training when done at initialization. On standard hardware, these benefits will be especially prominent if coarse-grained structures, like feature maps, are pruned. We devise global saliency-based methods for second-order structured pruning (SOSP) which include correlations among structures, whereas highest efficiency is achieved by saliency approximations using fast Hessian-vector products. We achieve state-of-the-art results for various object classification benchmarks, especially for large pruning rates highly relevant for resource-constrained applications. We showcase that our approach scales to large-scale vision tasks, even though it captures correlations across all layers of the network. Further, we highlight two outstanding features of our methods. First, to reduce training costs our pruning objectives can also be applied at initialization with no or only minor degradation in accuracy compared to pruning after pre-training. Second, our structured pruning methods allow to reveal architectural bottlenecks, which we remove to further increase the accuracy of the networks.

SOSP: Efficiently Capturing Global Correlations by Second-Order Structured Pruning
Anonymous Author(s)
Affiliation
Address
email
Abstract
Pruning neural networks reduces inference time and memory cost, as well accel- erates training when done at initialization. On standard hardware, these benefits will especially prominent coarse-grained structures, like feature maps, are pruned. We devise global saliency-based methods for second-order structured prun- ing (SOSP) which include correlations among structures, whereas highest efficiency is achieved by saliency approximations using fast Hessian-vectu products. We achieve state-of-the-art results for various object classification benchmarks, espe- cially for large pruning rates highly relevant for resource-constraing constrained applications. We showcase that our approach scales to large-scale vision tasks, even though it captures correlations across all layers of the network. Further, highlight two outstanding features of our methods. First, to reduce training costs our pruning objectives also be applied initialization with only minor degradation in accuracy compared to pruning after pretraining. Second, our structured pruning methods allow reveal architectural bottlenecks, which remove further increase the accuracy of the networks.
3
11
13
15
Introduction
18
17 Deep neural networks have consistently grown in size over the last years with increasing performance. However, this increase in size leads to slower inference, higher computational requirements and higher cost. To reduce the size of the networks without affecting their performance, a large number of pruning algorithms have been proposed LeCun al. 1990; Hassibi al. [993 Reed 1993
19
20
21
Han et al. 2015; Blalock et al. 2020). Pruning can either be unstructured, i.e. removing individual weights, or structured, i.e. removing entire substructures like nodes or channels. Single-shot pruning methods, as investigated in this work, usually consist of three steps: 1) training, 2) pruning, 3) another training step often referred to as fine-tuning.
22
23
24
25
Unstructured pruning can significantly reduce the number of parameters only little loss in the accuracy, but the resulting networks often show only a marginal improvement in training and inference time, unless specialized hardware is used (He et al., (017). In contrast, struc- tured pruning directly reduce inference time and training time when applied initialization
neural network with
27
29 Lee et al.) 2018). To exploit these advantages, in this work, we focus on structured pruning.
Most sensitivity-basee pruning methods such OBD (e.g., LeCun et al., 1990) C-OBD (Wang 31 et al. 2019a) evaluate the effect removing a single weight or structure on the loss of the neural network, while neglecting possible correlations between different structures and within the structures
33 themselves. This can significantly harm the estimation of the sensitivities. We take these correlations into account by applying efficient second-order estimations that not only consider the diagonal terms
35 of the Hessian, but also all off-diagonal terms.
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

36
Global pruning removes structure by structure from all available structures of network until predefined percentage pruned structures is reached. Recent examples for global structured pruming methods are NN Slimming (Liu et al. 2017). C-OBD and FigenGamage (Wang et al. 2019a). Local pruning, on the other hand, first subdivides all global structures into subsets (e.g. layers) and removes percentage of structures of each subset. Recent examples for local pruning methods are HRank [Lin et al. 2019), CCP (Peng et al. 2019). FPGM (He et al., 2019) and Variational Pruning (Zhao) al. 2019). Most local pruning schemes use predefined layer-wise pruning ratio, which fixes the percentage of structures removed per layer. While this approach prevents the layers from collapsing, it also reduces some of the degrees of freedom, since some layers may be less important than others.
37
38
39
40
41
43
45
Our main goal in this work devise simple and efficient second-order pruning method, which considers all global correlations for structured sensitivity pruning. In addition. we want to highlight the benefits that such methods may have over other structured global and local pruning schemes.
48
47
48
Our contributions are as follows:
49
We introduce two novel saliency- based methods for second-order structured pruning (SOSP), which consider all correlations across structures and layers. We benchmark our SOSP methods against a variety of state-of-the-art pruning methods on several networks and datasets and achieve comparable or better results at low computational costs.
51
53
We show that our pruning methods can also be applied initialization almost matching the performance of pruning after training and significantly reducing the cost of network training.
54
55
We exploit the structure of the pruning masks found by our SOSP methods to remove architectural bottlenecks, which further improves the performance of the pruned networks. In this work, we consider layers with disproportionally low pruning ratios architectural bottlenecks.
56
57
58
59
PyTorch code implementing our method is attached in the Supplementary Material and we will publish the code upon acceptance of this manuscript.
80
2
SOSP: Second-order structured pruning
61
A neural network (NN) maps an input Rd to an output fo(z) R,,, where O R are its P parameters. NN training proceeds, after random initialization = 0g of the weights, by mini-
63
batch stochastic gradient descent on the empirical loss L()) == ~ Σ é (ff((n)) ), yn), given the training dataset ((111 3/1 y1....... y.)). In the classification case, y/ E {1..... D} is discrete ground-truth label and ((fo(r)) == log a (fo (z)), the cross-entropy loss, with o IDD P + R
64
65
66
the softmax-function For regression, E TRP and {(fo(z).)) y) = 2 fo(()) yll² is the squared loss.
67
Structured pruning aims to remove weights or rather entire structures from NN Sa with parameters A structure can be a filter (channel) a convolutional layer, a neuron in a fully-connected layer, or an entire layer in parallel architecture. We assume the NN in question has been segmented into structures s = 1,.... S, which can potentially be pruned. We define the notation 0₂ E R as the vector whose only nonzero components are those weights from 0 that belong to structure s Then, pruning mask is a set M = {si..... Sm) } of structures. Applying a mask M to a NN fa means to consider the NN with parameter vector MMM : 0 ExEMs 0
69
71
73
74
75 We now develop our pruning methods that incorporate global correlations into their saliency as- sessment by efficiently including the second-order loss terms. The first method (SOSP-I) admits direct interpretation terms of individual loss sensitivities, while the second (SOSP-H) remains very efficient for the largest networks due to its Hessian-vector product approximation.
76
77
79
The basic idea behind both our pruning methods is select the pruning mask M so to (approxi- mately) minimize the joint effect on the network loss
80
X(M) = ((0) c()
We require that each weight is assigned to at most one structure. In practice, we associate with each structure those weights that go into the structure, rather than those that leave it. Note, mask Mf learned on 0 can be applied to different 0' A 0 on the same NN architecture.
2

---

81 of removing all structures in M, subject to a constraint on the overall pruning ratio. To circumvent 82 this exponentially large search space, we approximate the loss up second order, so that
MM) = Σ 6T dC 1 2
6T d' L
Σ
db det As'
dB
.EM
83 collapses to single-structure structure contributions plus pairwise correlations; note that the latter include interactions among the weights within a single s = 8', which can be sizeable for large structures.
84
The first-order terms A1 ( 0,- /C(())
RR. in d efficient evaluate by computing the gradient d E R once and then (sparse) dot product for every 8. In contrast to these first- order terms, the network Hessian H (0)= == de /(0) /d0² R in [1 prohibitively expensive to compute or store in full. We therefore propose two different schemes to efficiently overcome this obstacle. Each scheme entails its own way select the pruning mask. We name the full methods SOSP-I (individual sensitivities) and SOSP-H [Hessian-vectis product).
85
86
87
88
89
90
91
2.1
SOSP-1: Saliency from individual sensitivities
SOSP-I approximates each individual term 0 (0)0g in (1) efficiently, will show in Eq. (6)
We can therefore consider a modification of Eq. (I) in which the sensitivity is judged by considering
93
94
all single and pairwise sensitivities individually:
xM( - Σ
0
dCc(0)
Σ 10 10)00 S.''EM
do
2
85 To avoid cancellations between signed contributions, we take absolute values because this measures the strengths of the individual sensitivities (1 and pairwise correlations 8% H (0)0g,. While objectives other than X/2 are equally possible in the method. including A₂ and modifications with the absolute value not pulled in all the way, we found empirically that A22 performs best overall.
96
97
98
89
Then, SOSP-I iteratively selects the structures to prune, based on the objective (2) Starting from an empty pruning mask M = (). we iteratively add to M the structure s & M that minimizes the overall sensitivity 2 U {(}). In practice, the algorithm pre-computes the matrix Q E SxS
100
101
0 dC(O) 8ams's
Qs 00 H(000,+1+ +
(3)
do
102
and selects each iteration structure M to prune by
arg mig X/ s) XM arg min sgM
(0,, +2 Σ 20,0) S'EM
(4)
s
103
terminating at the desired pruning ratio.
In order to approximate the Hessian terms 0 H (0)0., efficiently, we omit from H())
104
N. Σ.. Vo((Jo(n))))) (.)) those terms that involve the expensive second-order derivatives Vzlo(an)
105
106
of the NN outputs, while including second-order couplings due to f. This is equivalent to approx-
imating (0) se H(ff")) == A Σ. Vze(fhi
(fi (In). yr. for the lincarized for (x) R f (I) :=
107
fo(r) + d(r) (0' -6) 8) with ((x)= == oof((r) E I DxP, which is well motivated by the NTK limit
108
109
(acoo al. 2018) for large NNs both initialization and after training. The terms in the sum become
V3e (fe (Ix), ym) = "x,,)"R,,, (T,)))
(5)
where R. DxD is diagonal for squared loss, and has an additional rank-1 contribution for pross-entropy (see App. B). Similar Hessian approximations were employed before in NNs (Hassibi al. 1993 Wang et al. 2019a Peng et al.. 2019) and also in the Gauss-Newton optimization method Fletcher 2013). Our final approximation is to use a random subsample of N' < N data points:
111
113
N'
8 (())g, ~ N'
Σ (o(I,))))) Rn (0.x
(6)
114 In practice, one pre-computes all (sparse) products d(z,)) E R starting from the efficient gradient 115 ((xn), before aggregating batch onto the terms 0² H())0g. Eq. (6) also has an interpretation as output correlations between certain network modifications, without using derivatives (App. C).
3

---

117
2.2
SOSP-H:
Saliency from Hessian-vector product
118
SOSP-H treats the second-order terms in T in way that is motivated by the limit of large pruning
ratios: At high pruning ratios, the sum EveM 0g in I can be approximated by Σ. 1 0₂' =;
119
Ostruc (this equals 0 if every NN weight belongs to some structure s). The second-order term
L. 0f H0)0g, Rd (E.EM (H()) thus becomes tractable since the feestrnn
121
122
product H (0)0strra is efficiently computable by a variant of the backpropagation algorithm. To
123
account for each structure s and for the first- and second-order contributions separately, above, we place absolute value signes in 1 so as to arrive at the final objective X (M) = Σ.EM X2 (s) with
124
dC(O) de
X (s)
67
10 (T(O))struc))
125
The last term measures the correlations between one structure s and all other prunable structures, although some of these may cancel unlike for SOSP-I. To minimize Xς (M), SOSP-H starts from an empty pruning mask M {}. and successively adds to M a structure s & MM with smallest X5 (s)-
126
127
128
Unlike the Gauss-Newton approximation in SOSP-I, SOSP-H uses the exact Hessian (0), but can therefore not account for individual absolute 8-8 ccrrrelaatinn, see Eq. 7 vs. ( Both methods reduce to the same first-order pruning method when neglecting the second order (1.e. H()) == 0).
129
130
131
2.3
Computational complexity
132
We detail here the computational complexities of our methods (for the experimental evaluation see Sec. 3.2). The approximation of Q in requires complexity O (N' D(F + P)) = O(NDF) for computing all ((v,,)0,. where F > P denotes the cost of one forward pass through the network (F R P for fully-connected NNs), plus O(N'DS for the sum in (6). This is tractable for modern
133
134
135
NNs, while including the exact (0) would have complexity at least O(N' DSF). Once Q has been computed, the selection procedure based on (4) has overall complexity O($*), which is feasible for
136
137
138
most modern convolutional NNs (Sec. (.2)p The total complexity of the SOSP-I method is thus O(N' DF) + O(N' + O(S*).
(8)
139
SOSP-H has computational complexity DF) to compute the sensitivities 7 which is com- parable to computing the sensitivities Q in SOSP-I when the number of structures is low (S² & F). Together with the sorting of the saliency values X5 (s), the overall complexity of SOSP-H is thus D(NDFF) + O(S log(S)).
140
141
142
Due to its weak dependency on S. in practice, SOSP-H efficiently scales to large modern networks and may even be used for unstructured second-order pruning, where S P.
143
144
Both of our methods scale much better than naively including all dff-diagonal Hessian terms, which is intractable for modern NNs due to its O DSF) scaling. Since SOSP-I builds on individual absolute sensitivities and the established Gauss-Newton approximation, we use SOSP-I in the following in particular to validate the more efficient SOSP-H method.
145
146
147
3 Results
148
149
To evaluate our methods, we train and prune VGGs (Simonyan & Zisserman, 2014), ResNets (He
et al. 2016), and DenseNets (Huang et al. 2017) on the Cifar 10/100 (Krizhevsky et al. 2009) and
150
151
Net (Deng al., 2009) datasets. Stochastie gradient descent with initial learning rate of 0. a momentum of 0.9 and weight decay of 10-4 is used to train these networks. For ResNet-32/56 and VGG-Net on Cifar 10/100, we use a batch size of 128, train for 200 epochs and reduce the learning rate by a factor of 10 after 120 and 160 epochs. To fine-tune the network after pruning, we exactly repeat this learning rate schedule. For DenseNet-40 on Cifar10/100. we train for 300 epochs and reduce the learning rate after 150 and 225 epochs. For ResNets on ImageNet, we use a batch size of 256, train for 128 epochs and use a cosine learning rate decay. For all networks, we prune feature maps (i.e. channels) from all layers except the last colnecco layer, for ResNets, we also exclude the downsampling-patt from pruning. We approximate the Hessians by a subsample of size N' = 1000 (see Sec. 2. We report the best or average final test accuracy over 3 trials if not noted otherwise. The experiments were run on an internal cluster with Nvidia Tesla V100 GPUs. Reproducing the results presented in this paper would take about 60 days of GPU run-time.
152
153
154
155
156
157
158
159
161
162

---

Table 1: Comparison of SOSP to other global pruning methods for high pruning ratios. The compari- son for moderate pruning ratios is defered to the appendix (see App. A.1). We tuned our pruning ratios to similar values as reported by the referred methods. To ensure identical implementations of the network models PyTorch, reference numbers are taken from Wang et al. (2019a) and (Mingjie & Zhuang 2018). In accordance with all referred methods, we report the mean and standard deviation of the best accuracies observed during fine-tuning. For final accuracies after fine-tuning see App. A.3. * denotes the baseline model. Both SOSP methods perform either on par outperform the competing global pruning methods.
Dataset
Cifar10 Reduct. in weights (%)
Cifar100
Test acc. (%)
Reduct. in MACs (%)
Test acc.
Reduct. in weights (5)
Reduct. in MACs (%)
Method
(%)
VGG-Net*
94.18
73.45
NN Slimming
85.01
97.85
97.89
58.69
97.76
94.09
NN Slim. +L1
91.99
97.93
86,00
57.07
97.59
93.86
C-OBD
92.34
0.18
97.68
0.02
77.39
0.36
58.07 +: 0.60
97.97 +: 0.04
77.55 0.25
EigenDamage
92.29
0.21
97.15
0.04
86,51
+ 0.26
65.18 = 0.10
97.31 + 0.01
88.63 0.12
SOSP-I (ours)
92.62
0.14
97.79
+ 0.02
83.52
+: 0.29
64,20
+: 0.23
97.83 + 0.04
87.02 +: 0.20
SOSP-H (ours)
92,71 0.19
97.81 0.01
86.32 +: 0.29
64.59 + 0.35
97.81 + 0.01
86.32 + 0.29
ResNet-32
95.30
76.8
C-OBD
91.75 = 0.42
97.30 0.06
93.50 : 0.37
59.52 = 0.24
97.74 + 0.08
94.88 0.08
EamanD
93.05
96.05 0.03
94.74 0.02
65.72 = 0.04
95.21 + 0.04
94.62 0.06
SOSP-1 (ours)
92.43
95.47 0.33
94.07
= 0.66
67.36 =
92.69 + 0.07
95.63 0.13
SOSP-H (ours)
92.23 0.12
95.26 +.10
0.40 94.45
0.21 68.42
94.08 0.21
95.06 0.14
Nen
94.58
74.11
NN Slim. L
94.22
54.21
73.19
54.21
SOSP-1 (ours)
94.21 0.04
$7.00=0.11
36.35 + 0.12
73.05 = 0.11
45.22 + 0.10
42.05 1.16
SOSP-H (ours)
94.23 0.05
49.39 + 0.65
38.86 + 0.70
73.05 + 0.24
48.58 + 0.22
42.05 + 0.35
ResNet-56
ResNet-56
DenseNet-40
DenseNet-40
b)
c)
SOSP-1
S0SP-
SOSP-H
So HH
accuracy
K
GAL
HRank
HRank VP
FROM
0.25
0.50
100
0.75
100
200 # MACs (10*)
# parameters (10")
MACs
(10")
parameters (10%)
Figure 1: Comparison of SOSP to local, ie. layer-wise, pruning methods on Cifar10. The best final test accuracy plotted over the effective number of model parameters (a, e) and MACs (b, d). A tabular representation as well as statistics across trials are shown in App. A.4. SOSP outperforms all competing layer-wise pruning methods, especially over the number of effective parameters.
163
3.1
Comparison to Literature
184
Here we benchmark the performance of our SOSP methods against existing pruning algorithms on different datasets and networks. First, we compare against other recent global pruning methods, then against local structured pruning methods, i.e. with pre-specified layer-wise pruning rates. In all comparisons
186
report the achieved test accuracy, the number of parameters of the pruned network and the MACs (often referred as FLOPs). To facilitate direct comparisons, we report test accuracies in the same way as the competing methods (c.g. best trial average over trials), but additionally report mean and standard deviation of the test error for our models in App. A. Our count of the network parameters and MACs is based the actual pruned network architecture (cf. App. D), even though our saliency measure associates with each structure only the weights into this structure.
168
170
172
173
We first compare our SOSP methods with global pruning methods on VGG-Net, ResNet-32 and DenseNet-40. We use the same variants and implementations of these networks as used by Neural Network Slimming (NN Slimming: Liu et al. 2017) as well as EigenDamage and C-OBD (Wang
174
176
al. 2019a), e.g. capping the layer-wise ratio removed structures at 95% for VGGs to prevent layer collapse and increasing the width of 32 by factor 4. C-OBD is a structured variant of
5

---

Table 2: Best final test accuracies and pruning ratios (PR) across 2 trials on ImageNet. For comparison to CCP, we also provide their alternative MAC count (for details, see App. D). * denotes SOSP with kemeel scaling (see main text). SOSP outperforms all three competing methods.
20
15
Runtime
50.P- 505P H
Model
Top-1% (Gap)
Parameters (PR)
MACs (PR)
Alt. MACs (PR)
0
ResNet-18
69.76
11.7M (0%)
1882 (0%)
I 1882 (0%)
1.0
1.5
2.0
SOSP (ours)
69.63 (0.13)
7.12M (39%)
1.378 (24%)
(28%)
Width network
FPGM
68.41 (1.87)
7.10M (39%)
1.06B (41%)
SOSP (ours)
68.78 (0.98)
6.42M (45%)
1.29B (29%)
L,00 (34%)
Figure 2:
Runtime to calculate the
ResNet-50
76.15 (0.0)
25.5M (0%)
3,8 (0%)
3,85B (0%)
SOSP (ours)
76.561-4..41
19.9M(22%)
3 3.06((215)
2.72842955)
pruning masks for ResNet-56 on Ci- fo over the width of the network for SOSP-I and SOSP-H. We vary the width of the network by increas- ing the width of each layer by mul- tiplicative factor.
SOSP* (ours)
76.60 (-0.45)
17.9M (30%)
2.79B (28%)
2,478 (36%)
HRank
74.98 (1.17)
(36%))
(445%)
FPGM
75.59 (0.56)
15.9M (37%)
2.368 (42%)
SOSP (ours)
75.85 (0.30)
15.4M (40%)
2,44B (279))
1.978 (49%)
HRank
71.98 (4.17)
13.8M (46%)
(62%)
CCP
75.21 (0.94)
1.77B (54%)
SOSP* (ours)
75.21 (0.94)
13.0M (49%)
2.138 (459)
1.688 (569))
SOSP (ours)
74.39 (1.76)
(549))
(51%))
(641%)
SOSP* (ours)
73.38 (2.77)
9.9M (61%)
1,58D (59%)
1.10B (72%)
178
the original OBD algorithm (Hassibi et al. 1993), which neglects all cross-structure correlations that, three trials for high pruning ratios shown in Tab. and for moderate pruning ratios in App. 1. To enable the comparison to NN Slimming on an already pretrained VGG, we included the results of NN Slimming applied to a baseline network obtained without modifications to its initial training, i.e. without L Ll-regularization
179
in contrast, SOSP takes into account. The results
180
181
182
on the batch- normalization parameters. For moderate pruning rates, all pruning schemes approximately retain the baseline performance for VGG-Net and ResNet-32 on Cifar and VGG-Net on Cifar 100 (see Tab. 3). The only exception is the accuracy for C-OBD applied VGG-Net on 100 which drops by approximately 1%. For ResNet-32 on Cifar100 the accuracy after pruning is approximately 1% lower than the baseline, for all pruning schemes. In the regime of larger pruning ratios of approximately 97%, SOSP and EinenDamage significantly outperform NN Slimming and C-OBD. SOSP performs on par with EigenDamage, except for ResNet-32 on Cifar100, where SOSP outperforms EigenDamage by almost 3%. This result indicates that SOSP outperforms all other methods especially in the regime of baseline networks that have relatively few parameters already and on difficult datasets most relevant for applications. For DenseNet-40, we achieve similar results compared to NN Slimming. However, note that NN Slimming requires the modification of network pretraining.
183
184
185
186
187
188
189
191
192
193
194
Next, we compare our SOSP methods against four recently published local. i.c. layer-wise, pruning algorithms: FPGM He et al. (2019). GAL (Lin et al., 2019)) CCP (Peng al.) 2019). Variational
195
196
Pruning (VP: ZZao et al. (019) and HRank (Lin et al. 2020). For ResNet-56, our SOSP methods outperform all other methods across all pruning ratios (see Fig. Th and b). For DenseNet-40, SOSP
197
198
achieves better accuracies when compared over parameters (Fig. [e) and is on par with the best other methods over MACs (Fig. Td). The reason for this discrepancy is probably that the SOSP objective is agnostic to the number of MACs (image size) in each individual structure.
199
201
3.2
Scalability and application large-scale datasets
202
Before going to large datasets, we compare the scalability of our methods SOSP-I and SOSP-H. As the preceding section shows, both methods perform basically on par with each other in terms of accuracy. This confirms that SOSP-H is not degraded by the approximations leading to the efficient Hessian-vector product, or is helped by use of the exact Hcssian. In terms of efficiency, however, SOSP-H shows clear advantages compared to SOSP-I, for which the algorithm to select the structures to be pruned scales with ( (see Sec. 2.3) potentially dominating the overall runtime for scale networks. Measurements of the actual runtimes show that already for medium sized networks SOSP-H is more efficient than S (Fig. 2). Since SOSP-I becomes impractical for large-sees scale networks, for the ImageNet dataset we will only evaluate SOSP-H and refer to it as SOSP.
203
204
205
207
209
210
211
On ImageNet, we compare our results to literature for ResNet- 18 and ResNet-50, see Tab. 2 Because SOSP assesses the sensitivity of each structure independently of the contributed MACS, it has bias towards pruning scall-scale structures. This tendency strongest for ResNet-50, due to its 1 I-convolutions tend to contribute disproportionately little to the
213
1 x I-convolutions. Since these 1 x

---

a) train-pruning
pruning ratio
0.9
NN
ResNet- accuracy
0.6
92
trinigruning
0.3
prunnin
NN
b) init- pruning
0.2
0.4
0.6
20
40
h)
20
40
pruning ratio
0.9
NN
mesk
VGG accuracy
0.6
92
0.3
0
10
15
0
10
15
parameters (109)
layer index
layer index
Comparison between pruning after training and at initialization on Cifar10. Both pruning schemes, train-pur pruning (a) and init- init-pruning (b), train the network for the same overall number of epochs, but generate and apply the pruning masks at different point in times. The average and standard deviation of the test accuracy across trials is plotted against the number of model parameters for ResNet-56 (top row; c) and VGG (bottom row; f). For single trial, in which overall 50% of the structures are pruned, we visualize the pruning masks of sain-pruning and init-pruning by showing the layer-wise pruning ratios in (d, g) and (e, h), respectively.
Figure 3:
215
overall number of MACs, we devised a scaled variant of SOSP, which divides the saliency of every structure by the kernel size (c.g. 1, 3 7). Compared to the vanilla SOSP, the scaled variant of SOSP is able to remove larger percentages of MACs with similar drops in accuracy (see Tab. (
217
218
For both networks SOSP outperforms HRank and FPGM, especially when considering the main objective of SOSP, which is reduce the number of parameters or structures. Since CCP uses different way of calculating the MACs, which leads to consistently higher pruning ratios, we added an alternative MAC count to enable a fair comparison (for details, see App. D). Since HRank and FPGM do not mention their MAC counting convention, assume they use the same convention 223
219
221
as we do. Taking this into account, our scaled SOSP variant able to prune more MACs than CCP, while having the same final accuracy.
3.3 Pruning at Initialization
226
Traditionally, pruning methods are applied to pretrained networks, as also done in the previous sections, but recently there has been growing attention on pruning at initialization following the works Lee et al. (2018) and Frankle & Carbin (2018). Since SOSP employs the absolute value of the sensitivities. it can also be applied
228
randomly initialized network without any modifications. Thus, SOSP can also be seen as an efficient second-order generalization of SNIP (Lee et al. 2018 Amersfoort al., 2020). While FigenDamage can in principle be modified and applied to randomly initialized network, NN Slimming can not be applied at initialization.
230
232
Usually pruning at initialization leads to worse accuracies than pruning after training (Liu al. 2018). However, pruning an already trained network is often followed by fine-tuning effectively training the network twice (Fig. 3R). For comparability with pruning at initialization, we unify the overall training schedule between these two settings and consequently apply two training cycles after pruming the randomly initialized network (Fig. 3b: for further discussion, see App. A.5).
234
236
238
We observe that applying SOSP at initialization performs almost equally well than applying SOSP after training (see ResNet-56 and VGG in Fig. 3 and f, respectively). In conclusion, applying SOSP at initialization can significantly reduce the time and resources required for network training with no or only minor degradation in accuracy.
240
3.4 Identifying & Removing Architectural Bottlenecks
243 Even though the previous section highlighted that applying SOSP after training and at initialization results in comparable accuracies, the pruning masks differ between these two scenarios (compare Fig.
7

---

a) expand-pruning
b)
expand-pruning
d)
widen-pruning
pruning ratio
width multiplier
NN
traiin
init
ResNet:S6
accuracy
94
0.6
expand
prune
93
traingp: expand-pr pr
0.3
NN
widen-pr.
92
0.0
mask
NN
e)
0.25
0.50
0.75
25
50
25
50
94.0
pruning ratio
0.9
width multippic er
VGG accuracy
mask
NN
93.5
NN
93.0
0.3
92.5
10
0
10
# parameters (10*)
layer index
lnyerindex
Figure 4: We remove architectural bottlenecks found by SOSP using the sreand-punningggg scheme (a) on Cifar10. The width of blocks and layers with low pruning ratios in the train-pruninn scheme (Fig. 3 and g) are expanded by width multiplier of 2 (c. f). As a bascline, layers in the network by a factor 1.1 (d, g)- The layer-wise pruning ratios of the enlarged network models shown bar plots in (c, d, f, g). The average and standard deviation of the test accuracy across 3 trials are shown over the number of model parameters (b, e). Note that the full ResNet-56 and VGG models have 0.86 10th and 20- 10 parameters, respectively.
uniformly expand all
3 and g to e and h, resp.). Despite this difference common feature of all masks is that some layers 245
248
are barely pruned or not pruned at all while others are pruned by up to 80%. This could indicate towards architectural bottlenecks. We consider a layer an architectural bottleneck if the respective
247
248
layer has considerably lower pruning ratio compared to the other layers. The low pruning ratio of bottleneck layers indicates that the substructures (e.g. filters) have a high sensitivity. Thus, widening these layers could improve the overall performance and allow for even smaller models with higher accuracies.
249
250
251
252
To utilize this insight we device a procedure that we call expand-pruning. The idea is to first calculate the pruning ratios of trained network and then identify architectural bottlenecks, i.c. the layers with the lowest pruning ratios. Next, we widen these layers by factor of two, which has been empirically shown to work well. Finally, we randomly initialize, train, prune, and fine-tune the expanded network (for a schematic, see Fig, Fh). As a naive baseline that we call the widen-pruming procedure, we widen all layers in the network with constant factor instead of widening specific layers. We choose the constant factor such that the overall number of parameters matches that of the expand-prung procedure (see width multiplier in Fig. 4
253
254
255
256
257
258
259
280
We evaluate the expand-pruning procedure for Nes and VGG on Cifar10, for which we expand the least pruned of the three main building blocks and the five least pruned layers, respectively (e.g.. see width multipliers in Fig. 4 and f selected on the basis of the pruning masks shown in Fig. and
262
283 g, resp.). Note that for ResNet-56 more -rnnigr removal of bottlenecks, e.g. on layer level,
254
is not possible without changing the overall ResNet architecture. In summary, a selective removal of bottlenecks results in smaller network models with higher accuracy than pruning the vanilla network or unselectively increasing the network size (compare expand-pruning to train-pruning and widen- wwden-pruninn pruning in Fig. 4p and e). While principle any global pruning method could be used for the expand- pruning procedure, SOSP is especially suited since it does not require to modify the network architecture like EigenDamage and can also be applied at initialization unlike NN Slimming, allowing for a similar expand scheme directly at initialization (see App. A.6).
285
266
267
268
269
4
Discussion
272
In this work we have demonstrated the effectiveness and scalability of our second-order structured pruning algorithms (SOSP). While both algorithms perform similarly well, SOSP-H more easily scalable to large scale networks and datasets. We highlighted two major features of our method. Firstly, SOSP can be applied at initialization with only minor degradation accuracy, which drastically reduces the required time and resources for training. Secondly, we showed that the pruning masks
273
275
8

---

277
found by SOSP can be used to systematically detect and remove architectural bottlenecks, further improving the performance of pruned networks.
278
279
Compared to other global pruning methods, SOSP captures correlations between structures by a sim- ple, effective and scalable algorithm that neither requires to modify the training nor the architecture of the to be pruned network model and achieves comparable or better accuracies on benchmark datasets.
280
281
282
The C-OBD algorithm (Wang et 2019a) a structured generalization of the original unstructured
283
OBD algorithm (LeCun et a 1990). In contrast to OBD, C-OBD accounts for correlations within
284
each structure, but does not capture correlations between different structures within and across layers.
285
We show that considering these global correlations consistently improve the performance, especially for large pruning ratios (Tab. This observation further confirmed by an ablation study in which we neglect all cross-structure correlations significantly decreasing the performance under otherwise identcaa experimental settings (App. A.2). The objective of FigenDamage (Wang et al., (019a) to
206
287
288
289
include second order correlations is similar to ours, but the approaches are significantly different. FagenDamage uses the Fisher-approximatton, which is similar to the Gauss-Newton approximation used for SOSP-I, and then, in addition to further approximations, apply low rank approximations that require the substitution of each layer by a bottleneck-block structure. Our SOSP method is simpler, easier to implement and does not require to modify the network architecture, but nevertheless performs on par with EigenDamage. The approach of NN Slimming (Liu et al.) (017) is more heuristic than SOSP and is easy to implement. However, networks need to be pretrained with П1
290
292
294
regularization on the batch-normalize aatch-normalzzimpe normmaiamaiinnionn parameters, otherwise the performance is severly harmed (Tab. D and 3). SOSP does not require any modifications to the network training and can be applied to any pretrained network. A recent variant of NN Slimming was developed by /Zhaang et al. (2020)
296
298
299
who optimize their hyperparameters to reduce the number of MACs. Using the number MACS an objective for SOSP is left for future studies.
300
In addition to the above comparison to other global pruning methods, we also compared our methods to simpler local pruning methods that keep the pruning ratios constant for each layer and, consequently, scale well to large-scale datasets. The pruning method closest to our SOSP-I method is the one by Peng et al. (2019). While both works consider second-order correlations between structures, theirs is based on pruning objective different from our absolute sensitivites in Al and considers only intra- layer correlations. Furthermore, they employ an auxiliary classifier with a hyperparameter, yielding accuracy improvements that are difficult disentangle from the effect of second-order pruning. Going beyond a constant pruning ratio for each layer [Su et al.] (2020) discovered, for ResNets, that pruning initialization seems to preferentially prune initial layers and thus proposed pruning scheme based on a "keep-ratio" per layer which increases with the depth of the network. Our experiments confirm
302
304
306
307
308
309
310
311
some of the findings Su et al. (2020), but we also show that the specific network architectures found by pruning can drastically vary between different networks and especially between initialization and after training (histograms in Fig. 3. While all local pruming methods specify pruning ratios for each layer, our method performs automatic selection across layers (histograms in Fig. 3)
312
313
314
315
This automatic selection allows us identify and remove architectural bottlenecks. However, global pruning method has a bias towards pruning small structures, absent from local pruning methods, as the size of structures is usually identical within layers. We propose simple solution by scaling each structure by the inverse of the kernel size which helps to remove some of the bias. Alternatively,
316
318
319
to better reflect the computational costs in real-world applications, each structure could also be
320
normalized by the number of its required MACs (like done by van Amersfoort et al.) 2020).
Recently, unstructured (Lee et al. 2018 8, Wang et al. 2019b Tanaka et al., 2020 and structured van Amersfoort et al./ 2020 Hayou et al., (021)) pruning schemes that are applicable to networks at initialization were proposed. While these methods fail to achieve similar accuracies compared to pruming after training, our SOSP method in the init- mut--runinn setting achieves accuracies comparable to pruning after training.
322
324
325
In accordance with juu/Elsken et al. (2019), our results suggest that pruning can be used to optimize the architectural hyperparameters of established networks (Liu et al. 2018)) or super-graphs (Noy et al.) 20 Instead of formulating this optimization as pruning process, we envision our second-order 328
326
327
sensitivity analysis to be a valuable tool to identify and remove bottlenecks to find good neural architectures more quickly. For example, whenever a building block of the neural network cannot be compressed, this building block may be considered a bottleneck of the architecture and could be inflated to improve the overall trade-off between accuracy and computational cost.
330
332

---

References
334
Blalock, D., Ortiz, J. G., Frankle, J., and Guttag, J. What the state of neural network pruning? arXiv preprint arXiv:2003.03033, 2020.
336
Deng, J., Dong, W. Socher. R., Li, L.-.,, Li, K., and Fei-Fei, Imagenet: A large-scale hierarchical
337
image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 255. leee, 2009.
338
339
Elsken, T., Metzen, J. H., and Hutter, F. Neural architecture search: A survey. Journal of Machine Learning Research, 20(55):1-21 2019.
340
341
Fletcher, R. Practical methods of optimization, John Wiley & Sons, 2013.
342
Frankle, J. and Carbin, M. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2018.
343
344
Han, S., Pool, J., Tran, J., and Dally, W. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28: 135-1143, 2015.
Hassibi, B., Stork, D. G., and Wolff, G. J. Optimal brain surgeon and general network pruning. In IEEE international conference on neural networks, pp. 293-299. IEEE, 1993.
347
Hayou, S., Ton, J.-F. Doucet, A., and Teh, Y. W. Robust pruning at initialization. In International Conference on Learning Representations, 2021. URL.Atttp:://openeeevee.e.ce fitmmmmm?i vXj ucZQ4hA
349
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
352
He. Y., Zhang, X., and Sun, J. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1389-1397, 2017
354
355
He, Y., Liu, P., Wang, Z., Hu, Z., and Yang. Y. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4340-43499 2019.
356
357
Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700 4700- 2017.
359
361 Jacot, A., Gabriel, F., and Hongler, C. Neural tangent kernel: convergence and generalization in neural networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 3580-8589, 2018
362
363
364
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
365
LeCun, Y., Denker, J. S., and Solla, S. A. Optimal brain damage. In Advances in neural information processing systems, pp. 598-55888909 605, 1990.
366
367 Lee, N., Ajanthan, T., and Torr, P. Snip: Single-shot network pruning based on connection sensitivity. In International Conference on Learning Representations, 2018.
389
Lin. M., Ji, R., Wang, Y., Zhang, Y., Zhang, B., Tian, Y., and Shao, L. Hrank: Filter pruning using high-rank feature map. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1538-1538. 2020.
370
371
372
Lin, S., Ji, R., Yan, C., Zhang, B., Cao, L.., Ye, Q., Huang, F., and Doermann, D. Towards optimal structured enn pruning via generative adversarial learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 2790-2799, 2019.
374
Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., and Zhang, C. Learning efficient convolutional networks through network slimming. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
376
10

---

378
Liu, Z., Sun, M., Zhou, T., Huang, G., and Darrell. T. Rethinking the value of network pruning. In International Conference on Learning Representations, 2018.
379
380
Mingjie, S. and Zhuang. fettorkk-limmingg 2018.
Network slimming.
Noy, A., Nayman, N., Ridnik, T., Zamir, N., Doveh, S., Friedman, I,, Giryes, R., and Zelnik, L. ASAP: Architecture search, anneal and prune. In Chiappa. S. and Calandra, R. (eds.), Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108
383
385 of Proceedings of Machine Learning Research, pp. 493-503, 2020.
386
Peng, H., Wu, J., Chen, S., and Huang, J. Collaborative channel pruning for deep networks. In International Conference Machine Learning, pp. 51 13-5122. PMLR, 2019.
Reed, R. Pruning algorithms-a survey. IEEE transactions Neural Networks, 4(5):740 747, 1993.
Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1409. 2014.
390
381
Su, J., Chen, Y., Cai, T., Wu, T., Gao, R., Wang. L.,, and Lee, J. D. Sanity-checking pruning methods: Random tickets win the jackpot. In Advances in Neural Information Processing Systems, 2020.
Tanaka, H., Kunin, D., Yamins, D. L.,, and Ganguli, S. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in Neural Information Processing Systems, 33, 2020.
394
395
396
Tang, Y., Wang, Y., Xu, Y., Tao, D., Xu, C., Xu, C., and Xu, C. Scop: Scientific control for reliable neural network pruning. Advances in Neural Information Processing Systems, 2020.
397
398
van Amersfoort, J., Alizadeh, M., Farquhar, S., Lane, N., and Gal, Y. Single shot structured pruning before training. arXiv preprint arXiv:2007.0033: 2020.
389
400
Wang, C., Grosse, R., Fidler, S., and Zhang, Figendamage: Structured pruning in the kronecker- factored eigenbasis. In International Conference on Machine Learning, pp. 1566-6575 2019a.
Wang, C.,, Zhang, G., and Grosse, R. Picking winning tickets before training by preserving gradient flow. International Conference on Learning Representations, 2019b.
405
404
Zhao, C., Ni, B., Zhang, J., Zhao, Q., Zhang, W., and Tian, Q. Variational convolutional neural network pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2780-2789, 2019.
406
407
Zhuang. T., Zhang, Z., Huang. Y., Zeng, X., Shuang, K., and Li, X. Neuron-level structured pruning using polarization regularizer. Advances in Neural Information Processing Systems, 33, 2020.
Checklist
409
1. For all authors...
411
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
412
contributions and scope? [Yes]
413
(b) Did you describe the limitations of your work? [Yes]
414
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
415
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
418
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
420
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]

---

423
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
424
425
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
428
429
4. If you are using existing assets (c.g., code, data, models) or curating/releasing new assets...
430
(a) your work uses existing assets, did you cite the creators? [Yes]
431
(b) Did you mention the license of the assets? [Yes]
432
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
434
435
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
438
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
439
440
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
443
12


Reviews:
[{"review_id": "nuegUBVnxdz", "reviewer": "Reviewer_edNm", "summary": "This work proposes a second-order structured pruning algorithms (SOSP), which can drastically reduce training time and further improve pruned network performance by removing architecture bottlenecks", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work is well-explained and easy to follow. This paper has very detailed background overview on the state-of-the-art pruning techniques and recent findings. This paper presents proposed method in good detail and complexity analysis. Extensive experiments on CIFARs with multiple architectures effectiveness of the proposed approach in pruning and further conducted experiments demonstrated its effectiveness in removing architectural bottlenecks.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper does discuss its limitations.\n\nQuestions and suggestions:\n \nWhat would be the scale limits to choose SOSP-I over SOSP-H? Can we have more experiment in performance comparison between those two as network scale increases?\n \n \nIt would be interesting to compare with more recent pruning works such as  [1,2].\n \n[1] AutoCompress: An Automatic DNN Structured Pruning Framework for Ultra-High Compression Rates.\n[2] EagleEye: Fast Sub-net Evaluation for Efficient Neural Network Pruning\n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "T4zcuQvhIQe", "reviewer": "Reviewer_mjMX", "summary": "This paper presents two novel approximation techniques for approximating the importance of a filter (the estimated loss difference between pruned and pre-trained networks) using second order approximations. The main difficulties of second order approximation lie in the computational burden of computing the full Hessian matrix. To circumvent this issue, this paper presents two novel ways to do so. The first approach, dubbed SOSP-I, linearize the neural network and only consider the coupling of the loss function. The second approach, dubbed SOSP-H, approximate the loss difference using a Hessian-vector product. Both approaches have much better efficiency compared to using the Hessian and empirically perform a little better than existing approaches. Overall, the approximations seem novel to me and I find the appendix helpful in understanding the derivations. However, their empirical advantages seem rather limited. Additionally, I think some of the related papers are missed and they need to be discussed and compared against.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n\n- Novel approaches for approximating the Hessian for filter pruning\n- The manuscript is well-written and easy to follow\n- Filter pruning is a practically relevant research direction\n- Comparisons with EigenDamage, NN-Slimming, and C-OBD are comprehensive for CIFAR\n\n## Weaknesses\n\n- The improvement over direct competitors (EigenDamage, NN-Slimming, and C-OBD) is relatively limited. Specifically, SOSP-H achieves 0.4% improvements over EigenDamage with similar MAC reduction for VGG on CIFAR-10 while being worse on CIFAR-100. Compared to EigenDamage, it seems only ResNet32 for CIFAR-100 works well. Not sure how to interpret this and an immediate question from the readers would be: When should we use EigenDamage and when should we use the proposed method? While I appreciate the results on ImageNet, direct competitors are not shown on ImageNet. More specifically, I do not think it is informative to compare local pruning methods with global pruning methods as the resulting pruned architectures are very important. It would be great if global pruning methods (e.g., EigenDamage, NN-Slimming, and C-OBD) are shown in the ImageNet table.\n\n- Missing global pruning methods. Specifically, both [1,2] approximate the loss differences using Taylor expansion as well. Although missing second order information, I think it is not clear how much second order approximation adds to the picture if these first-order methods are not compared. Additionally, as noted by the authors in line 215, different layers contribute to MAC differently while the approximation only considers loss but not MAC. Hence, I think it is also critical to compared to learned global ranking such as [3] which factors the heterogeneous MAC into the ranking learning process.\n\n- Missing comparisons with local pruning methods that have learned layerwise pruning ratio [4].\n\n- Missing analysis on compact networks such as MobileNets and EfficientNets. Accelerating networks is important, but if the proposed method does not perform well on already fast/small networks, it is less significant.\n\n[1] Theis, Lucas, et al. \"Faster gaze prediction with dense networks and fisher pruning.\" arXiv preprint arXiv:1801.05787 (2018).\n\n[2] Molchanov, Pavlo, et al. \"Importance estimation for neural network pruning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.\n\n[3] Chin, Ting-Wu, et al. \"Towards efficient model compression via learned global ranking.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n\n[4] He, Yihui, et al. \"Amc: Automl for model compression and acceleration on mobile devices.\" Proceedings of the European conference on computer vision (ECCV). 2018.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I do not think this paper has negative societal impacts", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "QEmZLo45OO", "reviewer": "Reviewer_qQtX", "summary": "This work proposes a structured pruning method that removes feature maps (channels) of convolutional neural networks without losing accuracy while increasing computational efficiency (MACs). The proposed method (SOSP-I/H) is based on a sensitivity based saliency and involves what they call global correlations through the second order partial derivatives with respect to different structures. SOSP shares a lot in common with previous works including OBD and C-OBD for the use of second order information except that they approximately incorporate all elements in the Hessian matrix. Then the paper evaluates the effectiveness of the proposed method for image classification task using VGG, ResNet, and Desnet on CIFAR and ImageNet datasets and achieve accuracies on par with previous works on both global and local (layerwise) pruning. The paper further presents that SOSP works on pruning at initialization and can improve further by performing expand-pruning based on skewed distribution of retained parameters over layers.\n", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Using second order information to locally approximate the changes in the loss function can be useful, and in fact it has been used in many prior works as mentioned in the paper.\nAs it is quite expensive to compute the entire Hessian accurately for a deep network with a large number of parameters, the paper proposes to approximate based on (6) which is using the Hessian vector products.\nHere, they make a rather strong assumption on the wide network referring to the NTK regime and the local first order approximation (linearization) of the network computation.\nThe paper attempts to demonstrate the effectiveness of the proposed method on large-scale experiments using ImageNet.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- The paper often misses clear explanations of why their approximation should work as intended. For instance, from (1) to (2) they take absolute values individually by saying that it is to avoid cancellations (L95-L98) however they clearly measure saliency differently from each other, and presumably the resulting pruned structures will most likely look very different.\n- The method is not evaluated clearly how efficient the proposed algorithm is compared to other approaches; the computational cost for a high pruning ratio seems to be estimated not quite tight or realistic.\n- A clear limitation of SOSP is that the method does not compare very strongly or improves over other existing prior works. Only in a few cases of global pruning seems to improve very marginally.\n- The evaluations and analysis performed seem a bit insufficient to support claims such as the argument that SOSP at initialization works on par with SOSP after training while performing training twice and claim against other pruning approaches at initialization.\n- Generalizing and automating expand-pruning rather than widening by a factor of two would make the process of finding bottleneck appeal stronger.\n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "G2smoCPbVQn", "reviewer": "Reviewer_pfs4", "summary": "This paper proposed a structured pruning method which include correlations among structures by second order information. The author also improved the computational efficiency by fast Hessian-vector products. The method is demonstrated to achieve comparable/better results than previous methods. And it can also be applied to pruning-at-initialization and reveal architecture bottlenecks.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. The method is well-motivated. The author provided thorough derivations to explain the method.\n2. The method can be applied both after-training and at-initialization.\n3. The application of identifying architecture bottleneck and correspondingly further improve the architecture is interesting.\n\nWeakness:\n1. In Table 1, it seems that the method only achieves comparable results with previous methods. In some cases it is better but the improvements are also very marginal.\n2. The compared methods didn't seem to be the most recent ones. Most of them are in 2019 and before. Could the author conduct some comparison with the most recent state-of-the-art?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a pruning technique that exploits second order information. In order to circumvent the high computational cost, an approximate computation is proposed. The improvements brought by the presented method are mostly marginal, and the method is not really compared to the most recent state of the art. Also, some of the critical parts of the paper lack clarity, for instance when motivating the approximation scheme. Overall, I recommend the rejection of this paper and encourage the authors to take the points raised by the reviewers into account when re-working the manuscript and preparing a submission to another venue. 
Decision: Reject