Succinct Network Channel and Spatial Pruning via Discrete Variable QCQP
Authors: Yeonwoo Jeong
Deokjae Lee
Gaon An
Changyong Son
Hyun Oh Song
Venue:  (2021)
Keywords: Network Pruning
Channel pruning
Spatial pruning
Network Compression
MIQCQP
Specified target resource constraint

Abstract:     Reducing the heavy computational cost of large convolutional neural networks is crucial when deploying the networks to resource-constrained environments. In this context, recent works propose channel pruning via greedy channel selection to achieve practical acceleration and memory footprint reduction. We first show this channel-wise approach ignores the inherent quadratic coupling between channels in the neighboring layers and cannot safely remove inactive weights during the pruning procedure. Furthermore, we show that these pruning methods cannot guarantee the given resource constraints are satisfied and cause discrepancy with the true objective. To this end, we formulate a principled optimization framework with discrete variable QCQP,  which provably prevents any inactive weights and enables the exact guarantee of meeting the resource constraints in terms of FLOPs and memory. Also, we extend the pruning granularity beyond channels and jointly prune individual 2D convolution filters spatially for greater efficiency. Our experiments show competitive pruning results under the target resource constraints on CIFAR-10 and ImageNet datasets on various network architectures.


Under review as a conference paper at ICLR 2021
SUCCINCT NETWORK CHANNEL AND SPATIAL PRUN- ING VIA DISCRETE VARIABLE QCQP
Anonymous authors Paper under double-blind review
ABSTRACT
Reducing the heavy computational cost of large convolutional neural networks is crucial when deploying the networks to resource-constrained environments. In this context, recent works propose channel pruning via greedy channel selection achieve practical acceleration and memory footprint reduction. We first show this channel-wise approach ignores the inherent quadratic coupling between channels in the neighboring layers and cannot safely remove inactive weights during the pruning procedure. Furthermore, we show that these pruning methods cannot guarantee the given resource constraints are satisfied and cause discrepancy with the true objective. To this end, we formulate a principled optimization framework with discrete variable QCQP, which provably prevents any inactive weights and enables the exact guarantee of meeting the resource constraints in terms of FLOPs and memory. Also, extend the pruning granularity beyond channels and jointly prune individual 2D convolution filters spatially for greater efficiency. Our experiments show competitive pruning results under the target resource constraints on CIFAR-I0 and ImageNet datasets on various network architectures.
INTRODUCTION
Deep neural networks are the bedrock of artificial intelligence tasks such as object detection, speech recognition, and natural language processing (Redmon & Farhadi 2018 Chorowski et al. 2015) Devlin et al.) 2019). While modern networks have hundreds of millions to billions of parameters to train, it has been recently shown that these parameters are highly redundant and can be pruned without significant loss in accuracy (Han et al. 2015 Guo et al. 2016). This discovery has led practitioners to desire training and running the models on resource-constrainee mobile devices, provoking large body of research network pruning.
Unstructured pruning, however, does not directly lead to any practical acceleration or memory footprint reduction due poor data locality (Wen et al. 2016). and this motivated research on structured pruning to achieve practical usage under limited resource budgets. To this end, line of research on channel pruning considers completely pruning the convolution filters along the input and output channel dimensions, where the resulting pruned model becomes smaller dense network suited for practical acceleration and memory footprint reduction (Li et al. 2017 Luo et al. 2017 He
et al. 2019 Wen et al. 2016 He et al. 2018a).
However, existing channel pruning methods perform the pruning operations with greedy approach and does not consider the inherent quadratic coupling between channels in the neighboring layers. Although these methods are easy to model and optimize, they cannor safely remove inactive weights during the pruning procedure, suffer from discrepancies with the true objective, and prohibit the strict satisfaction of the required resource constraints during the pruning process.
The ability to specify hard target resource constraints into the pruning optimization process is important since this allows the user to run the pruning and optional finetuning process only once. When the pruning process ignores the target specifications, the users may need to apply multiple rounds of pruning and finetuning until the specifications are eventually met, resulting in an extra computation overhead (Han et al. 2015 He et al. 2018a Liu al. 2017).
In this paper, we formulate principled optimization problem that prunes the network layer chan- nels while respecting the quadratic coupling and exactly satisfying the user-specified FLOPs and

---

Under review as a conference paper at ICLR 2021
with w-th
pruned
x(1-1)
W((+11
x(++))
w
x5 =(x(-11 ② ( Dir,
x leads to inactive weights 17(+1))
Equation
Figure I: Illustration of channel pruning procedure that leads to inactive weights. When j-th output channel of 1-th convolution weights w:( "j pruned, i.e. w KC,-1.K,KK, then the j-th feature map of 1-th layer x should also be 0. Consequently, x yields inactive weights wx(+1)) Note
that we use w:()) to denote the tensor W:( "J... following the indexing rules of NumPy Van Der Walt et al. 2011
memory constraints. This new formulation leads to interesting discrete variable QCQP (Quadratic Constrained Quadratic Program) optimization problem, which directly maximizes the importance of neurons the pruned network under the specified resource constraints. Also, we increase the pruning granularity beyond channels and jointly prune individual 2D convolution filters spatially for greater efficiency. Furthermore, we generalize our formulation to cover nonsequential convolution operations, such as skip connections, and propose principled optimization framework for handling various architectural implementations of skip connections in ResNet (He et al.) 2016). Our experiments on CIFAR-I0 10 and ImageNet datasets show the state of the results compared other channel pruning methods that start from pretrained networks.
MOTIVATION
In this section, we first discuss the motivation our method concretely. Suppose the weights in a sequential CNN form a sequence 4-D tensors, W RCL xgc_xxxxx x VI E [L] where Cza, C₁, and K; represent the number of input channels, the number of output channels, and the filter size of 1-th convolution weight tensor, respectively. We denote the feature map after (-th convolution as ( RCi Concretely, x-)) = a (-X(1) @ W(9 oEE
CL ( W where is the activation function, * denotes 2-D convolution operation, and @ denotes the sum of channel-wise 2-D convolutions. Now consider pruning these weights in channel-wir direction. We show that naive channel-wise pruning methods prevent exact specification of the target resource constraints due unpruned inactive weights and deviate away from the true objective by ignoring quadratic coupling between channels in the neighboring layers.
2.1 INACTIVE WEIGHTS
According to Han et al. (2015), network pruning produces dead neurons with zero input or output connections. These dead neurons cause inactive weight which do not affect the final output activations of the pruned network. These inactive weights may not be excluded automatically through the standard pruning procedure and require additional post-processing which relies on ad-hoc heuristics. For example, Figure shows standard channel pruning procedure that deletes weights across the output channel direction but fails to prune the inactive weights. Concretely, deletion of weights on j-th output channel of (-th convolution layer leads to ((()) KK. Then, x becomes dead neuron since x!" (xx(1) w!)) of
Cle x
(t-1) w!()) OC,,H.W.
Rigorous mathematical definition of inactive weights is provided in Supplementary material D.
2

---

Under review as a conference paper at ICLR 2021
The convolution operation the dead neuron results in trivially zero output, below:
x((+))
Σ x w(thi))
Σ
11 Lzy X
X
wi(t-) J.p
dead
inactive
O.ga Wist
Equation (1) shows that the dead neuron X causes weights (((H)) Vp Such inactive weights do not account for the actual resource usage, even when they remain in the pruned network, which prevents the exact modeling of the user-specified hard resource constraints (FLOPs and network size). Furthermore, inactive weights unpruned during the pruning procedure are a bigger problem for nonsequential convolutional networks due to their skip connections. address this problem, we introduce a quadratic optimization-base algorithm that provably eliminates all the inactive weights during the pruning procedure.
[Cou] be inactive.
j.p
2.2
QUADRATIC COUPLING
X
pruned
X
pruncd X(++)
X(1-I proned
pruncui XP++
inaclive
12
12 11
9
12
12
11
Objective value: (3 (3+55 5
(3
3) 18
Objective value: (1+ 21+2+11) 8)
(a) Greedy
(b) Ours
Figure 2: A comparison of the greedy channel pruning method and our pruning method. Parallelo- grams represent feature maps and squares represent 2-D filters of convolution weights. Gray squares filters which account for the objective. The numbers each squares represent the absolute sum of weights in the filter.
Existing channel pruning methods remove channels according to their importance. However, mea- suring channel's is contribution to the network should also take into account the channels in the neighboring layers, as illustrated in Figure In the example, we define the importance of a channel the absolute sum of weights in the channel, in Li et al. (2017), and assume the objective is to maximize the absolute sum of weights in the whole pruned network, excluding the inactive weights. We compare two different channel pruning methods: (a) standard channel pruning method that greedily prunes each channel independently. and (b) our pruning method that considers the effect of the channels in neighboring layers when pruning. As result of rumning each pruning algorithms, (a) will prune the second output channel of the first convolution and the third output channel of the second convolution, and (b) will prune the first output channel of the first convolution, the third output channel of the second convolution, and the first input channel of the second convolution. The objective values for each pruned networks are (a) 18 and (b) 21, respectively.
This shows that the coupling effect of the channels in neighboring layers directly affects the objective values, and finally results in a performance gap between (a) and (b). We call this coupling relationship as the quadratic coupling between the neighboring layers and formulate the contributions to the objective by quadratic terms of neighboring channel activations. To address this quadratic coupling, propose channel pruning method based on the QCQP framework with importance evaluation respecting both the input and the output channels.
3 METHOD
In this section, we first propose our discrete QCQP formulation of channel pruning for the sequential convolutional neural networks (CNNs). Then, present extended version of our formulation for joint channel and shape pruning of 2D convolution filters. The generalization to the nonsequential convolution (skip addition and skip concatenation) is introduced in Supplementary material A.
3

---

Under review as a conference paper at ICLR 2021
3.1 FORMULATION OF CHANNEL PRUNING FOR SEQUENTIAL CNNS
To capture the importance of weights in W we define the importance tensor as 7(0 E R XCiXKIX, Following the protocol of Han et al. (2015); Guo et al. (0016), we set Then, define the binary pruning mask as A E {0, _ ( For channel pruning in sequential
7 n | where "V is the & normalizing factor in 1-th layer or ||vee
CNNs, we define channel activation r()) E {0, 1) to indicate which indices of channels remain in the (-th layer of the pruned network. Then, the weights in W are active if and only if r
(1-1), (1)
[3 1,
which leads to A = = (1111) (1) JK. For example, in Figure 2b r!l-1)
[1, 1J. ,0) [0. 1JT,
0
17
0 0
and r (1+11) = [1, 1, T,, therefore, A = 0 8 JR, and All+1)
2 JK
1 1 0
1
We wish to directly maximize the sum of the importance of active weights after the pruning procedure under given resource constraints 1) FLOPs, 2) memory, and 3) network size. Concretely, our optimization problem is 2
maximize r(0:L)
T())
(2)
1
Σ a ||-00/
+Σ by ||,00, SM
subject to
1 (())),())) @ Jk.
,()) (0, 10
In our formulation, the actual resource usage of the pruned network is exactly computed by specifying the number of channels in the pruned network (= [|r(()]]1) and the pruning mask sparsity (= ||
in each layer. Concretely, the left hand side of the inequality in the first constraint in Equation 2
indicates the actual resource usage. Table 1 shows az, by terms used for computing usage of each resource. Note that this optimization problem is discrete nonconvex QCQP of the channel activations (r((0), ,..., r(D)), where the objective, which the same with the objective in Section 2.2 respects the quadratic coupling of channel activations (= r()).. Please refer Supplementary material E for the details on the standard QCQP form of Equation
3.2 FORMULATION OF JOINT CHANNEL AND SPATIAL PRUNING
For further efficiency, we increase the prun- ing granularity additionally perform spatial pruning in 2-D convolution filters. Concretely, we prune by each weight vector across the put channel direction instead of each channel to perform channel and spatial pruning pro- cesses simultaneously.
Resource constraint (M)
04
bi
Network size
0
Memory
H,W,
1
FLOPs
0
H,W;
First, we define the shape column W(() J.a.b by az. by values. the vector of weights at spatial position (a, b) of 2-D convolution filter along the j-th output channel dimension. Then, we define shape column activation q()) E {0. 1] xxxX Ki K2 to indicate which shape columns in the I-th convolution layer remain in the pruned network. Figure 3 shows the illustration of each variables.
Table 1: Resource constraints and the corresponding
Note that this definition induces constraints on the channel activation variables. In detail, the j-th output channel activation 1-th layer set if and only at least one shape column activation in the j-th output channel is set. Concretely, the new formulation should include the constraints (I) Ea. 2,0, and 9,,b Sr) Va,
2 denotes the outer product of tensors and Jn, is a n-by-n matrix of ones.
4

---

Under review as a conference paper at ICLR 2021
21111 E 1/
, e 10,
Figure 3: Input channel activation (= (=)(-))), shape column activation (= D)) and the correspond- ing mask A for I-th convolution layer, where A ((-l)) & qd)
We now reformulate the optimization problem to include the shape column activation variables. Again, we aim to maximize the sum of the importance of active weights after pruning under the given resource constraints. Then, our optimization problem for simultaneous channel and spatial pruning is
maximize (0:4) (
100, A
(3)
subject to
al |000
+ Σ
A
S M
(1)
ja.b and
Vl,i,a,
400 , (H1)
1I
(0 E {0, 13 C and ② E {0, CixKiXk A W (L].
We note that this optimization problem is also a discrete nonconvex QCQP. The details on the standard QCQP form of Equation ( is provided in Supplementary material E. Furthermore, Proposition below shows that the constraints in Equation 2 and Equation (3) provably eliminate any unprumed inactive weights and accurately model the resource usage well as the objective of the pruned network. Also, Proposition Cean generalized to nonsequential networks with skip addition. The generalization and the proofs given in Supplementary material D.
Proposition 1. Optimizing over the input and output channel activation variables (0:L) and shape column activation variables (1:L) under the constraints in Equation (3) provably removes any inactive weights in the pruned network, guaranteeing exact computation of resource usage and 2) the sum of the importance of active weights in the pruned network.
3.3 OPTIMIZATION
Concretely, Equation and Equation (3) fall into the category of binary Mixed Integer Quadratic Constraint Quadratic Programming (MIQCQP). We solve these discrete QCQP problems with the CPLEX library INC. 1993). which provides MIQCQP solvers based on the branch and cut technique. However, the branch and cut algorithm can lead exponential search time (Mitchell] 2002) large problems. Therefore, we provide a practical alternative utilizing a block coordinate descent style optimization method in Supplementary material B.

---

Under review as a conference paper at ICLR 2021
4 RELATED WORKS
Importance of channels Most of the channel pruning methods prune away the least important channels with a simple greedy approach, and the evaluation method for the importance of channels has been the main research problem (Molchanov et al., 2017; 2019; pruning is divided into two major branches according to the method of evaluating the importance of channels: the cettnaabl-importanee
Liu et al. 2019). Channel
method, which evaluates the importance of channels while training the whole network from scratch, and the fxed-importance method, which directly evaluates the importance of channels on the pretrained network. Trainable-impont importance channel pruning methods include the regularizer-based methods with group sparsity regularizers (Wen et al. [2016] Alvarez &
Salzmann 2016 Yang et al., 2019; Liu et al. 2017 Louizos et al. 2018 Liu et al., 201 7| Gordon
2018) and data-driven channel pruning methods (Kang & Han 2020 You et al. (019)) Fixed-
importance channel pruning methods first prune away most of the weights and then finetune the
significantly smaller pruned network (Molchanov et al. 2017: 2019: Hu al. 2016 He al.) [018a
Li et al.) 2017: He et al.) 2019 Luo et al. 2017). As a result, tixed-importance methods are much more efficient than the trainable-ing importtteeeeeeee channel pruning methods in terms of computational methods have to train the whole unpruned network. Our framework is on the line of tixed-importane channel pruning works.
cost and memory as trainable-importaan
Predefined target structure and Automatic target structure Layer-wise channel pruning meth- ods (Li et al. 2017: He et al. 2019), which perform pruning operations per each layer independently, require users to predefine the target pruned structure. Also, LCCL et al. 2017) exploits a predefined low-cost network to improve inference time. Another line of research finds the appropriate
target structure automatically (He et al.) [2018b Yang et al.) [018)] Liu et al. 2019 Molchanov et al. 2017). Our method also finds the target structure automatically under the explicit target resource constraints.
Dynamic pruning and static pruning Dynamic pruning has a different network structure depend- ing on the input during inference time, while static pruning has fixed network structure during inference time. CGNet (Hua et al. 2019) dynamically identifies unnecessary features to reduce the computation, and FBS (Cao et al. (019)) dynamically skip computations on the unimportant channels. Our framework is static pruning and has a fixed network structure during inference time.
Quadratic coupling CCP (Peng et al. 2019) formulates a QP (quadratic formulation) consider the quadratic coupling between channels in the same layer under layer-wise constraints on the maximum number of channels. On the other hand, our formulation considers the quadratic coupling between channels in the neighboring layers under the target resource constraints.
Channel pruning in nonsequential blocks Many network architectures contain nonsequential convolution operations, such as skip addition (He et al.] 2016; [Snnllr et al., 2018 Tan & Le 2019) and skip concatenation (Huang et al.) 2017). Since these network architectures outperform sequential network architectures, pruning network with nonsequential convolution operations is crucial. However, most channel pruning methods (Liu al. 2017 He al., 2018a; 2019 Molchanov et al. 2017 2019) do not consider the nonsequential convolutionn operations and use the same method from the sequential network architecture. However, channel pruning methods ignorant of nonsequential convolution operations may result in the misalignment of feature maps connected by skip connections (You et al. 2019). GBN Youu et al. (219)) forces parameters connected by a nonsequential convolution operation to share the same pruning pattern to solve this misalignment problem. In contrast, our formulation does not require strict pattern sharing. This ficxibility allows for our methods to delete more channels under given constraints.
Spatial pruning of convolution filters Spatial pruning methods aim to prune convolution filters along the channel dimension for inference efficiency. Spatial pruning methods manually define the spatial patterns of filters (Lebedev & Lempitsky 20016) Anwar et al. (017) or optimize spatial patterns of filters with group sparse regularizers (Wen et al. 2016: Lebedev & Lempitsky (016). Among these works, Lebedev & Lempitsky (2016) empirically demonstrates that enforcing sparse spatial patterns in 2-D filters along the input channel leads great speed-up during inference time

---

Under review as a conference paper at ICLR 2021
using group sparse convolution operations (Chellapilla et al. 2006). Our proposed method enforces
the spatial patterns in 2-D filters as in Lebedev Lempitsky (2016) for speed-up in inference.
EXPERIMENTS
We compare the classification accuracy of the pruned network against several pruning baselines on CIFAR-10 (Krizhessky et al.] 2009) and ImageNet (Kussakovsky al. (015) datasets using various ResNet architectures (He et al. 2016). DenseNet-40 (Huang et al. | 201 7p, and VGG-16 (Simonyan) & Zisserman 2015). Note that most pruning baselines apply an iterative pruning procedure, which repeatedly alternates between network pruning and finetuning until the target resource constraints are
satisfied (Han al. 2013 He et al., 2018a; Liu al. 2017 Yang al., 2018). In contrast, since our methods explicitly include the target resource constraint to the optimization framework, we only need one round of pruning and finetuning.
5.1 EXPERIMENTAL SETTINGS
We follow the noraller-oom--lss- wernn--ess---mpppe tesssiippotaat' criterion Ye al. 2018 Liu al. 2017). which evaluates the importance weights as the absolute value of weight (Han et al. 2015;] Guo et al. 2016). We assume the network size and FL.OPs reduction are linearly proportional to the sparsity in shape column activations, as empirically shown in Lebedev & Lempitsky (2016). Also, we ignore the extra memory overhead for storing the shape column activations due its negligible size compared to the total network size. In the experiment tables, FLOPs and the network size of the pruned network are computed according to the resource specifications in Equations 2 and (3). Also, 'Pruning ratio' in the tables denotes the ratio of pruned weights among the total weights in baseline networks. 'ours-c'
and 'ours-es' in the tables denote our method with channel pruning and our method with both the channel and spatial pruning, respectively.
Network
Pruning ratio(S:
Network
Method
Baseline
Pruned
Acc dropl
FLOPs(S))
Pruned acc
Ace drop1
ResNet-20
FPGM He 2019
92.21 (0.18)
91.26 (0.24)
0.95
46.0
91.26 (0.24)
0.95
54,0
ours-c
92.21 18)
90.96 (0.15)
1:25
46.0
91.26 (0.18)
0.95
54.1
ours-cs
(0.18)
(0.18)
92,02 (0.10)
LCCL (Dong et af 2017
92.74
91.68
1.06
62.1
91.68
1.06
33.1
STP (1l
20180
92.20 (8)18)
90.83 (0.31)
1.37
57.8
90.83 (0.31)
1.37
42.2
FPGMT [ee
92.21 (0.18) 18)
(0.20)
0.49
(0.20)
ours-c
92.21 (0.18)
91.74 (0.20)
0.47
58.5
92:27 (0.17).
-0.06
42.3
92.21
92,26 (0.10)
-0.05
$7.8
92.35 (0.10)
-0.14
ResNet-32
FPGM
He et al. 2019
92.88 (0.86)
91.96 (0.76)
0.92
46.8
91.96 (0.76)
0.92
53,2
oun--c
92.88 (0.86)
91.98 (0.42)
0.90
46.8
92.22 (1.02)
0.66
53.2
(0.86)
92.33 (0.41)
0.55
92.78 (0.97)
LCCL [Done et al 2017
92.33
90.74
1.59
69.0
90.74
1.59
37.5
SFP el 2013
92.63 (0.70)
92.08 (0.08)
0.55
58.5
92.08 (0.08)
0.55
41.5
[10
(0.86)
(0.90)
(0.90)
ours-c
92.88 (0.86)
92.52 (0.46)
0.36
$7.2
92.42 (0.77)
0.46
42.7
ours-cs
92.88 (0.86)
92.80 (0.61)
0,08
57.9
92.83 (0.83)
0.05
42.7
ResNet-56
SFP et
2018a
93.59 58)
92,26 (0.31)
1.33
47.5
92.26 (0.31)
1,33
$2.6
FPGMT He et al.
2019
93.59 (0.58)
93.49 (0.13)
0.10
47.5
93,49 (0.13)
0.10
52.6
CCP
PPny
93.50
93.42
0:08
47.4
AMC
He m
A0180
92.8
91.9
0.9
50.0
SCP
King K Fim
21:0
93.69
93.23
0.46
48.5
93.23
0.46
51.5
93.59 (0.58)
(0.68)
93.37 (0.96)
ours-cs
93.59 (0.58)
93,59 (0.36)
0.00
47.4
93,69 (0.69)
-0.10
52.6
Dense Net-40
SCP Kang & Ham 2020
94.39
93.77
0.62
29.2
1.21
ours-cs
95.01
94.25
0.76
29.2
slimming [Liu ours-c
2017
93.89
94.35
-0.46
45.0
0.63
oury cs
95.01
94.85
0.16
45.0
slimming (Luu
93.89
94.81
-0.92
71.6
ours-c
95.01
94.82
0.19
71.0
ours- cs
95.01
95.02
-0.01
71.0
Table 2: Pruned accuracy and accuracy drop from the baseline network at given FLOPs (left) and pruning ratios (right) on various network architectures 622,,2e and DenseNet-40) at CIFAR-10.
7

---

Under review as a conference paper at ICLR 2021
5.2 CIFAR-10
CIFAR-10 10 dataset has 10 different classes with 5k training images and 1k test images per each class Krizhevsky al. 2009). In CIFAR- experiments, we evaluate our methods various network architectures; ResNet-20, 32, 56, and DenseNet-40. We provide implementation of the details for the experiments in Supplementary material C. We show the experiment results of pruning under FLOPs constraints in the left column of Table and under final network size constraints in the right column of Table
On the FLOPs experiments in the left column of Table 2 FPGM, which the previous state of the art method, on ResNett-20, 32, and 56. Moreover, 'ours-cs' significantly outperforms both 'ours-c" and FPGM the same architectures showing the state of the art performance. Also, 'ours-c" shows comparable results against slimming (Liu et al.] (017) and SCP (Kang & Han 2020), while 'ours-cs' outperforms existing baselines by large margin DenseNet-40.
'ours-c' shows comparable results against
On the network size experiments in the right column of Table 'ours-c' shows results competitive to FPGM and SCP, while 'ours-cs' again achieves the state of the art performance on ResNet-20,, 32, and 56. Notably, in ResNet-56, 'ours-cs' achieves minimal accuracy drop of -0.10 with the pruning ratio 52.6%. These results show simultaneous channel and spatial pruning produces more efficient networks with better performance compared other channel pruning methods CIFAR-10.
5.3 IMAGENET
ILSVRC-2012 (Russakovsky et al. 2015) is a large-scale dataset with 1000 classes that comes with 1. M.28M training images and 50k validation images. We conduct our methods under the fixed FL.OPs constraint on ResNet-18.50 and VGG-16. For more implementation details of the ImageNet experiments, refer to Supplementary material C. TToble]]sowss the experiment results on ImageNet. In ResNet-50,, 'ours-c' and 'ours-cs' achieve results comparable to GBN, trainable- importance channel pruning method which is the previous state of the art, even though our method is a fixed- importance channel pruning method. In particular, topl pruned accuracy in 'ours-cs' exceeds SFP by 1. 1.32% using a similar number of FLOPs. Both 'ours-cs' and 'ours-e' clearly outperform FPGM in ResNet-50. Also, 'ours-c' and 'ours-es" show significantly better performance than Molchanov et al. (017) on VGG-16.
Network
Method SFP ee 2018a FPGM [Te
Topl Pruned Ace"
Topl Acc drop..
Top5 Pruned Accf
Top5 Acc drop)
FL.OPs(%))
R8sNet-18
67.10
3.18
87.78
1.85
2019
68.41
1.87
88.48
1.15
58.2
curs-c
67.48
2,28
87.78
1.30
60.9
0.17
88.94
0.14
58.2
LCCL (Dong et al. 2017
66.33
3.65
86.94
2.29
65.3
1.11
88.69
0.39
ours-2s
70.05
-0.29
89.24
-0.16
65.9
ResNet-50
SIP He a 20
74.61
1.54
92.06
58.3
FPGNT IR et 2019
75.50
0.65
92.63
0.21
57.8
ours-c
75.78
0.37
91.86
1.11
57.8
75.93
0.22
92.68
0.19
57.8
GBN You et al 2019
76.19
-0.31
92.5 83
-0.16
59.5
75.89
0.26
92.84
0.03
ours-cs
76.00
0.15
92.76
0.11
$9.0
ours-c
65 92
5.67
87.20
3.18
51.7
0urs-cs
87.36
Table Top1,5 pruned accuracy and accuracy drop from the baseline network at given FLOPs on various network architectures (ResNet-18.50 and VGG-16) at ImageNet.
CONCLUSION
We present discrete QCQP based optimization framework for jointly pruning channel and spatial filters under various architecture realizations. Since our methods model the inherent quadratic coupling between channels in the neighboring layers to eliminate any inactive weights during the
8

---

Under review as a conference paper at ICLR 2021
pruning procedure, they allow exact modeling of the user-specified resource constraints and enable the direct optimization of the true objective on the pruned network. The experiments show our proposed method significantly outperforms other fixed ixxe-importance channel pruning methods, finding smaller and faster networks with the least drop in accuracy.
REFERENCES
Jose M Alvarez and Mathieu Salzmann. Learning the number of neurons in deep networks. In NeurIPS, 2016.
Sajid Anwar, Kyuyeon Hwang, and Wonyong Sung. Structured pruning of deep convolutional neural networks. In JETC, 2017.
Kumar Chellapilla, Sidd Puri, and Patrice Simard. High performance convolutional neural networks for document processing. In /WFIR, 2006.
Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. Attention-based models for speech recognition. In NeurlPS. 2015.
J. Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
Xuanyi Dong, Junshi Huang, Yi Yang, and Shuicheng Yan. More is less: A more complicated network with less inference complexity. CVPR, 2017.
Xitong Gao, Yiren Zhao, Lukasz Dudziak, Robert Mullins, and Cheng-zhong Xu. Dynamic channel pruning: Feature boosting and suppression. 2019.
Ariel Gordon, Elad Eban, Ofir Nachum, Bo Chen, Hao Wu, Tien-J Yang, and Edward Choi. Morphnet: Fast & simple cesource-constrainee structure learning of deep networks. In CVPR, 2018.
Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dnns. In NeurIPS. 2016.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. NeurlPS, 2015.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Yang He, Guoliang Kang. Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. In IJCAI, 2018a.
Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In CVPR, 2019.
Yihui He, Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. Ame: Automl for model compression and acceleration on mobile devices. In ECCV, 2018b.
Hengyuan Hu, Rui Peng, Yu- fu-Wing Tai, and Chi-Keung Tang. Network trimming: A data-driven neuron pruning approach towards efficient deep architectures. In 107.002600. 2016.
Weizhe Hua, Yuan Zhou, Christopher M De Sa, Zhiru Zhang, and G Edward Suh. Channel gating neural networks. In NIPS, 2019.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. In CVPR, 2017.
CPLEX OPTIMIZATION INC. Using the cplex callable library and cplex mixed integer library. In Incline Village, NV, 1993.
Minsoo Kang and Bohyung Han. Operation-aware soft channel pruning using differentiable masks. In ICMI., 2020.

---

Under review as a conference paper at ICLR 2021
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. In Tech Report, 2009.
Vadim Lebedev and Victor Lempitsky. Fast convnets using group-wise brain damage. In CVPR, 2016.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In ICLR, 2017.
Z. Liu, H. Mu, X. Zhang, Z. Guo, Yang, K. Cheng, and J. Sun. Metapruning: Meta learning for automatic neural network channel pruning. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
Zhuang Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In ICCV, 2017.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In ICLR, 2019.
Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through 1_ regularization In ICLR, 2018.
Jian-Hao Luo. Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. ICCV, 2017.
John E Mitchell. Branch-and-cut algorithms for combinatorial optimization problems. In Handbboo of applied optimization, 2002.
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. In ICLR, 2017.
Pavlo Molchanov, Arun Mallya, Stephen Tyree, Turi Frosio, and Jan Kautz, Importance estimation for neural network pruning. In CVPR, 2019.
Hanyu Peng, Jiaxiang Wu, Shifeng Chen, and Junzhou Huang. Collaborative channel pruning for deep networks. In ICML, 2019.
Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. In arXiv:/804.02767. 2018.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. In IJCV, 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML,L 2019.
Stefan Van Der Walt, S Chris Colbert, and Gael Varoquaux. The numpy array: structure for efficient numerical computation. In CISE, 2011.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In NeurlPS, 2016.
Huanrui Yang. Wei Wen, and Hai Li. Deephoyer: Learning sparser neural network with differentiable scale-invariant sparsity measures. In ICLR, 2019.
Tien-Ju Yang, Andrew Howard, Bo Chen, Xiao Zhang, Alec Go, Mark Sandler, Vivienne Sze, and Hartwig Adam. Netadapt: Platform- aware neural network adaptation for mobile applications. In ECCV, 2018.
10

---

Under review as a conference paper at ICLR 2021
Jianbo Ye, Xin Lu, Zhe Lin, and James Z Wang. Rethinking the conlll--noom--ess-inoonaaaad assumption in channel pruning of convolution layers. In ICLR, 2018
Zhonghui You, Kun Yan, Jinmian Ye, Meng Ma, and Ping Wang. Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks. In NeurfPS, 2019.


Reviews:
[{"review_id": "tFCLoJiWefe", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper mainly improves the idea of \"PRUNING FILTERS FOR EFFICIENT CONVNETS\" by encouraging the pruning with a {0-1} optimization instead of a greedy manner. Experiments validate the effectiveness of the proposed method. \n\nPros: \n+ Writing is good, and the technical details seem sound and clear.\n+ The motivation makes sense.\n\nCons:\n- The novelty is limited. The formulation of the 0-1 optimization for pruning is simple and intuitive. Concretely, it leverages the pre-trained weights for the unpruned network and tries to select the kernels with the maximum magnitude. For me, I am not sure whether the novelty is up to the standard of ICLR venue. \n- The objective is to maximize the norm of selected filters. However, magnitude-based pruning is already challenged for it is not accurate to indicate the selection. \n- Authors claim that current pruning papers can not reach a strict constraint for FLOPs during pruning. However, it is not true for recent pruning methods, such as AutoSlim, TAS and MetaPruning. Necessary discussions are needed. \n- Pruning on recent compact networks is favoured, such as MobileNetV2, which is also a routine network for many pruning papers. \n\n[ICLR2017] PRUNING FILTERS FOR EFFICIENT CONVNETS  \n[2019] AutoSlim: Towards One-Shot Architecture Search for Channel Numbers\n[ICCV2019] MetaPruning- Meta Learning for Automatic Neural Network Channel Pruning.pdf\n[NIPS2019] Network Pruning via Transformable Architecture Search", "extra_scores": {}, "extra_text": {}}, {"review_id": "mxEswI8Ggxi", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces an optimization method for pruning channels in networks. The authors first motivated the proposed approach by showing that current pruning methods will result in \"inactive weights\" for the following layer.  Then the authors introduce a QCQP optimization method that can constrain the exact amout of resources during the optimization process. Extensive experiments are conducted on different benchmarks with different backbones. And the authors also performed spatial pruning to further reduce resource usage.\n\n\n####### Strengths######\n+ The motivation is clear and the presentation is generally good.\n+ The  idea of mitigating the effect of inactive weights is interesting.\n+ Extensive studies have been conducted in terms as different datasets/backbones.\n\n####### Weakness######\n- The term \"the inherent quadratic coupling\" used in the abstract is a bit confusing without any explannations.\n- I didn't quite follow section 2.2, where the authors discussed the quadratic coupling effect. In figure 2, I understand the prunned channels for the greedy part. But I don't quite get how the proposed approach is able to prune the last channel of the 2nd layer. It would be nice to discuss this when the optimization is introduced?\n- Following the previous point, the authors basically are saying QCQP is better than greedy pruning. I couldn't find experimentd comparing these two methods. I mean I understand previous methods are greey based. But it would be nice to have the same implementation by the authors for apple-to-apple comparisons.\n- Missing references\n[1]Rethinking the Value of Network Pruning\n[2] Channel Gating Neural Networks\n[3] Dynamic Channel Pruning: Feature Boosting and Suppression\n\n\n##############Post Rebuttal###############\n\nMy concerns are addressed by the authors. I'm keeping my original rating.", "extra_scores": {}, "extra_text": {}}, {"review_id": "hpDNAcfGwwh", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary：\nIn this manuscript, a new pruning method is proposed by considering the inherent quadratic constraint between consecutive layers. Without this constraint, inactive weights cannot be safely removed. Even with the same objective function, the optimized result is different, as shown in the motivation section. Based on this observation, the pruning task is models as a QCQP optimization problem. And a faster algorithm to solve this problem is proposed. Moreover, the pruning on filter size can also be modeled as the QCQP problem, making the pruning on both channel and filter size feasible.\n\nStrengths:\n-\tThe paper is well-written and well-motivated. The motivation is reasonable and the proposed method does alleviate the overlooked issue. \n-\tThe results on the CIFAR10 and ImageNet surpass some previous methods.\n-\tThe proposed method does not need iterative pruning procedure as other methods, making it simple to use.\n-\tThe proposed motivation may inspire following works in this area.\n\nWeaknesses:\n-\tI am not an expert in the area of pruning. I think this motivation is quite good but the results seem to be less impressive. Moreover, I believe the results should be evaluated from more aspects, e.g., the actual latency on target device, the memory consumption during the inference time and the actual network size.\n-\tThe performance is only compared with few methods. And the proposed is not consistently better than other methods. For those inferior results, some analysis should be provided since the results violate the motivation.\n\nI am willing to change my rating according to the feedback from authors and the comments from other reviewers.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "g2cLyBv_lV7", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors proposed a pruning method that aims to reduce the parameters and heavy computational cost of large convolutional neural networks (CNNs). According to the comparison experiments performed on several widely-used network fashions, the proposed strategy could help to efficiently reduce the numbers of parameters while maintaining less performance decreasing. I think this study is valuable in both theory and applications. \nHowever, several issues may be further emphasized to make the submission improved:\n\n(1) Were the CNN models deployed in the real resource-constrained environment such as automatic drive hardware with less computational capability? Or could this proposed strategy be applied to reduce the parameters of the CNN models to the level of MobileNet?\n\n(2) Authors determined that the proposed method is useful to tackle several classification tasks, did this method also perform well on the CNN models aimed at segmentation, detection et al. Performance of this kind of models may decrease more than the classification. \n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed a new optimization framework for pruning CNNs considering coupling between channels in the neighboring layers. Two reviewers suggested acceptance and two did rejection. The main concerns of the negative reviewers are (a) limited novelty, (b) limited performance metrics and (c) limited baselines. The authors' response did not fully clarify the reviewers' concerns during the discussion phase, and AC also agrees that they should be resolved to meet the high standard of ICLR. Hence, AC recommend rejection.

Here is additional thought from AC. The authors propose ours-c and ours-cs. The latter is reported to outperform the former in terms of FLOPs, but AC thinks the former may have merits in other more important performance metrics, e.g., the actual latency and/or memory consumption on a target device. More discussions and results for this would strengthen the paper.
Decision: Reject