Structural Pruning via Latency-Saliency Knapsack
Authors: Maying Shen
Hongxu Yin
Pavlo Molchanov
Lei Mao
Jianna Liu
Jose M. Alvarez
Venue: NeurIPS 2022 Accept (2022)
Keywords: model compression
deep neural network pruning
latency reduction

Abstract: Structural pruning can simplify network architecture and improve inference speed. We propose Hardware-Aware Latency Pruning (HALP) that formulates structural pruning as a global resource allocation optimization problem, aiming at maximizing the accuracy while constraining latency under a predefined budget on targeting device. For filter importance ranking, HALP leverages latency lookup table to track latency reduction potential and global saliency score to gauge accuracy drop. Both metrics can be evaluated very efficiently during pruning, allowing us to reformulate global structural pruning under a reward maximization problem given target constraint. This makes the problem solvable via our augmented knapsack solver, enabling HALP to surpass prior work in pruning efficacy and accuracy-efficiency trade-off. We examine HALP on both classification and detection tasks, over varying networks, on ImageNet and VOC datasets, on different platforms. In particular, for ResNet-50/-101 pruning on ImageNet, HALP improves network throughput by $1.60\times$/$1.90\times$ with $+0.3\%$/$-0.2\%$ top-1 accuracy changes, respectively. For SSD pruning on VOC, HALP improves throughput by $1.94\times$ with only a $0.56$ mAP drop. HALP consistently outperforms prior art, sometimes by large margins. Project page at \url{https://halp-neurips.github.io/}.

Structural Pruning via Latency-Saliency Knapsack
Maying Shen, Hongxu Yin," Pavlo Molchanov, Lei Mao, Jianna Liu, Jose M. Alvarez
NVIDIA
:mshen, dannyy, pmol chanov, Imao, jiannal, josea) anvidia... com
Abstract
Structural pruning simplify network architecture and improve inference speed. We propose Hardware-Awarr Latency Pruning (HALP) that formulates structural pruning a global resource allocation optimization problem. aiming at maximiz- ing the accuracy while constraining latency under predefined budget on targeting device. For filter importance ranking, HALP leverages latency lookup table track latency reduction potential and global saliency score to gauge accuracy drop. Both metrics be evaluated very efficiently during pruning, allowing late global structural pruning under reward maximization problem given target constraint. This makes the problem solvable via augmented knapsack solver, enabling HALP to surpass prior work in pruning efficacy and accuracy-efficiene trade-off. We examine HALP on both classification and detection tasks, over vary- ing networks, on ImageNet and VOC datasets, on different platforms. In particular, for ResNet-500-100 101 pruning on ImageNet, HALP improves network throughput by 1.60x/1.90x with -..%%%%%%% top-1 accuracy changes, respectively. For SSD pruning on VOC, HALP improves throughput by 1.94x with only 0.56 mAP drop. HALP consistently outperforms prior art, sometimes by large margins. Project page at hhttps://halp-neeeee wtt..ppp-nnurpps...hu....... github. 1o/
reformu-
Introduction
Convolutional Neural Networks (CNNs) act as the central tenet behind the rapid development in computer vision tasks such as classification, detection, segmentation, image synthesis, among others. As performance boosts, so do model size, computation, and latency. With millions, sometimes billions of parameters (e.g., GPT-3 deployment, that mostly faces stringent constraints such as energy and latency [9 46 45 [55. In certain cases like autonomous driving, a breach of real-time constraint not only undermines user experience, but also imposes critical safety concerns. Even for cloud service, speeding up the inference directly translates into higher throughput, allowing more clients and users benefit from the service.
modern neural networks face increasing challenges upon ubiquitous
One effective and efficient method reduce model complexity is network pruning. The primary goal of pruning is to remove the parameters, along with their computation, that are deemed least important for inference
20
37 45 Compatible with other compression streams of work such as
quantization [5] 64 773) dynamic inference [3]]
67 (OO). and distillation 23
47 69 pruning enables
flexible tuning of model complexity towards varying constraints, while requiring much less design efforts by neural architecture search [58
61 (55 and architecture re-designs [24 42 59 Thus, in this
work, we study pruning, in particular structured pruning that reduces channels benefit off the-thh-hhef platforms, e.g., GPUs.
Eqqual contribution.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

14
Lyo
teaalene
Conmaccsoo
0J
66
8.00
Kropsack
pruned fllters
terpet fitters
colliasin
grouphy
(very fast)
network global filter profile
filter selection per target constraint
Figure I:
The proposed hardware-awan latency pruning (HALP) paradigm. Considering both perfor-
mance and latency contributions, HALP formulates global structural pruning as global resource allocation problem (Section 3.T)p solvable using our augmented Knapsack algorithm (Section 3.2). Pruned architectures surpass prior work across varying latency constraints given changing network architectures for both classification and detection tasks (Section
Topt AostN)
75
73
4.5
inference time (mslmage)
FLOPs (G)
Original mode
ya RCCVSO
Autosiim
ExPRA CUPRIE
McssPraningg Ua et ICCV19 ....... GReg Wang et ICLR21
CHP et NourPP/21
...d... GBN You ct Nouri3119
OOOP Joong et NSTATS82 a HALP (Durs)
HALP cisiiiationn (Durs)
Figure 2: Pruning ResNet50 on the ImageNet dataset. The proposed HALP surpasses state-of-the-art structured pruning methods over accuracy, latency, and FLOPs metries. Target hardware NVIDIA Titan V GPU. Top-left is better.
As the pruning literature develops, the pruning criteria also evolve to better reflect final efficiency. The early phase of the field focuses maximum parameter removal in seek for minimum representations of the pretrained model. This leads to a flourish of approaches that rank neurons effectively to measure their importance
46 As each neuron/filter possesses intrinsically different computation, following works explore proxy to enhance redundancy removal, FLOPs being one of the most widely adopted metrics 66 to reflect how many multiplication and addition computes needed for the model. However, for models with very similar FLOPs, their latency can vary significantly [58]] Recently,
and more works start directly working reducing latency K 68 55 However,
much was done the field of GPU friendly pruning methods due to non-trivial latency- architectue trade-off. For example, recently observed in [51]. GPU usually imposes staircase-shaped latency patterns for convolutional operators with varying channels, which inevitably occur per varying pruning rate, the latency surface in Fig. D This imposes constraint that pruning needs be done in groups to achieve latency improvement. Moreover, getting the exact look-up table of layers under different pruning configurations will benefit maximizing performance while minimizing latency.
Pruning different layers the deep neural network will result in different accuracy-latenas trade-off. Typically, removing channels from the latter layers has smaller impact accuracy and smaller impact on latency versus removing channels from the early layers. We ask the question, is better to remove more neurons from latter layer or less from early layer achieve the same accuracy securacy-latency trade-off. By nature, the problem is combinatorial and requires the appropriate solution.
In this paper, we propose hardware-aware latency pruning (HALP) that formulates pruning resource allocation optimization problem maximize the accuracy while maintaining a latency budget on the targeting device. The overall workflow is shown in Fig. I For latency estimate per

---

pruned architecture, we pre-analyze the operator-level latency values by creating look-up table for every layer of the model on the target hardware. Then we introduce an additional score for each neuron group to reflect and encourage latency reduction. To this end, we first rank the neurons according to their importance estimates, and then dynamically adjust their latency contributions. With neurons re-calibrated towards the hardware-aware latency curve, we now select remaining neurons to maximize the gradient-basek importance estimates for accuracy, within the total latency constraint. This makes the entire neuron ranking solvable under the knapsack paradigm. To enforce the neuron selection order a layer to be from the most important to the least, we have enhanced the knapsack solver so that the calculated latency contributions of the remaining neurons would hold. HALP surpasses prior art in pruning efficacy, see Fig. 2 and the more detailed analysis in Section Our main contributions are summarized as follows:
We propose a latency-driven structured pruning algorithm that exploits hardware latency traits to yield direct inference speedups.
We orient the pruning process around a quick yet highly effective knapsack scheme that seeks for a combination of remaining neuron groups to maximize importance while constraining to the target latency.
We introduce group size adjustment scheme for knapsack solver amid varying latency contri- butions across layers, hence allowing full exploitation of the latency landscape of the underlying hardware.
We compare to prior art when pruning ResNet, MobileNet, VGG architectures on ImageNet, CIFARIO, PASCAL VOC and demonstrate that our method yields consistent latency and accuracy improvements over state-of-the the-art methods. Our ImageNet pruning results present viable 1.6x to 1 1.9x speedup while preserving very similar original accuracy of the ResNets.
2 Related work
Pruming methods. Depending on when to perform pruning, current methods can generally be
divided into three groups [14] i) prune pretrained models 20
33
45
16] ii) prune
initialization 1
30 T0 and iii) prune during training 17 11
Despite notable progresses in the
later two streams, pruning pretrained models remains as the most popular paradigm, with structural sparsity favored by off- tff-the-shelf inference platforms such as GPU.
To improve inference efficiency, many early pruning methods trim down the neural network aiming to achieve a high channel pruning ratio while maintaining an acceptable accuracy. The estimation of neuron importance has been widely studied in literature 255
For example, 45 proposes to use Taylor expansion to measure the importance of neurons and prunes a desired number of least-ranked neurons. However, channel pruning ratio does not directly translate into computation reduction ratio, amid the fact that a neuron at different location leads to different computations.
40 26
There are recent methods that focus primarily reducing FLOPs. Some of them take FLOPs into consideration when calculating the neuron importance to encourage penalizing neurons that induce high computations [66], An alternative line of work propose to select the best pruned network from However, it would take a long time for candidate selection due to the large amount of candidates. In addition, these methods use FLOPs
set of candidates 32 68).
proxy of latency, which is usually
inaccurate as networks with similar FLOPs might have significantly different latencies [58]
Latency-aware compression. Emerging compression techniques shift attention to directly cut down on latency. One popular stream is Neural Architecture Search (NAS) methods g 12
58 6S that
adaptively adjusts the architecture of the network for given latency requirement. They incorporate the platform constraints into the optimization process in both the architecture and parameter space to jointly optimize the model size and accuracy. Despite remarkable insights, NAS methods remain computationally expensive in general compared to their pruning counterparts.
Latency- -oriened pruning has also gained growing amount of attention. [ presents a framework for network compression under operational constraints, using Bayesian optimization to iteratively obtain compression hyperparameters that satisfy the constraints. Along the same line, NetAdapt [68] iteratively prunes neurons across layers under the guidance of the empirical latency measurements on the targeting platform. While these methods push the frontier of latency constrained pruning, the hardware-ineurred latency surface in fact offers much more potential under our enhanced pruning policy show later, large rooms for improvements remain unexploited and realizable.
3

---

3 Method
In this section, first formulate the pruning process optimization process, before diving deep into the importance estimation for accuracy and latency. Then, we elaborate on how to solve the optimization via knapsack regime, augmented by dynamic grouping of neurons. We finalize the method by combining these key steps under one realm of HALP.
3.1 Objective Function
Consider a neural network that consists of I. layers performing linear operations on their inputs, together with non-linear activation layers and potentially pooling layers. Suppose there are Nj neurons (output channels) the la layer and each neuron is encoded by parameters WT E R x K2x Ki where CR is the number of input channels and K' is the kernel size. By putting all the neurons across the network together, we get the neuron parameter set W [(WT J J 1 where N Σt= Ni is the total number of neurons in the network.
Given training set D [ ys)) the problem of network pruning with given constraint C can be generally formulated as the following optimization problem:
mig L(W,D) s.t. + (((W,))) sc
where W C Wi is the remaining parameters after pruning and L is the loss of the task. f(.) encodes the network function, and ((.) maps the network to the constraint C, such as latency, FLOPs, or memory. We primarily focus latency in this work while the method easily scales other constraints.
The key to solving the aforementioned problem relies on identifying the portion of the network that satisfies the constraint while incurring minimum performance disruption:
arg XTLEX Σ Ipt), S.L. Σ Ti(p-1,,)) S. S P S Ni
[=1
where denotes the number of kept neurons at layer 1, (p()) signals the maximum importance to the final accuracy with Pt neurons, and T( Pt) checks on the associated latency contribution of layer / with pt-1 imput channels and output channels, po denotes fixed input channel number for the first convolutional block, e.g., 3 for RGB images. We next elaborate on ((-) and T(-) in detail.
Importance score. To get the importance score of layer to final accuracy, namely (pi) in Eq. 2 we take it as the accumulated score from individual neurons ΣØ= Zí. We first approximate the
importance of neurons using the Taylor expansion of the loss change 45 Specifically, we prune on batch normalization layers and the importance of the n-th neuron in the 1-th layer is calculated as
If' = =!g 1
where y denotes the gradient of the weight, 7 and 39 are the corresponding weight and bias from the batch normalization layer, respectively. Unlike squared loss as we observe slight improvements.
45
we use absolute difference
In order to maximize the total importance, we keep the most important neurons a higher priority. To this end, we rank the neurons in the ln layer according to their importance score in descending order and denote the importance score of the juh ranked neuron as Z/. thus we have
((p)) ΣI. T. 0S No, I
M.
Latency contribution. We empirically obtain the layer latency (P Pt) in Eq. 2 by pre-building a layer-wise look-up table with pre-measured latencies. This layer latency corresponds to the aggregation of the neuron latency contribution of each neuron in the layer, c/:
Pt.
Ti(p-11P))
Σ d. S
The latency contribution of the j-th neuron in the 1-th layer can also be computed using the entries in the look up table as:
d = 1i(pi-1.j) Ti(p--1.5 1), 1 15i5 pt.
(6)

---

and for Algorithm Augmented Knapsack Solver
Input: Impurtance (2) 2/ where sorted descendingly: Neuron cummibutio {c) RM
) Latency constriint C.
maxV E R((+1), keep (Cxx(C+)) p maxV[c] max importance under constrmin et koept). c] neurooss to keep in layer achieve max
2: for I 1,... L
3. for 1....,
for 1....,,
5: Wkeep Ti + maxVle of). Vpouee maxV[c]
> total importince achieve under constmint with object is being not impoctant layer kept
6
If Phoop Vppms kesplt, c/1
then
check if higher
7:
keep|t. = update maxV(c] =
8-
the end keop|t. keep(f. 11, update_masVic Upeuse
9:
10:
11:
max update_maxV
and
15:
16:
keep_r
the L.....1
in model
retrivve kepe ncurons
18:
Pi keeplt, C)
19:
keep + keep_n n U (pi ranked neurons in layer
20: C - C Dj=
Output: important neuroms (keep_n).
In practice, we first rank globally neurons by importance and then consider their latency contribution. Thus, can draw the following properties. If remove the least important neuron in layer 1, then the number of neurons will change from Pt to Pt - 1, leading to a latency reduction cf as this neuron's latency contribution score. We assign the potential latency reduction neurons in the layer by the importance order. The most important neuron in that layer would always have a latency contribution ot. At this stage, finding the right combination of neurons keep imposes combinatorial problem, and in the next section we tackle it via reward maximization considering latency and accuracy traits.
3.2 Augmented Knapsack Solver
Given both importance and latency estimates, we now aim at solving Eq. 2 layer importance Eq.
By plugging back in the
Fl and layer latency Eq, 5 we come to
L.
PL
L.
Pi
ΣΣ
ΣEdso, OSPEN. 2 MM
b1
This simplifies the overall pruning process into a knapsack problem only with additional preceding constraints. The preceding enforcement originates from the fact that for neuron with rank in the ln layer, the neuron latency contribution only holds when all the neurons with rank r r= 1..... 1 are kept in the 4, layer and the rest of the neurons with rank Yet the problem is solvable by specifying each item with a list of preceding items that need to be selected before its inclusion.
+ 1. ... .N₂ are removed.
We augment the knapsack solver to consider the reordered neurons with descending importance score so that all the preceding neurons will be processed before it. A description of the pscudo code of the augmented knapsack solver is provided in Algo. (a detailed explanation is provided in Appendix A The augmented solver is required to make sure that the latency cost is correct.
3.3 Neuron Grouping
Considering each neuron individually results in burdensome computation during pruning. We next explore grouping neurons that number of them can be jointly considered and removed enabling faster pruning [69] Neuron grouping helps exploit hardware-incuree channel granularity guided by the latency, speeds up knapsack solving of Eq. and yields structures that maximize the GPU utilization, keeping as many parameters as possible under the similar latency. In addition, neuron grouping simplifies the knapsack problem that scales linearly with the number of candidates under consideration (see Line 3 of Alg. L. thus speedups the solver as we'll discuss later.
We refer to the difference of neuron counts between two latency cliffs of the paircase-patternes latency as the latency step size. In our method, we group channels in a layer as an entirety, where the value of is equal to the latency step size. The neurons grouped by the order of importance.
5

---

Then we aggregate the importance score and latency contribution for the grouped entity. For skip connections in ResNet and group convolutions in MobileNet, we not only group neurons within layer, we also group the neurons sharing the same channel index from the connected layers [1 [13.. For cross-layer grouping, as the latency step size for different layers might be different, we use the largest group size among the layers. Latency-awan grouping enables additional performance benefits when compared to a heuristic universal grouping, as we will later show in the experiments.
3.4 Final HALP Regime
With all aforementioned steps, we formulate the final HALP as follows. The pruning process takes trained network input and prunes it iteratively satisfy the requirement of a given latency budget C. We perform one pruning every r minibatches and repeat it k pruning steps in total. In particular, we set k milestones gradually decreasing the total latency to reach the goal via exponential scheduler [1O), with C¹ > c²>... > C*, C = C. The algorithm gradually trims down neurons using steps below:
Step 1. For each minibatch, we get the gradients of the weights and update the weights as during the normal training. We also calculate each neuron's importance score as Eq.
Step 2. Over multiple minibatches
calculate the average importance score for each neuron
and rank them accordingly. Then we count the number of neurons remaining in each layer and dynamically adjust the latency contribution in Eq. 6
Step 3. We group neurons described in Sec. 3.3 and calculate group's importance and latency reduction. Then we get the nearest preceding group for each layer.
Step 4. We execute the Algo. [ to select the neurons being remained with current latency milestone. Repeat starting from the Step until & milestones are reached.
Once pruning finishes we fine-tune the network to recover accuracy.
4 Experiments
We demonstrate the efficacy and feasibility of the proposed HALP method in this section. We use ImageNet ILSVRC2021 5 for classification. We first study five architectures (ResNet50, on different platforms and compare our pruming results
ResNet101, VGG16 and MobileNet- with the of-te- art methods on classification task. All the main results are obtained with our latency-aware grouping. We then study the impact of grouping size the pruned top-1 accuracy and the inference time to show the effectiveness of our grouping scheme. Finally, we further show the generalization ability of our algorithm by testing with object detection task. We introduce the details of experimental setting in appendix Sec. B and provide pruning results on CIFARI0 10 Sec. C We apply HALP targeting latency reduction on multiple platforms show the scalability of our method: NVIDIA TITAN V GPU, Jetson TX2, Jetson Xavier and Intel CPU, All the latencies are measured with batched images to take advantage of computation capacities. We include experimental results for batch size inference in the appendix, which also show the efficacy of our algorithm.
2 in appendix
4.1 Results on ImageNet
ResNets. We start by pruning ResNet50 and ResNet101 and compare our results with state-of-the-art methods in Tab. oon TITAN V. In order to have fair comparison of the latency, for all the other methods, we recreate pruned networks according to the pruned structures they published and measure the latency. Those methods showing '_" in the table do not have pruned structures published so we unable to measure the latency. For our method, by setting the percentage of latency to remain after pruning to be X, we get the final pruned model and refer to it as HALP-X% We report FPS (frames per second) in the table and calculate the speedup of pruned network the ratio of FPS between pruned and unprumed models.
From the results comparison in Table. | we can find that for pruned networks with similar FLOPs using different methods, our method achieves the highest accuracy and also the fastest inference speed. This also shows that FLOPs do not correspond 1:1 to the latency. Among these methods for ResNet50 comparison, EagleFye [2) yields the closest accuracy to ours, but the speedup is lower than ours. In Table. | for the pruned ResNet50 network with 3G FLOPs remaining, our method

---

Table 1: ImageNet structural pruning results. We compare HALP for ResNet50 with two different dense baselines (left), ResNet101 and VGG16 (right up), MobileNet-V V1 and MobileNet-V2 V2 (right bottom) pruning experiments, with detailed comparison state-of-the- art pruning methods over vary- ing performance metrics. More comparisons and CIFAR 10 experiments can be found in Appendix
METROU
FLOPs
(G)
Topl (9)
ToP5 (96
FPS (IMSS)
SPEEDUP
METROO
FLOPS (G)
(%)
(MMS)
SPERTUP
RESNET50
RuSNE1101
No PRUNING
4.1
76.2
92.87
1010
1x
7.8
77.4
THINET-7O
ETN
75.8
90:67
TAYLOR-
4.7
11.4
1:21x
ALTOSI/M
76.0
1215
1.14x
HALP-000
(QuRs)
4.3
78.3
847
1.37X
METAPRUNING
3.0
76.2
HALP-50% (OURS)
3.6
77.5
994
1. 1.60%
GREG-1 23
2.7
16.3
1171
1.15x
OR-1OR-50X
2.9
76.0
508
1.47x
HALP-807 (OURS)
77.2
83.4
1256
1.23x
HALP-00 (OuRs)
2.7
77.2
11:0
1.50X
RESNEY50
1007
HALP-30S (OURS)
2.0
76.5
1521
2 2.45x
THINET-50 [0
MICTA PRUIING
AUTOSLIM
77
16
2.0
2.0
2.1
75.4
75.6
74.7
90.02
1502
1604
1.56x
1.58%
No PRUNING
VGG-16
15.5
71.6
766
1x
GBN G
76.2
83
FBS-3x [0]
5.1
71.2
CATE
15.6
HALP-30/ (OURS)
4.6
72.8
1408
2.42x
LEGR
2.4
75.6
92.70
FX CB
3.0
70.5
GREG-2
13
18
75.4
1:1d
1.39
HALP-20S (OURS)
288
1958
5.49k
CHIP Ss
2.1
76.2
92.90
HALP-112 (OURS)
2.0
76.5
93. 05
1630
METHOD
FLOPS (M)
Top1
(IM/5)
FPS
SPEEDUP
0.50x RESNET5O A1
1.1
72.0
2:98
2.45x
Mon KN
THINET-30 2T
1.2
72.1
88.30
No PRUNING
569
72.6
3415
1x
AUTONI/M
2390
MITTAPRUNING
26
142
66.1
7050
2.00
MICTA PRIIIIN
1.0
73-4
2381
2.34x
AUTOSLIM
150
67.0
7743
2.27%
CATE Imn
1.3
73.9
HALP-1155 (OuRS)
171
2.32x
GREG-2 383
1.3
73.9
1514
1.49x
CHIP Sx
1.0
73.3
91.
2509
2.32x
0.70x MoBILENerVI
325
4078
1.37x
HALP-50X (OURS)
265
70.5
4857
1.42x
NETADAFE MX
284
69.1
RUSNET5O EAGLEEVE
32] BASEL INK
MINTAPRNIING 20
316
70.9
4838
1.42X
No PRUNING
77.2
50.70
1019
EAGLEEYE
13
28-1
70.9
50220
1.47%
GDP 10
267
71.3
EAGLE RGG
k2
3.0
77.1
1165
1.14x
HALP--00 (OuRS)
297
71.3
5754
1. 1.68%
HALP-80% (OURS)
3.0
77.5
13.660
1200
1.18%
MOBILNNuR-2
EAGLEEYB-2G [R]
2.1
76.4
92. .9
1:71
1.44X
No PRUNING
301
72.1
3080
1x
HALP-55% RS)
HALP-60% (OuiS)
183
70.4
0688
1.84x
EAGLEEYE-10
1.0
74.2
77
2/200
2.38x
HALP-305 (Ot RS)
1.2
74.5
91.5 87
2597
2.55
HALP-75% (OuRS)
249
72.2
4110
1.53x
NVIDIA Jetson TX2
Intel CPU Xeon E5
NVIDIA Xavier
Tool Ner75)
Topl Acatho
10
23
Iflsurce aie (rEP magee
Eagikge
Mealhhnnngg
Figure 3: Pruning ResNet50 on the ImageNet dataset with NVIDIA Jetson TX2 (left), Intel CPU Xcon E5 (middle) and NVIDIA Xavier (right). The latency on Jetson TX2 and CPU is measured using PyTorch; on Xavier is measured using TensorRT FP32. Top-left is better.
achieves a .4% higher topl accuracy and slightly (.04x) faster inference. It is expected that the advantage of our method for accelerating the inference is more obvious when it comes to a more compact pruned network, which is 14% (or .20x) additionally faster for a 2G-FLOPs network while increasing accuracy by .2%, and .17 faster with .3% higher accuracy compared EagleEye-IG. IG. We analyze the pruned network structure in detail in the supplementary material (Appendix Sec. Q We plot the results comparison in Fig. where we also add the results of training our pruned network with a teacher model RegNe:Y-16GG (top1 82.9%) [50]. With knowledge distillation, our model is 2. 70x faster than the original model at 1 1% accuracy drop.
Scalability to other networks. We next experiment with three other models: VGG 56 Mo-
b I ! and MobileNetV2 [54]. Same as pruning ResNets, in Tab. we perform pruning with different latency constraints and compare with prior art. As shown, among these methods, the proposed HALP performs significantly better with higher accuracy and larger inference speedup.
Scalability to other platforms. Our approach not limited to a single platform. In this section, we conduct the same ResNet50 experiments three new platforms: NVIDIA Jetson TX2, NVIDIA
7

---

Table HALP acceleration of ResNet50 on GPUs with TensorRT (version 7.2.1.6)
TITAN
GPU
RTX3080 GPU
MODEL
Ace DROP
FP32
FP16
FP52
FP16
INT8 (Acc DROP)
EAGLEBYD-3G
-0.90%
1,11x
1.26x
1.06x
3.09%
6.31x (-0.81%)
HALP-S0% (OURS)
-1.25%
4.70x
1.18x
6.40x (-1.02%)
EAGLEEYE
-0.18%
1.54x
5.10x
1.35x
3.68X
7.46x (0.27%)
HALP-55% (OURS)
-0.35%
6.36x
1.68x
(0.13%)
EAGLEBYE IG
2.02%
2.73x
7.81x
2.20x
5.61x
12.29x (2.55%)
HALP-30% (OURS)
1.76%
2.91x
9.61x
2.56x
14.12x (2.38%)
Xavier, and Intel CPU Xeon E5, and compare the results Fig Aswhonn, our approach consistently outperforms the other methods with more speedup and higher accuracy, HALP leverages the latency characteristics of the platform to achieve a better accuracy-cfficiency trade-off. Specifically, on these platforms, HALP yields up to relative 1.27x speedup with slightly increment in top-1 accuracy compared to EagleEye [2] and 1.72x faster than the original model. From these results, we can conclude that method generalizes well to different platforms.
4.2 HALP Acceleration GPUs with TensorRT
To make it closer to the real application in production, we also export the models into onnx format and test the inference speed with TensorRT. We run the inference of the model with FP32, FP16 and also INT8. For INT8, we quantize the model using entropy calibration with 2560 randomly selected ImageNet training images. Since the INT8 TensorCore speedup not supported in TITAN GPU, only report the quantized results on RTX3080 GPU. The accelerations and the corresponding topl accuracy drop (compared to Py Porch baseline model) are listed in Tab, 2 We include more pruning results specifically for INT8 quantization in Appendix N
4.3 Design Effort for Pruning
In addition to noticeable performance boosts, HALP fact requires less design effort compared to prior art, as summarized in Tab. 3 (details in Appendix O). NetAddat [68] and Au-
Table 3: Comparison of extra computation required by prun- ing methods ImageNet. Our approach is around 4.3x faster than the next best method. Sub-network selection tim- ing is approximated rumning same device (a NVIDIA V100).
toSlim 722 generate many proposals during iterative pruning. Then evalu- ations the proposals are needed to select the best candidate for the next pruning iteration. EagleEye [32] pre-
METHOD
EVALUSTE PROPOSALS?
AUXILIARY NET TRAINING?
SUB-NETWORK SELECTION (RESNET30)
NETADAPT
63
185h (GPU)
THINKT
(GPU)
EAGLEBYE AUTOSLIM
3ith (GPU)
obtains 1000 candidates before prun-
ing and evaluates all of them order
AMC 23
to get the best one. Such pruning can- didate selection intuitive but causes a lot of additional time costs. The computation cost for MetaPruning [G] and AMC [22) can be even higher because they need to train auxiliary network to generate the pruned structure.
HALP (OuRS)
N
6.5h (GPU) 0.5h (CPU)
Compared to these methods, our method does not require auxiliary network training nor sub-network evaluation. The latency contribution in our method can be quickly obtained during pruning by the pre- mre-generated latency lookup table. Although creating the table for the target platform might cost time, we only do once for all pruning ratios. Solving the augmented knapsack problem brings extra computation, however, after neuron grouping, it only takes around additional 30 minutes of CPU time in total for ResNet50 pruning and less than minute for MobileNetV1, which is negligible compared to the fine-tuning process or training additional models. Moreover, this is significantly lower than other methods, for example the fastest of them EagleEye
32 requires 30 GPU hours.
4.4 Efficacy of Neuron Grouping
We then show the benefits of latency-aware neuron grouping and compare the performance under different group size settings.
8

---

we group neurons in a layer as an entirety so that they are removed kept together. Choosing different group sizes leads different performances. and also different computation cost on the augmented knapsack problem solving. In our method, we set an individual group size for each layer according to each layer's latency step size the look-up table. We name the grouping in our method as latency-awar grouping (LGG) For instance, for ResNet50, using this approach we set the individual group size 2f layers to 32, of 20 layers to 64, and 10 layers to 128. Layers closer to the input tend use smaller group size. Another option for neuron grouping is to heuristically set a fixed group size for all layers as literature does [69].
Performance comparison. As described in Sec. 3.3
Fig. 4 shows the performance of our grouping approach compared to various fixed group sizes for ResNet50 prumed with different constraints.
As shown, using small group sizes yields the worst performance independently of the latency constraint. At the same time, a very large group such 256 do also harm the final performance. Intuitively, a large group size averages the con- tribution of many neurons and therefore is not discriminative enough to select the most impor- tant ones. Besides, large groups might promote pruning the entire layer in a single pruning step, leading to performance drop. On the other hand, small group sizes such as 2 promote removing unimportant groups of neurons. These groups do not significantly improve the latency, but can contribute to the final performance. In contrast, our latency-aware grouping performs the best, showing the efficacy of our grouping scheme.
75
Top1 Apc%%)
-4
- gs+2 g5+8
pS-16
Yin et al. 2020
gS-256
gS-LG (ours)
0.45
0.5
0.55
0.65
0.75
0.8
inference time (ms/lmage)
Figure 4: Performance comparison of different group size settings for ResNet50 pruning Im- ageNet. We compare to heuristic-based group se- lection studied by 69 LG denotes our proposed latency-aware grouping in HALP that yields con- sistent latency benefits per accuracy.
Algorithm efficiency improvement. Setting the group size according the latency step size not only improves the performance, but also reduces computation cost on knapsack problem solving for neuron selection since it reduces the total number of object N to a smaller value. In our ResNet50 experiment, except for the first convolution layer, the group size of other layers varies from 32 to 128. By neuron grouping, the value of N can be reduced 215, which takes around one minute on average each pruning step to solve the knapsack problem on CPU, We have 30 pruning steps in total in our experiments, thus the time spent on neuron selection is around 30 minutes total, which can negligible compared to training time.
4.5 Generalization to Object Detection
To show the generalization abil- ity of our proposed HALP algo- rithm, we also apply the algorithm to the object detection task.
80
SSD512-RN50-HALP (Ours)
RetinaNet-
SSD512-AN50
take the pop- this experiment ular architecture Single Shot De- tector (SSD) 35 on the PASCAL VOC dataset "07+12" setting in 35 we use the union of VOC2007 and VOC2012
SSD300-VGG16
mAP (96)
BSD512-RN50-sh5
SSD300-RNS0
13
Following the
FLOPs
FPS
S80512-RN50, base model
77.90
65.56
60.24
SSDD12-AN50-22
75.83
45.09
76.49
trainval our training set and use the VOC2007 test as test set. We pretrain a SSD512 detector with ResNet50 as the backbone. The de- tails of the SSD structure are elab- orated in the appendix. Same to classification task, we prune the trained detector and finetune after- wards. We only prune the backbone network in the detector. The results
580330-AN55
75.58
1823
128.85
950805-VGG16
76.72
122.20
Faste/RCAN-VGG16
FasterRCNN V9G16
70.10
29:21
106.50
36.92
S5D612-RNED-HALP
132.57
68
35
inference time (ms/image)
Table 4: HALP for object detection on the PASCAL VOC dataset. Detailed numbers in the Appendix H

---

in Fig. show that the pruned detector maintains the similar final mAP but reduce the FLOPs and improve the inference speed greatly. with 77% FLOPs reduction and around 1.94x speedup the cost of only 0.56% mAP drop. We compare the pruned detector to some other commonly-used detectors in the table. The results show that pruning detector using HALP improves performance in almost all aspects.
5
Conclusion
We proposed hardware-aware latency pruning (HALP) that focuses on pruning for underlying hardware towards latency budgets. We formulated pruning as a resource allocation optimization problem to achieve maximum accuracy within a given latency budget. We further proposed latency-aware neuron grouping scheme to improve latency reduction. Over multiple neural network architectures, classification and detection tasks, and changing datasets, we have shown the efficiency and efficacy of HALP by showing consistent improvements over state-of-the-art methods. Our work effectively reduces the latency while maintaining the accuracy, which could significantly impact applications in cessurrc-consttrinee environments, such as autonomous vehicles other mobile devices.
References
[1] Yonathan Aflalo, Asaf Noy, Ming Lin, Itamar Friedman, and Lihi Zelnik. Knapsack pruning with inner distillation. arXiv preprint arXiv:2002.08255 08258, 2020.
[2] Jose M Alvarez and Mathieu Salzmann. Learning the number of neurons in deep networks. In Advances in Neural Information Processing Systems, pages 2270-2278, 2016.
[3] Rumen Andonov, Vincent Poirriez, and Sanjay Rajopadhye. Unbounded knapsack problem: Dynamic programming revisited. European Journal of Operational Research, 123(2):394-407 2000.
[4] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. arXiv preprint arXiv:2005. 1416., 2020.
[5] Yaohui Cai, Zhewei Yao, Zhen Dong, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Zeroq: A novel zero shot quantization framework. In Proceedings of the IEÉE/CVF Conference on Computer Vision and Pattern Recognition, pages 13 13169-13178, 2020.
[6] Changan Chen, Frederick Tung, Naveen Vedula, and Greg Mori. constraint-waar deep neural network compression. Proceedings of the European Conference on Computer Vision (ECCV), pages 400-415, 2018.
[7] Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran, Bryan Catanzaro, and Evan Shelhamer. cudnn: Efficient primitives for deep learning. arXiv preprint arXiv:1410.079,, 2014.
[8] Ting-Wa Wu Chin, Ruizhou Ding, Cha Zhang, and Diana Marculescu. Towards efficient model compression via learned global ranking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1518-1528, 2020.
[9] Xiaoliang Dai, Peizhao Zhang, Bichen Wu, Hongxu Yin, Fei Sun, Yanghan Wang, Marat Dukhan, Yunqing Hu, Yiming Wu, Yangqing Jia, P Vajda, M Uyttendaele, and Niraj K Jha. ChamNet: Towards efficient network design through platform-avd aware model adaptation. In CVPR, 2019.
[10] Pau de Jorge, Amartya Sanyal, Harkirat S Behl, Philip HS Torr, Gregory Rogez, and Puneet K Dokania. Progressive skeletonization: Trimming more fat from a network at initialization. arXiv preprint arXiv:2006.0908) 2020.
[11] Xiaohan Ding, Guiguang Ding, Yuchen Guo, and Jungong Han. Centripetal sgd for pruning very deep convolutional networks with complicated structure. In Proceedings of the IEFF/CV Conference Computer Vision and Pattern Recognition, pages 4943-4953 2019.
10

---

[12] Jin-Dong Dong, An-Chich Cheng, Da-Cheng Juan, Wei Wei, and Min Sun. Dpp-ner net: Device- aware progressive search for pareto-optimal neural architectures. In Proceedings of the European Conference on Computer Vision (ECCV), pages 517-531, 2018.
[13] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2):303-338 2010.
[14] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019.
[15] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. International Conference on Learning Representations (ICLR), 2019.
[16] Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint 1etp::9920095... 2019.
[17] Susan Gao, Xin Liu, Lung-Sheng Chien, William Zhang, and Jose M Alvarez. Vacl: Variance- aware cross-layer regularization for pruning deep residual networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0-0, 2019.
[18] Xitong Gao, Yiren Zhao, Lukasz Dudziak, Robert Mullins, and Cheng-zhong Xu. Dynamic channel pruning: Feature boosting and suppression. arXiv preprint arXiv: 18100033,, 2018.
[19] Yi Guo, Huan Yuan, Jianchao Tan, Zhangyang Wang, Sen Yang, and Ji Liu. Gdp: Stabilized neural network pruning via gates with differentiable polarization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5239-5250, 2021.
[20] Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural net- works with pruning, trained quantization and huffman coding. arXiv preprint 2015.
[21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. corr abs/1512.0333 (2015), 2015.
[22] Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li. and Song Han. Ame: Automl for model compression and acceleration on mobile devices. In Proceedings of the European Conference on Computer Vision (ECCV), pages 784-800, 2018.
[23] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv: (0000,02551,,0000
[24] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 04861, 2017.
[25] Hengyuan Hu, Rui Peng. Yu-Wing Tai, and Chi- Chi-Keung Tang. Network trimming: A data-driven neuron pruning approach towards efficient deep architectures. arXiv preprint 1eXivv1607.0322 2016.
[26] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708, 2017.
[27] Jonathan Huang, Vivek Rathod, Chen Sun, Menglong Zhu, Anoop Korattikara, Alireza Fathi, Ian Fischer, Zbigniew Wojna, Yang Song, Sergio Guadarrama, et al. Speed/accuracy trade-offs for modern convolutional object detectors. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7310-7311, 2017.
[28] Yeonwoo Jeong, Deokjae Lee, Gaon An, Changyong Son, and Hyun Oh Song. Optimal channel selection with discrete qeqp. In International Conference Artificial Intelligence and Statistics, pages 7915-7941. PMLR, 2022.
[29] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
11

---

[30] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Tomr SNIP: Single-shot network pruning based on connection sensitivity. In International Conference on Learning Representations. 2019.
[31] Sam Leroux, Pavlo Molchanov, Pieter Simoens, Bart Dhoedt, Thomas Breuel, and Jan Kautz. lamnn: Iterative and adaptive mobile neural network for efficient image classification. arXiv preprint arXiv: 1804. 10/23, 2018.
[32] Bailin Li, Bowen Wu, Jiang Su, and Guangrun Wang. Eagleeye: Fast sub-net evaluation for efficient neural network pruning. In European Conference Computer Vision, pages 339-654. Springer, 2020.
[33] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. International Conference on Learning Representations (ICLR). 2017.
[34] Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988, 2017.
[35] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yare Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In European conference on computer vision, pages 21-37. Springer, 2016.
[36] Zechun Liu, Haoyuan Mu, Xiangyu Zhang. Zichao Guo, Xin Yang, Kwang-Ting Cheng, and Jian Sun. Metapruning: Meta learning for automatic neural network channel pruning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3296-3305, 2019.
[37] Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In International Conference on Learning Representations, 2019.
[38] Ilya Loshchilov and Frank Hutter. Theoretical analysis of self-training with deep networks on unlabeled data. In International Conference on Learning Representations, 2017.
[39] Jian-Hao Luo and Jianxin Wu. Neural network pruning with residual-connectinn and limited- data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1458-1467, 2020.
[40] Jia Luo. Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. Proceedings of the IEEE international conference on computer vision, pages 5058 5066, 2017.
[41] Sangkug Lym, Esha Choukse, Siavash Zangeneh, Wei Wen, Sujay Sanghavi, and Mattan Erez. Prunetrain: fast neural network training by dynamic sparse model reconfiguration. In Proceedings of the International Conference for High Performance Computing, Networking. Storage and Analysis, pages 1-13, 2019.
[42] Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In Proceedings of the European conference on computer vision (ECCV), pages 116-131, 2018
[43] Silvano Martello, David Pisinger, and Paolo Toth. Dynamic programming and strong bounds for the 0-1 knapsack problem. Management science, 45(3):414-424, 1999.
[44] Silvano Martello and Paolo Toth. Knapsack problems: algorithms and computer implementa- tions. John Wiley & Sons, Inc., 1990.
[45] Paylo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 11264-11277 11222, 2019.
[46] Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. arXiv preprint 1rX1v:1611.06440, 2016.
12

---

[47]
Ravi Teja Mullapudi, Steven Chen, Keyi Zhang, Deva Ramanan, and Kayvon Fatahalian. Online model distillation for efficient video inference. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 3573-3582, 2019.
[48]
Nvidia. torch.
Convolutional
networks for image classification in py-
2020.
[49] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vito,,, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS- W, 2017.
[50] Iija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Design- ing network design spaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10428- 10436, 2020.
[51] Valentin Radu, Kuba Kaszyk, Yuan Wen, Jack Turner, José Cano, Elliot J Crowley, Björn Franke, Amos Storkey, and Michael O' Boyle. Performance aware convolutional neural network channel pruning for embedded gpus. In 2019 IEEE International Symposium on Workload Characterizatiin (IISWC), pages 24 34. IEEE, 2019.
[52] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28:91-99, 2015.
[53] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. Int. J. Comput. Vision, 11132221 December 2015.
[54] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chieh Chen. Mobilenetv2: Inverted residuals and linear boutlnecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510-4520, 2018.
[55] Maying Shen, Pavlo Molchanov, Hongxu Yin, and Jose M Alvarez. When to prune? a policy towards early structural pruning. In Proceedings the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 12247-12256, 2022.
[56] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations (ICLR), 2015.
[57] Yang Sui, Miao Yin, Yi Xie, Huy Phan, Saman Aliari Zonouz, and Bo Yuan. Chip: Channel independence- baseed pruning for compact neural networks. Advances in Neural Information Processing Systems, 34:24604-2466 -46166, 2021.
[58] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoe V Le. Mnasnet: Platform-awer neural architecture search for mobile. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2820-2828, 2019.
[59] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference Machine Learning, pages 1114. PMLR, 2019.
[60] Rishabh Tiwari, Udbhav Bamba, Arnav Chavan, and Deepak Gupta. Chipnet: Budget-aware pruning with heaviside continuous approximations. In International Conference on Learning Representations, 2021.
[61] Arash Vahdat, Arun Mallya, Ming- Yu Liu, and Jan Kautz. Unas: Differentiable architecture search meets reinforcement learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 12266-1127, 2020.
13

---

[62] Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations, 2020.
[63] Huan Wang, Can Qin, Yulun Zhang, and Yun Fu. Neural pruning via growing regularization. In International Conference on Learning Representations (ICLR), 2021.
[64] Tianzhe Wang, Kuan Wang, Han Cai, Ji Lin, Zhijian Liu, Hanrui Wang, Yujun Lin, and Song Han. Apq: Joint search for network architecture, pruning and quantization policy. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2078 2087, 2020.
[65] Bichen Wu, Xiaoliang Dai. Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10734-10742, 2019.
[66] Yu-Cheng Wu, Chih-Ting Liu, Bo- Ying Chen, and Shao- Yi Chien. Constraint-aware importance estimation for global filter pruning under multiple resource constraints, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 686-687, 2020.
[67] Wenhan Xia, Hongxu Yin, Xiaoliang Dai, and Niraj K Jha. Fully dynamic inference with deep neural networks. IEEE Transactions on Emerging Topics in Computing, 2021.
[68] Tien-Ju Yang. Andrew Howard, Bo Chen, Xiao Zhang, Alec Go, Mark Sandler, Vivienne Sze, and Hartwig Adam. Netadapt: Platform- aware neural network adaptation for mobile applications. Proceedings of the European Conference on Computer Vision (ECCV), pages 285-300, 2018.
[69] Hongxu Yin, Pavlo Molchanov, Jose Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K Jha, and Jan Kautz. Dreaming to distill: Data-free knowledge transfer via DeepInversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9715-8724, 2020.
[70] Hongxu Yin, Arash Vahdat, Jose M Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. V-ViT: Adaptive tokens for efficient vision transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10809-10818, 2022.
[71] Zhonghui You, Kun Yan, Jinmian Ye, Meng Ma, and Ping Wang. Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks. arXiv preprins urXiv:190.008172 2019.
[72] Jiahui Yu and Thomas Huang. Autoslim: Towards one-shot architecture search for channel numbers. arXiv preprint arXiv: 11728, 2019.
[73] Chenzhuo Zhu, Song Han, Huizi Mao, and William J Dally. Trained ternary quantization. arXiv preprint 1612.0161. 2016.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A]. You strongly encouraged to include justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Abstract.
Did you include the license the code and datasets? [No] The code and the data are proprietary.
Did you include the license to the code and datasets? [N/A]
14

---

Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations your work? (Yes) Please see the Section M about the discussion of latency lookup table.
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL.) [Yes] Please see Section of main paper and Section of supplementary material. The link the code available in the abstract.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Please see Section g and more details appendix Sec. B
(c) Did you report error bars (c.g., with respect to the random seed after running ex- periments multiple times)? [No] We observed quite stable final performance of the pruning.
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs. internal cluster, cloud provider)? [Yes] Please see Section 4 and more details in appendix Sec. B
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] Please see Section
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] Code will be released. The link to the code provided in the abstract.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] The ImageNet
3 and PASCAL VOC [13 dataset are open
source and available for non-commercial academic research.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] The ImageNet (53) and PASCAL VOC [ dataset are open source and available for non-commercial academic research.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "pe0SlV_t7xu", "reviewer": "Reviewer_6smA", "summary": "This paper proposes latency-aware structural pruning (LASP), which performs pruning as a resource allocation optimization problem to maximize the accuracy while maintaining a latency budget. The proposed methods achieve good performance on the ImageNet dataset for compressing ResNet, VGG, and MobileNet models.\n", "questions": "Is the average importance score of each neuron affected by the different batches of data? For example, for different batches of data, the score might be different.\n", "limitations": "The authors addressed some limitations of the previous work. It's good that they evaluted the efficiency of the pruned model on real hardware.\n", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality: Each part of the technique is not that novel. It is like a combination of several existing tricks to perform pruning, not enough analysis of the convergence or correctness. \n\nQuality: \n1) The method lacks mathematical proof. \n2) Evaluation is not comprehensive. It only performs on one dataset, ImageNet. Most of the experiments are on ResNet models. It will be better if more model architectures, like densenet and shufflenet, and more datasets, like cifar10 and cifar100, are included. More related works should be compared in evaluation, like [1-3].\n\n2) In related works, authors mentioned that one limitation of the related works is that a compression ratio does not directly translate into a computation reduction ratio. I do not think it is true. A lot of previous work has studied computation reduction ratio. For example, they studied FLOPs, inference time on hardware.\n\nClarity: Writing is clear.\n\nSignificance: This paper addresses a difficult structuring pruning task in a slightly better way than previous works.\n\n[1] Layer-adaptive Sparsity for the Magnitude-based Pruning\n[2] GDP: Stabilized Neural Network Pruning via Gates with Differentiable Polarization\n[3] The lottery ticket hypothesis: Finding sparse, trainable neural network\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "hcEQgBYh3XC", "reviewer": "Reviewer_NWGG", "summary": "- The author defines an optimization problem that maximizes accuracy with constraints on latency. \n- The author estimates the importance score of neurons based on Taylor Expansion (Molchanov, 2019) and uses the sum of the importance score as the proxy objective of the optimization problem. \n- Moreover, the author estimates latency as the sum of latency contributions of neurons. \n- Using these estimations, the author could view the optimization problem as a knapsack problem and solves the problem by using the knapsack solver augmented by the author.", "questions": "- As the author said in supplementary material, solving pruning problems with a knapsack problem is an existing idea (`KP`). Thus, the main novelty of this paper seems to be in the latency modeling method. However, there are some existing filter pruning methods that regress the latency with a simple model and utilize the model to solve constrained optimization problem. For example, `QCQP` models latency using a quadratic equation and solve a latency constrained optimization problem using QCQP. Hence, LASP seems to require more evidence that shows its benefit.\n- For example, I want to see the correlation between real latency and the sum of latency reduction potential, which LASP proposed as a proxy of real latency (I'm curious about the real latency vs the sum of latency reduction potential plot for some pruning recipes). \n- Also, if possible, I want to see the comparison between the sum of latency reduction potential and the quadratic modeling in `QCQP`.\n- For the training step, the training recipe and update steps greatly affect the performance. This method requires a few training steps for iterative pruning steps and 90 additional fine-tuning epochs. I would like to see the comparison with the baseline methods (such as `GBN` and `QCQP`) under the training recipe and the number of update steps as unified as possible.\n\nTo sum up, my main concern is (1) the analysis on latency proxy, and (2) fair comparison results.\n\n`KP` Knapsack Pruning with Inner Distillation, Aflalo et al.\n\n`QCQP` Optimal channel selection with discrete QCQP, Jeong et al.\n\n`GBN`   Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks, You et al.\n\nPOST REBUTTAL COMMENTS : \nThe authors answer the questions. My concerns have been addressed and I adjust my score. I recommend highlighting the strength of the latency estimation method with evidence in your responses in the revised version.", "limitations": "This paper deals with a socially important topic, filter pruning method, that can reduce the computational resources during the inference step of DNN-based APIs.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n- The idea is simple and intuitive.\n- The proposed latency proxy, the sum of latency reduction potential, seems an efficient and simple estimator. \n- The method does not require additional regularization term during the training. Thus the pruning process can start from the pretrained network. (Some existing pruning methods require regularization term in training step before pruning, so these methods usually start from scratch.)\n\n# Weaknesses\n- The novelty of the paper seems insufficient.\n- The effectiveness of the proposed latency proxy needs more evidence. \n\nPOST REBUTTAL COMMENTS:\nThe authors suggest a novel and simple latency estimation technique that can consider latency staircase pattern. They show the effectiveness of proposed latency estimation method with some evidence.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "aUfVEL-oVry", "reviewer": "Reviewer_NH1a", "summary": "The paper proposes a method for latency-driven filter pruning, by considering both hardware and importance scoring. To that extent, the authors formulate the pruning problem as a knapsack problem, where the goal is to maximize the importance given the latency constraints, which is optimized using an efficient problem-specific knapsack solver. The authors proceed to show the effectiveness of their pruning method on various networks and hardware configurations.", "questions": "I have no further questions, and will appreciate addressing my concerns from the main review.", "limitations": "The authors did not discuss limitations in their main paper, but did refer to a section in the appendix.\nI see no potential negative impact from this work.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The paper is well-written, highly-motivated, and easy to follow\n* Results are shown on different devices (including both GPU and CPU, and quantized networks as well)\n* The work focuses on real run-times rather than theoretical speedups\n* While most of the experiments focus on Imagenet classification, the inclusion of object detection is highly appreciated\n\nConcerns:\n* Figure 2 shows LASP+distillation, but I can't find information of it in the paper\n* It should be made clearer if Subsection 4.1 include the neuron grouping variation", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "A5md4YzJGAx", "reviewer": "Reviewer_7C8F", "summary": "This paper addresses the important research question of to prune neural networks such that the gain in inference is maximized. Authors to convert the neural network pruning problem into a resource allocation problem and solve it using Knapsack strategy. For estimation latency for different parts of the network, authors employ a lookup table. Experiments are conducted on multiple hardwares and authors demonstrate improved performance with the proposed method.", "questions": "Authors need to provide justifications/or address the points stated in the points mentioned in the weaknesses of the paper.", "limitations": "This work, in my opinion, does not have ant direct negative impact on the society. The limitations are not discussed thoroughly, but I hope with addressing the weaknesses outlined above, those as well can be included.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \nThe approach of optimizing the performance of neural networks w.r.t latency budget is interesting and a very relevant research problem from a practical point of view. The strategy presented by the authors permits designing pruned networks for different choices of latency budget which makes this approach quite useful. Further, the lookup table approach of the authors makes it very easy to include the d latency parameter into the optimization problem. Authors demonstrate the efficacy of the method on various choices of hardwares and present an extensive ablation study around the proposed method. Overall, the research content seems original, well written and the message is clear.\n\n\nWeaknesses: The presented paper in its current format has certain weaknesses and I would like to request the authors to respond with their remarks to the weaknesses listed below.\n\n1.\tA major concern for me is the use of lookup table to construct the latency of the overall network. In my opinion, latency cannot be computed as a sum of latency from different parts, rather it should also take into account the parallelizability across the cores of the GPU. Although authors demonstrate strong correlation between the computed latency and actual latency of the model, I am still skeptical about how well the look up table can model the latency. I would like the authors to present a discussion on the limitations of using the lookup table, if any.\n\n2.\tThe approach of pruning the model while taking latency into account is not something new and has been addressed in previous works. Moreover, the use of lookup table itself is not very new. In this regard, the novelty of this paper is limited. \n\n3.\tThe baselines presented in the paper seem outdated. Authors present a comparison with EagleEye, however, there are more recent pruning methods such as BAR [] and ChipNet [2] and CHIP [3] which perform very efficient pruning. Although these methods do not take latency into consideration, it is of interest to compare with these methods. These methods have been shown to reduce the FLOPs remarkably, thus authors should compare if their latency-aware method can outperform such recent methods. \n\n\n##############################\nPost rebuttal: Increased the rating from 5 to 6.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept