HALP: Hardware-Aware Latency Pruning
Authors: Maying Shen
Hongxu Yin
Pavlo Molchanov
Lei Mao
Jianna Liu
Jose M. Alvarez
Venue: ICLR 2022 Submitted (2022)
Keywords: Efficient deep learning
deep neural network pruning
latency reduction
hardware-aware pruning

Abstract: Structural pruning can simplify network architecture and improve the inference speed. We propose Hardware-Aware Latency Pruning (HALP) that formulates structural pruning as a global resource allocation optimization problem, aiming at maximizing the accuracy while constraining latency under a predefined budget. For filter importance ranking, HALP leverages latency lookup table to track latency reduction potential and global saliency score to gauge on accuracy drop. Both metrics can be evaluated very efficiently during pruning, allowing us to reformulate global structural pruning under a reward maximization problem given target constraint. This makes the problem solvable via our augmented knapsack solver, enabling HALP to surpass prior work in pruning efficacy and accuracy-efficiency trade-off. We examine HALP on both classification and detection tasks, over varying networks, on ImageNet1K and VOC datasets. In particular for ResNet-50/-101 pruning on ImageNet1K, HALP improves network speed by $1.60\times$/$1.90\times$ with $+0.3\%$/$-0.2\%$ top-1 accuracy changes, respectively. For SSD pruning on VOC, HALP improves throughput by $1.94\times$ with only a $0.56$ mAP drop. HALP consistently outperforms prior art, sometimes by large margins.

Under review as a conference paper at ICLR 2022
HALP: HARDWARE-AWARE LATENCY PRUNING
Anonymous authors Paper under double-blind review
ABSTRACT
Structural pruning can simplify network architecture and improve inference speed. We propose Hardware-Aware Latency Pruning (HALP) that formulates structural pruning a global resource allocation optimization problem, aiming at maxi- mizing the accuracy while constraining latency under predefined budget. For filter importance ranking. HALP leverages latency lookup table to track latency reduction potential and global saliency score to gauge accuracy drop. Both met- rics can be evaluated very efficiently during pruning, allowing us to reformulate global structural pruning under reward maximization problem given target con- straint. This makes the problem solvable via our augmented knapsack solver, enabling HALP to surpass prior work in pruning efficacy and accuracy-efficiene trade-off. We examine HALP on both classification and detection tasks, over vary- ing networks, IeageNet and VOC datasets. In particular, for ResNet-500-10) pruning on ImageNet, HALP improves network throughput by 1.60 x90x with +0.3%/O.22% top-1 accuracy changes, respectively. For SSD pruning on VOC, HALP improves throughput by 1.94x with only 0.56 mAP drop. HALP consis- tently outperforms prior art, sometimes by large margins.
INTRODUCTION
Convolutional Neural Networks (CNNs) act as the central tenet behind the rapid development in computer vision tasks such as classification, detection, segmentation, image synthesis, among oth- ers. As performance boosts, so do model size, computation, and latency. With millions, sometimes billions of parameters (e.g., GPT-3 Brown et al. (2020)), modern neural networks face increasing challenges upon ubiquitous deployment, that mostly faces stringent constraints such as energy and latency Dai et al. (209)): Molchanov al. 2016 2019). In certain cases like autonomous driving, breach of real-time constraint not only undermines user experience, but also imposes critical safety concerns. Even for cloud service, speeding up the inference of neural networks directly translates into higher throughput, allowing more clients and users to benefit from the service.
One effective and efficient method to reduce model complexity is through network pruning. The primary goal of pruming is to remove the parameters, along with their computation, that are deemed least important for inference Alvarez & Salzmann| (2016): Han et al.) (2015); Liu et al. (2019b); Molchanov al. (2019). Compatible with other compression streams of work such as quantiza- tion Cai et al.] ( Wang et al. (2020b): [Zhu et af.] (016)) and distillation Hinton et al. (2013); Mullapudi et al. (2019); Yin al. (2020), network pruning enables flexible tuning of model com- plexity towards varying constraints, while requiring much less design efforts by neural architecture search Tan al. (2019; Vahdat et al. (2020): Wu et al. (2019) and architecture re-designs Howard et al. 2017);; Ma et al. (2018); Tan & (209)). Thus, in this work, we study pruning, in particular structured pruning that removes filters (or neurons) to benefit off-the-shelf platforms. e.g., GPUs.
As the pruning literature develops, the pruning criteria also evolves to better reflect final efficiency. The early phase of the field focuses on maximum parameter removal in seek for minimum represen- tations of the pretrained model. This leads to flourish of approaches that rank neurons effectively to measure their importance Molchanov et al. (2016); Wang et al. (2020a). As each neuron/filter pos- sesses intrinsically different computation, following works explore proxy to enhance redundancy removal, FLOPs being one of the most widely adopted metrics Li et al. (2020): Wu et al. (2020); Yu & Huang 2019) reflect how many multiplication and addition computes needed for the model.
Hovewer, for models with very similar FLOPs, their latency can vary significantly Tan et all 20099)
Recently, more and more works start directly working reducing latency Chen al. (2018) Yang

---

Under review as a conference paper at ICLR 2022
pridence
Augmentted
Knassack
(very
Larget flters
fiterr selection
turget constralnt
Figure I: The proposed hardware-aware latency pruning (HALP) paradigm. Considering both per- formance and latency contributions, HALP formulates global structural pruning global resource allocation problem (Section B.1). solvable using our augmented Knapsack algorithm (Section (.2) Pruned architectures surpass prior work across varying latency constraints given changing network architectures for both classification and detection tasks (Section 4).
et al. (018). However, not much was done in the field of GPU friendly pruning methods due to non- trivial latency-architectur trade-off. For example, recently observed in Radu al. (2019). GPU usually imposes staircase-shape shaped latency patterns for convolutional operators with varying channels, which inevitably occur per varying pruning rate, see the latency surface in Fig. [ This imposes constraint that pruning needs to be done in groups to achieve latency improvement. Moreover, get- ting the exact look-up table of layers under different pruning configuration will benefit maximizing performance while minimizing latency.
Pruning different layers in the deep neural net- work will result in different accuracy-latency trade-off. Typically, removing channels from the latter layers has smaller impact accu- racy and smaller impact on latency versus re- moving channels from the early layers. We ask the question, if is better to remove more neu- rons from latter layer or less from early layer to achieve the same iccuracy-latenge trade-off. By nature, the problem is combinatorial and re- quires the appropriate solution.
Topn Accts)
Infernecce (enal
FL.PP
Oriprna musi
Exdkitee ueaal PSTT) Pudioillien X0mn)
Aaldiie Ve ate Nou 2OTT
In this paper, we propose hardware-aware la- tency pruning (HALP) that formulates pruning as a resource allocation optimization problem to maximize the accuracy while maintaining latency budget. The overall workflow is shown in Fig. [ For latency estimate per pruned ar- chitecture, we pre-analyze the operator-level latency values by creating a look-up table for every layer of the model on the target hardware. Then introduce additional score for each neuron group to reflect and encourage latency reduction. To this end, we first rank the neurons according their importance estimates, and then dynamically adjust their latency contributions. With neu- rons re-calibrated towards the hardware-aware latency curve, we now select remaining neurons to maximize the gradient- basedd importance estimates for accuracy, within the total latency constraint. This makes the entire neuron ranking solvable under the knapsack paradigm. To enforce the neuron selection order in a layer be from the most important to the least, we have enhanced the knap- sack solver so that the calculated latency contributions of the remaining neurons would hold, HALP surpasses prior art in pruning efficacy, see Fig. 2 and the more detailed analysis Section
HAUP Xham
Figure 2: Pruning ResNe150 the ImageNet dataset. pruposed HAL. surpasses state-of of the structured pruning methods over accu- mey. latency, and FLOPs metrics. Target hardware NVIDIA Titan GPU. Top-left better.
Our main contributions are summarized as follows:
We propose a latency-driven structured pruning algorithm that exploits hardware latency traits to yield direct inference speedups.
We orient the pruning process around a quick yet highly effective knapsack scheme that seeks for a combination remaining neuron groups to maximize importance while constraining to the target latency.
2

---

Under review as a conference paper at ICLR 2022
We introduce group size adjustment scheme for knapsack solver amid varying latency contri- butions across layers, hence allowing full exploitation of the latency landscape the underly- ing hardware.
We compare to prior art when pruning ResNet, MobileNet. VGG architectures ImageNet, PASCAL VOC and demonstrate that our method yields consistent latency and accuracy im- provements over oftte-of-theart methods. Our ImageNet pruning results present viable 1.6x to 1.9x speedup while preserving very similar original accuracy of the ResNets.
RELATED WORK
Pruning methods. Depending on when to perform pruning, current methods can generally be
divided into three groups Frankle & Carbin (2019b): i) prune pretrained models Haan et al. (2013);
Luo et al. 2017):Li Li et al. He al. 2018); Molchanov et al. (2016) 2019): Gale et al
2019)
ii) prune initialization Frankle Carbin (2019a): Lee et al. (2019) de Jorge et (2020), and ii)
prune during training Alvarez & Salzmann (2016); Gao et al. (2019); Lym et al. (2019). Despite
notable progresses the later two streams, pruning pretrained models remains as the most popular paradigm, with structural sparsity favored by off-the-shelf inference platforms such as GPU.
To improve on inference efficiency, many previous pruning methods trim down the neural network aiming to achieve high compression rate while maintaining an acceptable accuracy. The estimation of neuron importance has been widely studied in literature Hu et al. 2016); Lao et al. (2017);
Molchanov et al. (2016). For example, Molchanov al. 2019) proposes to use Taylor expansion to measure the importance of neurons and prunes the least-ranked ones until a desired number of neurons are pruned. However, compression ratio does directly translate into computation reduction ratio, amid the fact that each neuron/filter possesses different computation.
There are recent methods that focus primarily reducing FLOPs. Some of them take FLOPs into consideration when calculating the neuron importance to encourage penalizing neurons that induce high computations Wu et al. (2020). An alternative line of work propose to select the best pruned network from set of candidates Li et al. (2020): Yang et al. (2018). However, it would take a long time for candidate selection due to the large amount of candidates. In addition, these methods use FLOPs as a proxy of latency, which is usually inaccurate as networks with similar FLOPs might have significantly different latencies Tan al. (2019).
Latency-aware compression. Emerging compression techniques shift attention to directly prune to cut down on latency. One popular stream Neural Architecture Search (NAS) methods DDai et al.) 2019):Dong et al. (2018) 8); Tan et al.] (2019): of the network for a given latency requirement. They incorporate the platform constraints into the optimization process in both the architecture and parameter space jointly optimize the model size and accuracy. Despite remarkable insights, NAS methods remain computationally expensive in general compared their pruning counterparts.
[ Wu et al. (2019) that adaptively adjusts the architecture
Latency-oriented pruning has also gained growing amount of attention. Chen et al. (2018) presents framework for network compression under operational constraints, using Bayesian optimization to iteratively obtain compression hyperparameters that satisfies the constraints. Along the same line, NetAdapt Yang al./ 2018) iteratively prunes neurons across layers under the guidance of the em- pirical latency measurements on the targeting platform. While these methods push the frontier of latency constrained pruning, the perdware-incurres latency surface in fact offers much more poten- tial under our enhanced pruning policy as we show later, large rooms for improvements remain un-exploited and realizable.
3 METHOD
In this section, we first formulate the pruning process as an optimization process, before diving deep into the importance estimation for accuracy and latency. Then, elaborate how to solve the optimization via knapsack regime, augmented by dynamic grouping of neurons. We finalize the method by combining these key steps under one realm of HAL.P
3

---

Under review as a conference paper at ICLR 2022
3.1 OBJECTIVE FUNCTION
Consider neural network that consists of L layers performing linear operations on their inputs, to- gether with non- non-linear activation layers and potentially pooling layers. Suppose there are Ni neurons (output channels) in the lh layer and each neuron encoded by parameters W E RC x where C is the number of input channels and K is the kernel size. By putting all the neurons across the network together. we get the neuron parameter set W [[W]] I==1)1=1 is the total number of neurons in the network.
Ny 1. where N -E Ni
Given a training set D {(r; ):)),== the problem of network pruning with given constraint C can be generally formulated as the following optimization problem:
arg min C(W,D) s.t. ①b (f((W. SC
(1)
where W C W the set of remaining neurons after pruning and L is the loss of the task. f(-) encodes the network function, and ((-) maps the network to the constraint C, such as latency, FLOPs, or memory. We primarily focus on latency for ((.) in this work while the method easily scales to other constraints.
The key to solving the aforementioned problem relies on identifying the portion of the network that satisfies the constraint while incurring minimum performance disruption:
arg max PA PL
Σ I(Pi).
s.t. Σ T((P--1,P)) S AI
0
S Pt S S
(2)
where pt denotes the number of kept neurons at layer I, (pi) signals the maximum importance to the final accuracy with pt neurons, and Ti Pt) checks on the associated latency contribution of layer I with pt-1 input channels and ir output channels. Po denotes fixed input channel number for the first convolutional block, e.g., 3 for RGB images. We next elaborate on [(-) and T(-) :) in detail.
Importance score. To get the importance score of layer to final accuracy, namely (p:) in Eq. take it as the accumulated score from individual neurons Σn Zí. We first approximate the im- portance of neurons using the Taylor expansion of the loss change Molchanov et al. (2019). Specifi- cally, we prune on batch normalization layers and the importance of the n-th neuron in the I-th layer calculated
If' = 19 9
where g denotes the gradient of the weight, 71th and ST are the corresponding weight and bias from the batch normalization layer, respectively. Unlike squared loss in Molchanov et al. (2019), we use absolute difference as we observe slight improvements.
In order to maximize the total importance, we keep the most important neurons at higher priority. To this end, we rank the neurons in the 4n layer according to their importance score descending order and denote the importance score of the -unkeed neuron as If. thus we have
k())) I/
II
(4)
Latency contribution. We empirically obtain the layer latency Ti(p- Pr) in Eq. by pre-building layer-wise look-up table with pre-measured latencies. This layer latency corresponds the aggre- gation of the neuron latency contribution of each neuron in the layer, d:
P1
71(1-1.P))
Σ
OSPS N..
(S)
1=1
The latency contribution of the j-th neuron in the 1-th layer also be computed using the entries the look up table as:
d= T(ppp 1.5)
T(pp-115111) - 1 S pt
(6)
4

---

Under review as a conference paper at ICLR 2022
Algorithm Augmented Knapsack Solver
Input: Importance (T4 *RA1) where desconiingly: Neuron lateney contribution R
Lalency C. I: man E R(C++)), keep E (24x((41) I. maxV[c] max importance under constminn et keepl). c]: neurons to keep in layer to achieve maxfic]
}
3: for do
2. for
for 1....,
Meeep
maaV(e ef. Sprane maxV[c]
importunce achieve under construit with object being kept higher more impoctant neurons layer kept
Skeey Sprone keepll, <] 5
7:
looplt,e d] updine_maxVie Vkeep
8-
tbe
end leoplt, e| keep(t, c 1). updone maxV(c] [c] Pyrane
10:
11:
und for upditeunaae
14:
und for
15:
keepun
kept neurons
17: for .....
retrieve the set kept neurons
18:
keepl, C))
top ranked layer
20:
C C 20=1 of
21:
end
Output: Kept impurtant neurons (keep.n).
In practice, we first rank globally neurons by importance and then consider their latency contribution. Thus, we can draw the following properties. If we remove the least important neuron in layer I, then the number of neurons will change from Pi pt 1, leading to latency reduction c," as this neuron's ' latency contribution score. We assign the potential latency reduction to neurons in the layer by the importance order. The most important neuron in that layer would always have latency contribution c.. At this stage, finding the right combination of neurons to keep imposes a combinatorial problem, and in the next section tackle it via reward maximization considering latency and accuracy traits.
3.2 AUGMENTED KNAPSACK SOLVER
Given both importance and latency estimates, now aim at solving Eq. 2 layer importance Eq.
By plugging back in the
4 and layer latency Eq.
5
we come to
max ΣΣ
T
s.t.
ΣΣ 1=1
c CC
0 0 Ni 1 17 I..M
-=1=1
This simplifies the overall pruning process into knapsack problem only with additional preceding constraints. The preceding enforcement originates from the fact that for neuron with rank in the 4n layer, the neuron latency contribution only holds when all the neurons with rank I.....j are kept in the leh. layer and the rest of the neurons with rank r = 1,j + 2. Pt are removed. Yet the problem solvable by specifying each item with a list of preceding items that need to be selected before its inclusion.
We augment the knapsack solver to consider the reordered neurons with descending importance score so that all the preceding neurons will removed before it. A detailed description of the pseudo code of the augmented knapsack solver is provided Algo. T The augmented solver required to make sure that the latency cost is correct.
3.3 NEURON GROUPING
Considering each neuron individually results in burdensome computation during pruning. We next explore grouping neurons so that a number of them can be jointly considered and removed enabling faster pruning Yin et al. (2020). This helps exploit hardware-incuree channel granularity guided by the latency, and speed up knapsack solving of Eq. 7
We refer to the difference of neurons counts between two latency cliffs of the staircase-patternel latency as the latency step size. In our method, we group channels in a layer as an entirety, where the value of is equal to the latency step size. The neurons grouped by the order of importance. Then we aggregate the importance score and latency contribution for the grouped entity. When dealing with skip connections in ResNet and group convolutions in MobielNet,
only group
5

---

calculate the average importance score for each neuron Under review as a conference paper at ICLR 2022
neurons within a layer, we also group the neurons sharing the same channel index from the connected layers Ding et al. (2019); Luo & Wu (2020). Note that the latency step size for different layers might the largest group size among the layers. Latency-
be different. For cross-layer grouping. aware grouping enables additional performance benefits when compared to a heuristic universal grouping, as we will later show in the experiments.
One noticeable benefit of neuron grouping is the simplification of knapsack problem that scales linearly with the number of candidates under consideration (see Line of Alg. [.. As an example, by grouping neurons together for a ResNet50, the total number of (grouped) neurons can be greatly reduced from 26, 560 only 215, this drastically speedups the solver. Such saving can be especially beneficial as network size scales up, very likely trend that literature foresees.
3.4 FINAL HALP REGIME
With all aforementioned steps, formulate the final HALP as follows. The pruming process takes a trained network as input and prunes it iteratively to satisfy the requirement of a given latency budget C. We perform one pruning every minibatches and repeat it k pruning steps in total. In particular, we set k milestones gradually decreasing the total latency to reach the goal via exponential scheduler de Jorge et al. 22200, with C C CC C C. The algorithm gradually trims down neurons using steps below:
Step 1.
For each minibatch. we get the gradients of the weights and update the weights as during the normal training. We also calculate each neuron's 's importance score Eq.
Step 2. Over multiple minibatches
and rank them accordingly. Then we count the number neurons remaining in each layer and dynamically adjust the latency contribution in Eq. 6
Step 3. We group neurons as described in Sec. 3.3 and calculate group's importance and latency reduction. Then we get the nearest preceding group for each layer.
Step 4. We execute the Algo. to select the neurons being remained with current latency milestone. Repeat starting from the Step until & milestones are reached.
Once pruning finishes we fine-tune the network to recover accuracy.
4 EXPERIMENTS
We demonstrate the efficacy and feasibility of the proposed HALP method in this section. We use ImageNet ILSVRC2021 Russakovsky et al. (2015) for classification. We first study the impact of grouping size on the pruned top-1 accuracy and the inference time to show the effectiveness of our grouping scheme. We then study four architectures (ResNet50, ResNer101, VGGI6 and MobileNet- V1/V2) on different GPUs and compare our pruning results with the state-of-the-art methods on classification task. Finally, we further show the generalization ability of our algorithm by testing with object detection task. We introduce the details of experimental setting in appendix Sec. A Unless otherwise stated, we target NVIDIA TITAN V GPU for main experiments and measure latency at batch size 256 for ResNets and VGG, and 512 for MobileNets. We include experimental results for smaller batch size in the appendix, which also show the efficacy of our algorithm.
4.1 RESULTS ON IMAGENET
To demonstrate the effectiveness of the proposed HALP method, we compare HALP with several state-of-the-art methods on the large scale dataset ImageNet.
ResNets. We start by pruning ResNet50 and ResNet101 and compare our results with state-of-the- art methods in Tab. on TITAN V. In order to have fair comparison of the latency, for all the other methods, we recreate pruned networks according to the pruned structures they published and measure the latency. Those methods showing "-' in the table do not have pruned structures published so we are unable to measure the latency. For our method, by setting the percentage of latency to remain after pruning to be X, we get the final pruned model and refer to it as HALP- X.%. We report FPS (frames per second) in the table and calculate the speedup pruned network as the ratio of FPS between pruned and unpruned models.

---

Under review as a conference paper at ICLR 2022
Methol
FlLO%s
Top1
Top5
(%)
(mm/))
Speedup
Mathol
ELOPs
Topi (16
FPS
Spesdap
ResNet101
ResNer50
No pruning
7.8
77.4
620
No pruning
4.1
76.2
92.87
1019
1x
Taslos 75%
Molchatov
2019)
77.4
750
1.21x
ThiNer- Ma Manne al
2017
2.9
75.8
90.67
HALP-800
4.3
78.3
547
1.37%
AunoSiim 16 Timig
2019
3.0
76.0
1215
1.14%
HALP--00%
(Ours)
77.8
994
1.60x
GRey-] (ann
HAIP--00 (Ours)
MotaPaun mine
3.0
27
5.1
77.2
76.22
76.3
93.47
1256
1171
1.35x
1
Taylnc 65%
HALF 201
HALP 30% (Ouns)
Molchanov et al. 2019)
(Owas))
2.0
2.7
2.9
77.2
70.3
76.0
1521
1180
508
2:45x
1,47
1,50K
0.75x RexNer le ct 2015
ThaNet MANee-50laaoo
AutoSlim
2019
2.5
74.8
90,02
1592
1667
1.56x
1.4x
No
VGG-16
15.5
71.6
7066
MonaPunn
1934
FBS-5x
Clav 2015
5.1
71.2
GBN
92.83
HALP- S0r
4.6
72.3
1405
2.42x
PRS-5- Ca 2018
3.0
70.5
(anGReene KVR0
92.70
HAI.--0% (Oun))
2.8
70.8
1958
5.19
GRez Waangers
1414
HALP-DI7 [OWF5)
70.5
98.05
1630
1.60%
Method
ELOPs
Topl
(16)
Specthp
0.50x RexNorfi He 2015
2408
2.45x
ThiNer- da RALEs
1,2
72.1
$8.30
MobllcNet-VV
AutoSlim 1immin 2019
1,0
76.00
2500
2.45x
No praning
500
72.6
3415
Metaltunhgg Le Le 2019
1,0
Til.4
2981
x
MetaPronns cl. 2015
142
66.1
7000
2,00
CAIR Ware in Iupu
1.3
73.9
AutoSii 1onig
150
67.9
7745
GRey- Wain a 2021
1.3
73.9
1514
1.49x
HALP-427
68.3
7040
HALP (0Wrs)
1.0
74.3
91.81
2755
2070x
0.75% MobillNeeV/
325
08.4
4078
1.37X
ResNer50 EngleEye Li
2020 baseline
AMC He glal. 201
255
70.5
0557
1.42X
No prining
50.70
1010
NeiAdape Yorks ai 2011
264
09.1
MataPrubin Ci 201
516
70.9
6535
1.42x
Eaplefiee Giet
2020
90.97
1165
BagkeEye Srar 20ra
284
70.9
D020
1.47%
HALP-S0% (ows)
77.5
93.60
1203
1.18%
HALP (0ar5)
207
71.3
5764
1.68%
20200
MobileNet-VV V2
HALP-55% lows))
90.16
1672
1.64x
No prusing
301
72.3
3080
18
Eickttye-GGLier
20201
91.77
2420
2,88x
HALP-00% (Ours)
185
70.4
5668
1.84K
HALF-10% (ow2))
74.5
91.87
2507
2.35
BALP- (Ouns)
210
72.2 4.10
1,33%
Table I: ImageNet structural pruning results. We compare HALP for ResNet50 with two different dense baselines (left), ResNet101 and VGG16 (right up), MobileNet-V VI and MobileNet-V2 (right bottom) pruning experiments, with detailed comparison to state-of-the-art pruning methods over varying performance metrics.
From the results comparison we can find that for pruned networks with similar FLOPs using differ- methods, our method achieves the highest accuracy and also the fastest inference speed. This also shows that FLOPs do not correspond 1:1 to the latency. Among these methods for ResNet50 comparison, EagleEye Li al. (2020) yields the closest accuracy ours, but the speedup is lower than ours. In Table. [ for the pruned ResNet50 network with 3G FLOPs remaining, our method achieves .44 higher topl accuracy and slightly (.04x) faster inference. is expected that the advantage of our method for accelerating the inference is more obvious when it comes to a more compact pruned network, which 14% (or .20x) additionally faster for a 2G-FL.PPs network while increasing accuracy by .2% When we further prune the network to IG, we get pruned network that has more FLOPs but still faster inference speed (.17x) compared to EagleEye-IGG IG, while we also get .3% higher accuracy. We analyze the pruned network structure in detail in the supplemen- tary material. We plot the results comparison in Fig. 2 where we also add the results of training our pruned network with teacher model YegNet 16GF (top1 82.9%) Radooavoviii et al. (2020). With knowledge distillation, our model is 2.70x faster than the original model at 1% accuracy drop.
Scalability to other networks. We next ex- periment with other models including both VGG [Simonyan Zisserman (2015). Mo-
Aon Top
bileNetV1 Howard et al. (2017) and Mo- bileNetV2 Sandler al. (2018). Same as prun- ing on ResNets, we perform pruning with many different latency constraints and compare with prior art in Tab. [ As shown, among these methods, the proposed HALPP performs signif- icantly better with higher top-1 I accuracy and larger inference speedup.
Inference time (mstingge)
FLOPs (3)
TeCtllmueal 2015
Weafonngguees www.rngeeeeeiiioooooooooo dome
Ofee Weig at (0U12200
Figure 3: Pruning ReaNet50 on the ImageNet dataset with Jetson TX2.
Top left better.
Scalability to other hardware. Our approach is not limited to single platform. We can compute the latency look-up table to apply HALP to a different platform. We repeat the ResNet50 experi- ments described earlier on a Jetson TX2 and compare our results with other methods. The latency is measured with a batch size 32, As shown in Fig 3] in a Jetson TX2, our approach presents faster inference time while maintaining similar FL.PPs and better accuracy.
7

---

Under review as a conference paper at ICLR 2022
4.2 HALP ACCELERATION ON GPUs WITH TENSORRT
To make it closer to the real application in pro- duction, we also export the models into onnx format and test the inference speed with Ten- sorRT. We run the inference of the model with FP32, FP16 and also INT8. For INT8, quan- tize the model using entropy calibration with 2560 randomly selected ImageNet training im- ages. Since the INT8 TensorCore speedup is not supported in TITAN V GPU, we only re- port the quantized results on RTX3080 GPU. The accelerations and the corresponding topl accuracy drop (compared to PyTorch baseline model) are listed in Tab. 2 We can further build a TensorRT INT8 lookup table to achieve potentially larger speedup, which remains as our future work.
dep
FP32
EP16
1932
FP16
DNTS (ua deop)
Englettee 3G
-ULM05
1.14x
4.36x
1,85
3,00k
b-atic (-1883))
HALP-905 (Oars)
1.24x
4.70x
3.32%
(-1628)
HAI IP-35% (Oar)
-0.355
6.36x
1,08
4.45x
8:34x 13%)
EugkEEee IG
2,625
2.73x
7,61x
222K
0.81
12.20 (2 (2.65%)
HAI HALE--30% (Our))
1.70%
2.01
9.61%
2.56-
6.44%
16:12x (2 (8.38%)
Table 2: HALP acveleration of ResNet50 GPUs with TensorTT (ver- sion 7.2.1.6).
4.3 DESIGN EFFORT FOR PRUNING
In addition to noticeable performance boosts, HALP in fact requires less design effort compared to prior art, as summarized in Tab. 3 (details in Appendix [0). NetAdapt Yang et al.] (2018) and AutoSlim Yu & Huang (2019) generate many proposals during iterative pruning. Then evaluations of the proposals are needed to select the best candidate that will proceed to the next pruning iteration. EagleEye Li et al. 2020) pre-obtains 1000 candidates before pruning and evaluates all of them in order to get the best one. Such pruning candidate selection is intuitive but causes a lot of additional time costs. The computation cost for MetaPruning Liu all (2019)) and AMC He et al. (2018) can be even higher because they need to train auxiliary network to generate the pruned structure.
Compared to these methods, method does not require auxiliary network training nor sub- network evaluation. The latency contribution in our method can be quickly obtained during pruning by the pre-generated latency lookup ta- ble. Although creating the table for the target platform might cost time, we only do it once for all pruning ratios. Solving the augmented knap- sack problem brings extra computation, how- ever, after neuron grouping, it only takes around additional 30 minutes of CPU time total for ResNet50 pruning and less than 1 minute for MobileNetVI. 1, which is negligible compared to the fine-tuning process or training additional models. Moreover, this is significantly lower than other methods, for example the fastest of them EagleEye et al. (020) requires 30 GPU hours.
Method
Evaluati proposals?
Ausiliary net training?
Sub-setwork selection (RseNer
NetAdapt ThiNet
195h (CIPU)
210hh (GPU)
EngiEye
30h (GPL)
AutoStim
MetaPruning
AMC
HALP (Ours)
N
6 6.5h (GPU) + 0.5h (CPU)
Table 3: Comparison of extra computation required by pruning methods ImageNet. Our appreach is around 4.3x faster than the next best method. Suh network selection timing is approximated as rumning on device (a NVIDIA V100).
4.4 EFFICACY OF NEURON GROUPING
We first show the benefits of latency-aware neuron grouping and compare the performance under different group size settings.
Performance comparison. As described Sec. 3.3 we group s neurons in a layer as an entirety so that they are removed kept together. Choosing different group sizes can lead to different performances, and also different computation cost on the augmented knapsack problem solving. In method, set individual group size for each layer according to each layer's ' latency step size in the look-up table. We name the grouping in our method as latency-aware grouping (LG). For instance, for ResNet50, using this approach
set the individual group size of 23 layers to 32, of 20 layers to 64, and 10 layers to 128. Layers closer to the input tend to use a smaller group size. Another option for neuron grouping is to heuristically set fixed group size for all layers as literature does Yin et al. (202)).
Fig. shows the performance of our grouping approach compared to various fixed group sizes for a ResNet50 pruned with different constraints. As shown, using small group sizes yields the worst performance independently of the latency constraint. At the same time, a very large group such as
8

---

Under review as a conference paper at ICLR 2022
256 do also harms the final performance. Intuitively, large group size averages the contribution of many neu- rons and therefore is not discriminative enough to select the most important ones. Besides, large groups might pro- mote pruning the entire layer in a single pruning iteration, leading to performance degradation. On the other hand, small group sizes such as promote removing unimpor- tant groups of neurons. These groups do not significantly improve the latency, but they can contribute to the fi- nal performance. In contrast, our latency-aware group- ing performs the best, showing the efficacy of our neuron grouping scheme.
Topt AcoSc)
22
55-16
2020
g9.256
SSHLG (ours)
70 0.4
0.85
0.5
0.08
inferenoe (mnktegge)
Figure 4: Performance comparison of our larency-aware grouping to different fixed sizes ResNet50 pruning on ImugeNet. We compure to heuristic hased group se- lection studied by Yin al. (2020) denotes our pro- posed latency
Algorithm efficiency improvement. Setting the group size according to the latency step size not only improves the performance, but also reduces computation cost on knapsack problem solving for neuron selection since it reduces the total number of object N to smaller value. In our ResNet50 experiment, except for the first convolution layer, the group size of other layers varies from 32 128. By neuron grouping, the value of N can be reduced 215, which takes around one minute on average at each pruning step to solve the knapsack problem on CPU. We have 30 pruning steps in total our experiments, thus the time spent on neuron selection is around 30 minutes in total, which can be negligible compared to training time.
grouping in HAAP that yields sistent Intency benefits per accuracy.
4.5 GENERALIZATION TO OBJECT DETECTION.
To show the generalization ability of our pro- posed HALP algorithm, we also apply the al- gorithm to the object detection task. In this ex- periment we take the popular architecture Sin- gle Shot Detector (SSD) Liu et al. (2016) on PASCAL VOC dataset Everingham (2010). Following the "07+12" setting in Liu jet al. (2016). we use the union of VOC2007 and VOC2012 trainval use the VOC2007 test as test set. We pretrain SSD512 detector with ResNet50 bone. The details of the SSD structure are elab- orated in the appendix. Same to classification task, we prune the trained detector and finetune afterwards. We only prune the backbone net- work in the detector.
9505112/IN50-HA (Ouns)
950512-4M60
SS09CO
70-
SS0512 RINEO dimm
IAA
PLOPs
500019 Rida bass
37.56
95
09:24
123.2
training set and
550309-6619
76.70
21.44
199.20
Farer 16101111
76.10
M120
99.1
Palrshe FR83
77.27
100.0
86.00
the back-
68
Inferamce 5mo (msllmage)
Figure 5: HALP for object dosection on the PASCAL VOC dataset. Detailed numbers can be found in the Appendix
The results in Fig. D show that the pruned detectors maintain the similar final mAP but reduce the FLOPs and improve the inference speed greatly, with 77% FLOPs reduction and around 1.94x speedup the cost of only 0.56% mAP drop. We compare the pruned detector to some other commonly-used detectors in the table. The results show that pruning detector using HALP im- proves performance in almost all aspects.
CONCLUSION
We proposed hardware-aware latency pruning (HALP) that focuses on structured pruning for under- lying hardware towards latency budgets. We formulated pruning as a resource allocation optimiza- tion problem to achieve maximum accuracy within given latency budget. We further proposed a latency-aware neuron grouping scheme further improve latency reduction. Over multiple neural network architectures, classification and detection tasks, and changing datasets, we have shown the efficiency and efficacy of HALP by showing consistent improvements state-of-the-art methods.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
For reproducibility purpose, we have included all the experimental setup, details, and all hyperma- rameters in the paper and the Appendix. The code will be released upon acceptance.
REFERENCES
Jose M Alvarez and Mathieu Salzmann. Learning the number of neurons in deep networks. In Advances in Neural Information Processing Systems, pp. 2270-2278, 2016.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint 2xXx::0005.14166 2020.
Yaohui Cai, Zhewei Yao, Zhen Dong, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Zeroq: novel zero shot quantization framework. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13169-13178, 2020.
Changan Chen, Frederick Tung, Naveen Vedula, and Greg Mori. Constraint-aware deep neural network compression. Proceedings of the European Conference on Computer Vision (ECCV), pp. 400-415, 2018.
Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran, Bryan Catan- zaro, and Evan Shelhamer. cudnn: Efficient primitives for deep learning. arXiv preprint arXiv:1410.0759, 2014.
Ting-W Ting-Wu Chin, Ruizhou Ding, Cha Zhang, and Diana Marculeseu. Towards efficient model com- pression via learned global ranking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1518-1528, 2020.
Xiaoliang Dai, Peizhao Zhang, Bichen Wu, Hongxu Yin, Fei Sun, Yanghan Wang, Marat Dukhan, Yunqing Hu, Yiming Wu, Yangging Jia, P Vajda, M Uyttendaele, and Niraj K Jha. ChamNet: Towards efficient network design through platform-aware model adaptation. In CVPR, 2019.
Pau de Jorge, Amartya Sanyal, Harkirat S Behl, Philip HS Torr, Gregory Rogez, and Puneet K Dokania. Progressive skeletonization: Trimming more fat from a network at initialization. arXiv preprint arXiv:2006. 09000,,,0000081, 2020.
Xiaohan Ding, Guiguang Ding, Yuchen Guo, and Jungong Han. Centripetal sgd for pruning very deep convolutional networks with complicated structure. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 4943-4953, 2019.
Jin-Dong Dong, An-Chieh Cheng, Da-Cheng Juan, Wei Wei, and Min Sun. Dpp-net: Device- aware progressive search for pareto-optimal neural architectures. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 517-531, 2018.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2): 303-338, 2010.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. International Conference on Learning Representations (ICLR), 2019a.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019b. URL https:
//openreview wtt.:/eteeettooommddo.------
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv 2019.
preprint arXiv:1902.000
Susan Gao, Xin Liu, Lung-Sheng Chien, William Zhang, and Jose M Alvarez. Vacl: srrancee-warr cross-layer regularization for pruning deep residual networks. In Proceedings of the IEEE/CVF International Conference Computer Vision Workshops, pp. 0-0, 2019.
10

---

Under review as a conference paper at ICLR 2022
Xitong Gao, Yiren Zhao, Lukasz Dudziak, Robert Mullins, and Cheng-zhong Xu. Dynamic channel pruning: Feature boosting and suppression. arXiv preprint arXiv:1810.05331, 2018.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning. trained quantization and huffman coding. arXiv preprint 1rXiv:1510000144 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. abs/1512.03385 (2015), 2015.
Yihui He, Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. Ame: Automl for model compression and acceleration on mobile devices. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 784-800, 2018.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.0253/ 2015.
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andrectto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: / 774.04461. 2017.
Hengyuan Hu, Rui Peng, Yu-Wing Tai, and Chi- Chi-Keung Tang. Network trimming: A data-driven neuron pruning approach towards efficient deep architectures. arXiv preprint 1007.v:600..0... 2016.
Jonathan Huang, Vivek Rathod, Chen Sun, Menglong Zhu, Anoop Korattikara, Alireza Fathi, Ian Fischer, Zbigniew Wojna, Yang Song, Sergio Guadarrama, et al. Speed/accuracy trade-offs for modern convolutional object detectors. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7310-7311, 2017
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. SNIP: Single-shot network pruning based on connection sensitivity. In International Conference on Learning Representations, 2019. URL
https::/cpenree//
net Forum?1d=B1122gjaa
Bailin Li, Bowen Wu, Jiang Su, and Guangrun Wang. Eagleeye: Fast sub-net evaluation for efficient neural network pruning. In European Conference on Computer Vision, pp. 639 6594.55. Springer, 2020.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. International Conference on Learning Representations (ICLR), 2017.
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference computer vision, pp- 2980-2988, 2017.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In European conference on computer vision, pp. 21-37. Springer, 2016.
Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang, Kwang- Ting Cheng, and Jian Sun. Metapruning: Meta learning for automatic neural network channel pruning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3296-3305. 2019a.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In International Conference on Learning Representations, 2019b. URI. httpp::/
.et/ hhtt//oouumLG--111BB33
Ilya Loshchilov and Frank Hutter. Theoretical analysis of self-training with deep networks on un-
labeled data. In International Conference on Learning Representations, 2017. URL https:
7/ //ppnrevieww net fenet/forunee S
Hao Luo and Jianxin Wu. Neural network pruning with residual-comneetion and limited-data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
1458 1467, 2020.
11

---

Under review as a conference paper at ICLR 2022
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruming method for deep neural network compression. In Proceedings of the IEEE international conference on computer vision, pp. 5p.85058-000 2017.
Sangkug Lym, Esha Choukse, Siavash Zangeneh, Wei Wen, Sujay Sanghavi, and Mattan Erez. Prumetrain: fast neural network training by dynamic sparse model reconfiguration. In Proceed- ings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1-13, 2019.
Ningning Ma, Xiangyu Zhang, Hai-Tao Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In Proceedings of the European conference on computer vision (ECCV), pp- 116--31, 2018.
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440. 2016.
Pavlo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11264-11272 2019.
Ravi Teja Mullapudi, Steven Chen, Keyi Zhang, Deva Ramanan, and Kayvon Fatahalian. Online model distillation for efficient video inference. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3573-3582, 2019.
Nvidia. Torch.
Convolutional Networks for Image Classification in Py-
2020.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS-WW W, 2017.
Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2020.
Valentin Radu, Kuba Kaszyk, Yuan Wen, Jack Turner, José Cano, Elliot J Crowley, Björn Franke, Amos Storkey, and Michael O Boyle. Performance aware convolutional neural network channel pruning for embedded gpus. In 2019 IEEE International Symposium Workload Characteriza- tion (IISWC), pp. 24-34 IEEE, 2019.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28: 91-99, 2015.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei- Fei. Imagenet large scale visual recognition challenge. Int. J. Compus. Vision, 11523) December 2015. ISSN 0920-5691. doi: 10. 00.1007/112233-01-008666. URL http:::////d// 1007/ htte//00.0007/11122--011----0-00
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations (ICLR), 2015.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International Conference on Machine Learning, pp. 6105-6114, PMLR, 2019.
12

---

Under review as a conference paper at ICLR 2022
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoe V Le. Mnasnet: Platform-awrre neural architecture search for mobile. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2820-2822222 2019.
Arash Vahdat, Arun Mallya, Ming-Yu Liu, and Jan Kautz. Unas: Differentiable architecture search meets reinforcement learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11266- 1275, 2020.
Chaoqi Wang. Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow, In International Conference on Learning Representations, 2020a. URL
https::/openrevilee net/ for.mrid= Forummid=-SggsccVkee
Huan Wang, Can Qin, Yulun Zhang, and Yun Fu. Neural pruning via growing regularization. In International Conference Learning Representations (ICLR), 2021.
Tianzhe Wang, Kuan Wang, Han Cai, Ji Lin, Zhijian Liu, Hanrui Wang, Yujun Lin, and Song Han. Apq: Joint search for network architecture, pruning and quantization policy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2078-2087, 2020b.
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangging Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10732-10742, 2019.
Yu-Cheng Wu, Chih-Ting Liu, Bi-ging Chen, and Shao-Y: Yi Chien. Constraint-aware importance estimation for global filter pruning under multiple resource constraints. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 686-687, 2020.
Tien-Ju Yang, Andrew Howard, Bo Chen, Xiao Zhang, Alec Go, Mark Sandler, Vivienne Sze, and Hartwig Adam. Netadapt: Platform-awar neural network adaptation for mobile applications. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 285-300, 2018
Hongxu Yin, Pavlo Molchanov, Jose M Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Ni- raj K Jha, and Jan Kautz. Dreaming to distill: Data-free knowledge transfer via DeepInversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 8715-8724, 2020.
Zhonghui You, Kun Yan, Jinmian Ye, Meng Ma, and Ping Wang. Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks. arXiv preprint 1rX 2019.
Jiahui Yu and Thomas Huang. Autoslim: Towards one-shot architecture search for channel numbers. arXiv preprint arviv: / 1903. /728, 2019.
Chenzhuo Zhu, Song Han, Huizi Mao, and William J Dally. Trained ternary quantization. arXiv preprint 16X.v: 2016.
13

---

Under review as a conference paper at ICLR 2022
A EXPERIMENTAL SETTINGS
For image classification, we focus on pruning networks on the large-scale ImageNet ILSVRC2012 dataset Russakovsky et al. (2015) (1.3M images, 1000 classes). Each pruning process consumes single node with eight NVIDIA Tesla V100 GPUs. We use PyTorch Paszke al. (2017) V1.4.0 model zoo for pretrained weights for our pruning for a fair comparison with literature.
In our experiments we perform iterative pruning. Specifically, we prune every 320 minibatches after loading the pretrained model with finetune the network for 90 epochs in total with an individual batch size at 128 for each GPU, For finetuning, we follow NVIDIA's recipe Nvidia (2020) with mixed precision and Distributed Data Parallel training. The learning rate is warmed up linearly in the first 8 epochs and reaches the highest learning rate, then follows a cosine decay over the remaining epochs Loshchilov & Hutter (2017). For latency lookup table construction, we target NVIDIA TITAN V GPU with batch size 256 for latency measurement to allow for highest throughput for inference, and target a Jetson TX2 with inference batch size 32. We pre-generate layer latency look-up table on the platform by iteratively reducing of the number of neurons in layer to characterize the latency with NVIDIA cuDNN Chetlur et al. (2014) V7.6.5. We profile each latency measurement 100 times and take the average to avoid randomness.
30 pruning steps in total to satisfy the constraint. We
B EFFICACY OF NEURON GROUPING ON MOBILENET
In this section, we show the benefits of latency- aware neuron grouping and the performance under different group size settings on MobileNetV
Since MobileNet has group convolutional layers speedup the inference, we take the group convo- lutional layer with its preceding connected convolutional layer together as coupled cross-layers Gao al. (019) make sure the input channel number and output channel number of the group con- volution remain the same. All the 27 convolutional layers can be divided into 14 coupled layers. In our method, with the neuron grouping, we set the individual group size of coupled layer to 16, of 3 coupled layers to 32 and 10 coupled layers to 64. Also, for MobileNerVI pruning, we add the additional constraint that each layer has at least one group of neurons remaining to make sure that the pruned network is trainable.
We compare our latency-aware neuron grouping with an heuristic option by setting a fixed group size for all layers. Fig. 6 shows the comparison results between neuron grouping method and various fixed group sizes for a MobielNet pruned with different latency constraints on ImageNet. As shown, similar to ResNet50, using small group sizes such as 16 leads worse performance: a large group size like 128 also harms the performance significantly. Our observations on ResNet50 pruning also hold in MobileNetV1 setting, further emphasizing the efficacy of our latency-aware neuron grouping.
C COMPARISON WITH EAGLEEYE ON IMAGENET
We now use the same unpruned baseline model provided by Eagle Li et al. (2020) to compare our proposed HALP method with EagleEye Li et al. (2200) varying the latency constraint. As shown in Fig. [7] our approach dominates EagleEye by consistently delivering higher top-1 accuracy with a significantly faster inference time.
We then analyze the structure difference between our pruned model and the EagleEye model. As mentioned in the main text that the proposed HALP method tries to make the number of remaining neurons each layer fall the right side of a step if the latency on the targeting platform presenting staircase pattern. Fig. g shows two examples of pruned layers after pruning from HALP-45% and EagleEye-2GG model. In the left figure, we show that the layer in our pruned model has only 5 more neurons pruned than that in EagleEye model, the latency is reduced 0.76ms drop while we have 31 more input channels. In the right figure, we also show that sometimes remain a lot more neurons (30 neurons) in layer with only little latency (0.21ms) increase. These two examples both show the ability of method to fully exploit the latency traits and benefit the inference speed.
much lower level which is
14

---

Under review as a conference paper at ICLR 2022
Model
mAP
FL.Pss (G)
params (M)
FPS (BS=1)
FPS (BS-32)
SSDS12-RN50, base model
77.98
65.56
21.97
68.21
103.48
SSD5 12- 1-2--550-shhh
T5.83
46.09
16.33
76.49
114.80
SSD300-RN50
75.69
128.85
319.32
SSD3ID VGGL6 Liu et al 2016
76.72
31.44
26.29
122.25
262.93
FasterRCNN- VOG
RetinaNet-HR RNN- Lin or 2017
77.27
106.50
36.50
36.92
1D5112-NN5--HALYPS
323.36
Table 4: HALP for object detection the PASCAL VOC dataset.
77
Apo/%
70
60
gs+2 g8-B
Topt
778
BS-16 Vin et 2020
No praning
08
g2-128
EagleEye Liet al. ECCV (2020)
LG (ours)
HALP (Ours)
0.12
0.22
0.26
0.25
0.3
0.4
0.5
0.8
0.9
Interence time (msllmage)
Inference time (mallmage)
Figure Performance comparison of our latency-aware grouping to different fixed sizes for a MobielNetV1 pruned with different la- tency constraints on ImageNet. We compare to heuristic-based group selection studied by Yin et al. (2020). LG denotes the proposed latency- aware grouping in HALP that yields consistent latency benefits per final accuracy.
Figure Pruning ResNet50 on the ImageNet dataset using the same baseline model as in Ea- gleEye with top-1 accuracy of 77.23%. The proposed HALP surpasses EagleBye ECCV20 et al. (2020) in accuracy and latency. Top-left is better.
Our method benefits a lot from the non-linear latency characteristic since we are trying to keep as many neurons as possible under the latency constraint. If the latency of the layer on the targeting platform shows linear pattern, the advantage of our method becomes smaller. Fig. 8 shows the latency behavior of the example layers on the targeting platform when reducing the number of input and output channels. As we can see, the staircase pattern becomes less obvious as the number of input channel reduces and the GPU has sufficient capacity for the reduced computation. This happens during pruning, especially for large prune ratios. In such a case, the FL.OP count reflects the latency more accurately, and the performance gap between reducing FLOPs and reducing latency can possibly become small. Nevertheless, method can help avoid some latency peaks as shown in Fig. which could otherwise happen using other pruning methods.
D PRUNING RESULTS ON OBJECT DETECTION
In this section we show the detailed pruning results on objection detection task for Sec. 4.5 To
prune the detector, we first train a SSD512 with ResNet50 as backbone. We also train some other popular models for performance comparison. The detailed numbers of Fig. 5 are shown in Tab,
E RESULTS WITH SMALL BATCH SIZE
In the main paper, we use a large batch 256 in the experiment to allow for highest throughput for inference, which also makes the latency of the convolution layers show apparent staircase pattern that we can take full advantage of the latency characteristic. In this section, we show that with small batch size 1 that obvious staircase pattern showing up in layer latency, our HALP algorithm still delivers better results compared to other methods.
When batch size 1 for inference, the layer latency of ResNet50 does not show obvious stair- case pattern most of the layers due to the insufficient usage of GPU. Therefore in this experiment, use the latency lookup table granularity neuron grouping size, which in our case is 2, fully
15

---

Under review as a conference paper at ICLR 2022
FLOPs (G)
Topl Acc (%)
Top5 Acc (%)
FPS
Method
Speedup
(imgs/s)
No pruning
4.1
76.2
92.87
181
1x
0.75x ResNet50 He el al. (2015)
2.3
74.8
192
1.06
AutoSlim Yu & Huang
2019
2.0
75.6
181
1.00
MetaPruning Liu et al.
2019a
2.0
75.4
190
x.05x
EagleByy- Li et al.
2020
2,1
76.4
92.89
190
1.05x
GReg-2 Wang et 2021)
1.8
75.4
196
1.09
HALP-90% (Ours)
2.9
76.4
93.10
220
1.22x
0.50 ResNect5 He al. (2015)
1.1
72.0
193
1.07x
AutoSlim Yu & Huang
2019
1.0
74.0
191
1.06x
MetaPruning Liu et al. EagleBye-
2U19a
1.0
73.4
196
1.09
2020
1.0
74.2
91.77
192
1.06
GReg-2 Wang ct al. HALP-80% (Ours)
2021
1.3
73.9
206
1.14x
2.3
75.3
92.35
247
1.37x
Table 5: Pruning ResNet50 the ImageNet dataset (TITAN V) targeting inference with batch size 1. HALP-X indicates that X% latency to remain after pruning. The speedup is calculated as the ratio of FPS between the pruned network and the unpruned model.
HOP
2ATE
150
253
FOU charrdd
Figure 8: Two examples of pruned layers from HALP model and EagleEye Li et al. (2020) model. The scattered black points are the locations of the layers fall to after pruning.
exploit the hardware latency traits during pruning. We show our pruned results and the comparison with other methods in Tab. 5 As shown the table, while other methods reduce the total FLOPs of the network after pruning, they do not reduce the actual latency much, which is up to 1.09x faster than the original one at the of 2.8% topl accuracy drop. Compared to these methods. although we get less FLOPs reduction using our proposed method, the pruned models are faster and get higher accuracy, which is 1.22x faster than the unpruned model while getting slightly higher accuracy and 1.37x faster with only 0.9% accuracy drop.
F IMPLEMENTATION DETAILS
Convert latency in float to int. Solving the neuron selection problem using the proposed augmented knapsack solver (Algo. in the main paper), requires the neuron latency contribution and the latency constraint be integers shown in line 4 of the algorithm. To convert the measured latency from full precision floating boint number to integer type, we multiply the latency by 1000 and perform rounding. Accordingly, we also scale and round the latency constraint value.
Deal with negative latency contribution. The neuron latency contribution in our augmented knap- sack solver must be non-negative value since we have dp array E R and we need to visit dp_arrayíc- - cn] in line of Algo. 1 in the main paper. However, by analyzing the layer la- tency from the look-up table we find that for some layers the measured latency might even increase when reducing some number of neurons. This means that the latency contribution could possibly be negative. The simplest way to deal with the negative values to directly set the negative latency contributions to be O. This leads the problem that the summed latency contribution would be larger than the actual latency value, causing less neurons being selected. Thus, during our imple- mentation, keep those negative latency values as they are, but update the vector size of dp array
16

---

Under review as a conference paper at ICLR 2022
to mi-ii((min()))) where min(e) is the minimum latency contribution. With such, the vector size of dp_array would be extended when there is negative latency contribution. This makes it possible to add one neuron with negative latency contribution subset of neurons whose summed latency larger than the latency constraint. After the addition, the total latency will still remain under the constraint.
Pruning of the first layer. In our experiments, we leave the first convolutional layer of ResNets unpruned to help maintain the top-1 classification accuracy. For MobileNet, the first convolutional layer is coupled with its following group convolutional layer. In our MobileNet experiments, we prune the first coupled layers at most to the half of neurons.
SSD for object detection. Our SSD model is based on Liu et al. (016)). When we train SSD- VGGI6, we use the exactly same structure as described in the paper. When we train SSD- ResNet50, the main difference between our model and the model described in the original paper in the backbone, where the VGG replaced by the ResNet50. Following Huang et al. (2017). we apply the following enhancements in our backbone:
The last stage of convolution layers, last avgpool and fc layers are removed from the original ResNet50 classification model.
All strides in the 3rd stage of ResNet50 layers are set 1 x 1.
The backbone is followed by 6 additional coupled convolution layers for input size 512 x 512, or 5 for input size 300 x 300. A BatchNorm layer is added after each convolution layer. The settings of these additional convolution layers are listed in Tab. 6 each layer is represented (output channel, kernel size, stride, padding).
layer
SSD512
SSD512-slim
SSD300
layerl-convl
(512, 1)
(256, 1, 1,
(512, 1, 1)
layer1-conv2
(512, 3, 1)
(,12,
(512, 2,
layer2-convi
(256, 1,00)
(256,
1,10,
layer2-conv2
(512, 1)
(,12,
6,12,3,2.
layer3-convi
1128,111,
1,,0,
(128, 1,
layer3-conv2
(,26,3,211)
(,2,,
(,56,3,2,)
layer4-convl
(128,
1128,
1,28,1,1,
layer4-conv2
(256,3,2,)
(256, 1)
(256,3, 1,0)
layer5-convl
1128,1,1,)
1,
(128, 1,00
layer5-conv2
(256,3,2))
(256, 3,2,1)
(256, 1,0)
layer6-convl
(128, 1,0)
(128, 1,0)
layer6-conv2
(256, 1,
(256, 1,
Table 6: The additional convolution layers in SSD.
The detector heads are similar the ones in the original paper. The first detection head is attached to the last layer of the backbone. rest detection heads are attached to the corresponding additional layers. No additional BatchNorm layer in the detector heads.
G FLOPS PRUNING
Our implementation of lansney-constrained pruning can be easily converted be a FLOPs- constrained. When constraining on FLOPs, 4(-) in the objective function (Eq.1 in the main paper) becomes the FL.OPs measurement function and becomes the FL.OPs constraint. Since the FLOPPs of layer linearly decreases as the number of neurons decreases in the layer, we do not need to group neurons a layer any more. The problem can also be solved by original knapsack solver since each neuron's s FLOPs contribution in a layer is exactly the same and no preceding constraint is required. We conduct some experiments by constraining the FLOPs and compare the results with EagleEye et al. (2020). We name the experiments using the same algorithm HALP but targeting optimiz- ing the FLOPs as FLOP-T. As shown in Tab. with our pruning framework applying the knapsack solver, our results show higher top-1 accuracy compared to the pruned networks of EagleFye with
17

---

Under review as a conference paper at ICLR 2022
Method
(G)
Topl (X)
Top5 (%)
No praning
4.1
77:23
93.70
EagleEye-30
3.08
FLOP-T (Ours)
2.99
77.36
93.62
FagleFye-2G
2.06
FLOP-T (Ours)
1.95
76.64
93.21
745
EagleEye-II
1.3
FLOP-T (Ours)
0.96
74.84
92.26
FI.OFf
Moprusing eEyo (2000)
FLOP-T (Duns)
Table 7: Pruning ResNet50 on the ImageNet dataset with FLOPs constraint and comparison with state-of-the-art method EagleEye (ECCV'20) Li et al. (2020). We remeasure the FLOPs, top1 and top5 accuracy EagleEye to get results with two digits.
similar FLOPs remaining. We also observe a larger gap between the methods when it comes to more compact network.
H FLOPS vs. LATENCY
FLOPs can be regarded as proxy of inference latency; however, they are not equivalent [4, 33, 35, 40, 42). We do global filter-wise pruning and have the same problem NAS. The latency on GPU usually imposes staircase-shaped patterns for convolutional operators with varying channels and requires pruning groups. In contrast, FLOPs will change linearly. Depth-wist convolution, compared dense counterparts, has significantly fewer FLOPs but almost the same GPU latency due to execution being memory-bounder amount of FLOPs impose increases and feature map dimension shrinks both increase compute parallelism. For example, the first 7 X 7 conv layer and the first bottleneck 3 x 3 conv in ResNet50 have nearly identical FLOPs but the former is 60% slower on-chip.
The discrepancy also holds for ResNets where the same latency in earlier layers than later ones the number of channels
We compare our results of FLOPs-targeted (FLOP-T) showed in Sec. G and results using latency- targeted pruning (HALP) in Tab. g As shown in the table, using different optimization targets leads quite different FPS FLOPs curves. In overall, with similar FLOPs remaining, using our HALP algorithm targeting reducing the actual latency can get more efficient networks with more image being processed per second.
method
Top1(%)
FLOPs (G)
FPS (imgs/s)
FPS FLOPs
FLOP-T
74.84
0.962
2202
HALP
74.92
1.210
2396
19% imqes
FLOP-T HALP
FLOP-T
76.64
1.949
1436
HALP
76.55
1.937
1672
FLOP-T
77.36
2.988
1146
100
HALP
77.45
2.988
1203
FLOPs (S)
Table 8: ResNet50 pruning with FLOPs/latency constrain.
We also show more straightforward relationship be- tween the actual latency of layer and its FLOPs in Fig. 9
latency (ms)
We the 2nd convolution layer in the 1st residual block
of ResNet50 as an example. We vary the number of neu- rons of the layer from 0 to 128 and measure the actual latency on GPU (TTTAN V) as well as the FLOPPs of the layer. We can see from the figure that the actual latency does not strictly linearly decrease as the FLOPs decreas- ing.
100
120
FLOPs
Figure 9: The meusured latency FLLOPs of the 2nd convolution layer in the residual block of ResNet50.
tons/
18

---

Under review as a conference paper at ICLR 2022
I ALGORITHM CEATHHIMEEERE DEXPLANATION
We follow the standard Knapsack problem dynamic programming solution break down the origi- nal problem into sub-problems. Specifically, for each neuron (or neuron group) in layer we can choose either include or include it under the latency constraint c. When is kept, the total importance score increases I/ while the latency constraint for the other neurons becomes c - c/: If the neuron is removed, the latency constraint for the other neurons remains We choose to keep or remove the current neuron to maximize total importance. At the same time, we check whether more important neurons in the same layer are included to ensure the correctness of the latency. The neuron selection from the remaining neurons is a sub-problem to solve.
Precisely, we use vector max E (R(++) to store the maximum importance that we can achieve under the latency constraint c, < denotes the number of neuron groups we need to maintain in layer to obtain the maximum impor- tance max V[c]. We process the neurons according their importance score in decreasing order. In this way, all preceding neurons to the current one (i.e., neurons with higher importance score in the same layer) will always considered first. To decide if keep or remove the current neuron, we check the total importance score and the inclusion status of its preceding neurons, we can maximize the total importance and ensure the latency cost correctness.
and keep E (R+x()+)), 2D vector where keepll, [1, cl
J ABLATION STUDY OF PRUNING STEP k
In this work, similar to many other prior meth-
79
ods Alvarez & Salzmann (2016); Molchanov al. (2019); Yu Huang (2019), we do iter- ative pruning with l: pruning steps in total. In this experiment, we analyze the the accuracy of the final result as function of k. We set the value of to 10, 20, 30 and 40 for itera- tive pruning, and also use i 1 to perform single-shot pruning. The result of this experi- get
77.
Topt
K.10
K6200
k--00
k-40
EagleEye
0.5
ment is shown in Fig. [0] As shown, similar results independent of k. Imporantly, all these results outperform EagleFye Li et al. 2020).. As expected, there is drop in accu- racy for single-shot pruning neuron importance would change as we remove some other neurons and, in this setting, the value not updated. Iterative pruning does not have this limitation as the importance score and the la- tency cost of the remaining neurons is updated after each pruning step to reflect any changes. In our experiments, we use 30 as it provides a good trade off between latency and accuracy.
Inferonce time (msimage)
Figure
Performance compurison of different pruning steps for ResNet50 praning ImageNet.
1). especially for large pruning ratios. The main reason is the
K MORE PRUNING RESULTS ON MOBILENETS
Table. 9 and Fig. I provide additional prun- ing results for lightweight networks such as MobileNet-Vi and MobileNet-V3 V2. For the un-
pruned models, we find that even MobileNet- V2 has significantly lower FLOPs, the infer- ence time is larger compared to MobileNet- V1.In both cases, HALP yields inference speeds-ups of 1.22x and 1.33x for MobileNet- VI and MobileNet-V V2 respectively, while maintaining the original topl accuracy.
Mobilchier
Nobrihooe's VT unproned"
Krobeshed umprunes
0.16
Infennce fino imsinagge)
Figure 11: Pruning MubileNets on the ImageNet dataset.
19

---

Under review as a conference paper at ICLR 2022
Method
FLOPs
Topl
Top5 (%)
FPS
Speedup
FLOPs
Topl (%)
Top5 (9)
FPS
(M)
(5))
(im/s)
Method
(M)
(im/s
Speedup
MobileNet-V1
MobileNet- V2
No pruning
569
72.64
90.88
3415
No pruning
301
72.10
90.60
3050
1x
HALP-10%
154
67.20
87.32
8293
2
HALP-50%
HALF P-42%
HAI-P-60 P.0%
297
171
237
71.31
69.79
68.30
90.05
88.08
89.08
3754
6887
7940
2.32x
2.02x
1.68%
HALP-65%
HALP-70% 70%
HALP-60%%
227
218
183
71.41
70.42
71.88
89.75
90.08
90.39
5668
5003
4478
1.84x
1.62x
1.45x
HALP-70%
360
71.78
90.39
4870
1.43x
HALP-90%
HALP-75%
273
249
72.45
72.16
90.44
90.68
3443
4109
1.12x
1.33x
HALP-90%
HALP-80%
507
416
72.95
72.52
90.78
91.02
3765
4167
1.22x
1.10x
HALP-95%
28.
72.55
90.79
3265
1.06x
Table 9: Pruning MobilcNet-VI and MobileNet-V? V2 on the ImageNet dataset with different targets.
L DETAILED CONFIGURATION OF PRUNED MODELS
We provide the detailed configuration of our pruned models of Tab, [ For each model, we list the number of neurons remaining in each convolution layer, starting from the input to the output. For ResNets we use [-] to denote a residual block and (+) to denote the neuron number of the residual bypass layer.
Note that for ResNets, the configuration is the "raw" configuration after the pruning. For each residual block #2, because of the existing of bypass layer, we allow the entire layer pruning and it still leads to trainable networks. Whenever there is entire layer pruning (#1 == 0 or 222 == or Jg == 0), all the other layers in the block can be removed and this branch can be replaced by constant value. In such case, the corresponding block in our final pruned model is cleaned to be [0, O, Oj. We also provide the detailed structure plot of two pruned ResNet50 models in Fig. [2 for better visualization.
ReaNin50- FopllFoye
haseline
HALP-900
64. 644, 256 (256), (92, 256), (0, 250 [138, 138, (19)61121101000
[256, 1024), (256, 160, 1004), (256, 1024), (256, 180. 1024), (612. (2015) 416, 2009), [612, 2018)
512), 12% 512), (256, (020) (256, 1024).
HALP-155
64, 164, 125 (129), (12, o, 128. 10.32, 1281,
1296, 64, 1024), (256. 1024), 1128, 3z, 1024), 96, 1024), 1812, 448, 2048 (2048), (416, 288, 2018). (812, 352, 2018]
(2844
2841, 96, 164, 284), (256, 192, 2024((1024), [128, o6, 1024].
HALP-3005
64, 0.0, 611/01), 0.000 J0, 52, 164, 64, 2501(250), 1-1, 64, 250), 161, 128, 256, 164, 96, 250,, (250, 01.800 (899), [125, 640) 0,8))
[128, 61, 896,, 10.0 896, 1.,0 806, (512, 520, 2018 (2048), (820, 160,, 2018), [680, 160, 2018]
ResNett01
64. 64, 92, 192 (192). 112, 32. 192). (64, 32, 1991 164, 128, (84)) [64, 56, 384], 196, 50, 384] (128. 128, 384) (206, 206, (90](996), (206, DS, sost.
HALP 0056
(206, 128, s9d), (256, 06, 800), (206) 16,, s96). (256, 125. e00) 256. 56, Heod) (250, 56, 8o0], (200) 96,, soo),, [256, 96. s06), (200, 64, st6 1206, 64, suc).
2266, DE sM6. 206. 64,, s06, (200. 6-1, soo). 2000. 96, 800), (266. 64, 806). (256, ti. sto). (1700 30, ams), (650, DS, 806, (an6, 64, 806), 2000, 128, sod).
(266, GL sMc. (112, 512, (2148)) [1222 468, 20148) (612. 480, 2018
64, 64, 32, 128 (126), (0. 82, 128, (641 32, 128), (64. 125, 384 (1884), (32, 83 284) [128, 56, 381). [128 06, 85-1) (256, 256, soG (896), (256, 96, 800).
HALP-900
W. 200) 1256, 61, 800, 1250, 64, 806 2556, 64, e00),, (266, 64, 800), (956, oc 800), (250, 01, 500 (250) D1. 596, (256, 64, 200, 7766, 64, 800).
Ra00, 06 506 2200, (66 606) 2266. 64, 800)) 1255 96, 900), 1266, 95, s06), (250 D6, 5061, 1200) 64, 806), 1200, 64, 806. 13.0 896, K.O. 806) 256, 64, 8o0))
[612, 519 (200)) (1800, 116. (D4e)) (212, 416, 2018)
64, 92, 125 (128), (0. A2, 128, JO.O, 125). 1-4, 128, (284)) (42 64, 184) 104, 36, 184 64, 128, 2844) (256, 250, snn) (8963) (o. G4, sna), (128, os, en1,.
ta 64, H00), 1o G4, stio, [128, 64. 8516 1128, isd. 201. jo. o 800, 1o o S00), 1128, 36, so0), (125, ca. 800), 226. n6, st6). 1256, 64, snn), 1206, 6i, soid.
HALP-40%
128, n2 abi 1228, 644 200L, 10. O. H00), O. O, 890). (256, 64, H00), (206, 64) stoj, [206, 96, 536), 1256, 64, 5066, (206, 96, 806 (312, 512, 2045)
(012, 500, 20555), [812, 900, 20145)
61,
128 1255, G4. 1128, 61, 768. [128, 64, 288), (128. 64, 1886. 10, 0, 268), (0, o, 706), (256, 64, hoa), (200) 0, V65), (256 04, 7dle, 15.22 512, 20045) (480 285, 20as
1981, (o. o. 1988, (oo. 20011250)
250),
250), (61. 128, 250), (250, 7008) 7788) 7088 (198, rust.
HALJ-SOSE
64, 7688, [1228, 64, Rais) (128, 708 o, 7885 O. rusl. [128, Gil, 708 1128 90, 768)
768), [0. 0. 768), (0.0 mis), (128, 92, 705).
(650, 250, 20448)
MoleleNas-V
HALP--00%
806, 14, 34, 32. 61. 64, 192, 192, 192, 384, 3211, 220, 384, 384 384, 484, 384, 468 990, 960
HALP 1255
16, 32, 02. 30, 64, 122, 320 320 200, 206,, 200, 256 320, 320, 896
MoNillNas-V2
HAL
16, 64, 24,
612, 96. 576, 526,
112,
9100, 1,00 960)) 96A0, 320. 1250
52, 192, 384
3G2 64, 352, 384, 522,
HALP-R0W
576,, 570,,
16, 16,16,55
900, 160, 9600, 160,, 9900, 910,, .152
16, 18, 18,
02, 64, 52, 176,
288, 04, 920, (200 04, S20, 320, 384, 96, 468, 448 96, 468, 448,
Table 10: The detailed configuration of the HALP pruned models.
M DIFFERENT CHOICE OF IMPORTANCE CALCULATION
We use the first Taylor expansion Molchanov et al. 2019) to estimate the loss change induced by pruning as the importance score of the neurons. It madieent-based importance calculation and is shown to be given promising results. In this section, we use the L2 norm of the neuron weights as the importance measurement and apply HALP framework to ResNet50 ImageNet classification
20

---

Under review as a conference paper at ICLR 2022
Number Nourcre
2046
Iemain
1004
(a) ResNe150 HAALP-45%
2010
Jemain Iprure
Numbor ef
(b) ResNet50 HALP-30%
Figure 12: Visualization of the pruned ResNet50 structure.
task. As shown in Tab. 11
Our algorithm is generic applying to different importance measurements. As shown, using L2 norm of weights as importance measurements leads to slightly lower accuracy.
First-Taye Taylor Expansion (gradient- based)
L2
(magnitude-
Method
FLOPs(G)
Top](%)
Top5(%)
FPS(im/s)
FLOPs(G)
Top1(%)
Top5(%)
FPS(m//s)
10%%
3.0
77.5
93,60
1203
3.0
77.3
93.60
1196
HALP-55%
2.1
76.7
93.16
1672
2.0
75.7
92.66
1595
The results of HALP algorithm on ResNet50 ImageNet classification task with different choices of neuron importance measurements.
Table 11:
N LATENCY LPOK-UP TABLE CREATION AND CALIBRATION
In this section, we provide additional details to build the latency look-up table used in HALP, computational cost and the correlation between the estimated and the real ones. As men- tioned in Appendix A we pre-generate the
Reshiet0
layer latency look-up table on the platform with NVIDIA cuDNN [Chetlur et al.) (2014) V7.6.5. For each layer, we iteratively reduce the num- ber of neurons in the layer (each time reduce neurons) and characterize the corresponding latency. For each latency measurement, we use one profile for GPU warm up and another pro- files and take the average to avoid randomness. The average standard deviation of profiles for an operation is 8.67e-3.
53
97
014
2.5
Expected Labonoy Redustion
Figure correlation between predicted latency reduction the real latency reduction of the praned models.
21

---

Under review as a conference paper at ICLR 2022
On single TITAN V GPU, it takes around hours to build the look-up table for ResNets family and 1 hour for MobileNets family. Note that the LUT can be shared by the network architectures within the same family since they usually have similar layer structures, We only need to create the latency look-up table once for all the possible latency targets.
There indeed some gaps between the predicted latency and the real latency of the model, because the latency look-up table is created layer-wise on convolution layers. There are additional costs in real inference such as pooling, non-linear activation etc. We plot the correlation between the expected latency reduction from look-up table and the real latency reduction ratio of our pruned models in Fig. [3 We also calculate the Pearson Correlation Coefficient for all the networks in the figure. As shown, we can clearly see linear correlation between the predicted and real latency reduction.
O BREAKDOWN OF THE ALGORITHM EXECUTION TIME
We provide additional details of the algorithm process of different methods in this section. We estimate the time cost needed to get the pruned network structure for each method. The time of following finetuning is not taken into consideration. For fair comparison, we set the number of pruning steps & for all iterative pruning methods to 30. All the values are approximated as all the methods are running on the same device (a NVIDIA V100 GPU) to get a pruned ResNet50. For
AutoSlim Yu & Huang (019), MetaPruning Liu et al. (2019)) and AMC He et al. (2018), more GPU time is needed for additional training of the network.
Methed
Draluate poposals?
Auxilurr training?
Sab-netwer's
Additional
Estimate time (KN50)
NenAdape
N candidates
Repert
prane.
Litency look-up craation
GPU
ThiNe:
Y
N
Repeot
pruning.
Addiionnl
importince
2110. GPU
FagleFye
Y
N
1000 condicbiiss colluttion
Mome Carlo smmpling poie 1000 cardidates.
30h CIP
AutoSlim
Y
Y
Train slimmable model & cundidares evaluation
Men2runing
Y
Y
Train mnxilirry network candidares evalustion
AMC
N
Y
Trais in RL agent
Ours)
N
N
Repast 30 tanes, (<) trai quach in total)
60 Iaano Seniiioss afte cach pouting.
Auginanted Krapsack solves (- 90min in koal) Lutency look-up table creation
6. Gi GIU 0. CPU
Table 12: The breakdown details of the execution process of different methods.
P PRUNING TARGETING OTHER HARDWARE
In this section, provide additional experi- ments targeting two additional hardware plat- forms: Intel CPU Xeon E5 and NVIDIA Xavier. In particular, we generate look up ta- bles for those two hardware and compare re- sults to the state-of-the-art EagleEye Li et al. 2020). For inference on CPU, we use batch size of 8; for NVIDIA Xavier, we use batch size 128. Results for this experiment are show in
WAS Sar
Oriinn 122 - Cugaaa - HALP Daan
Figure 14: The ReaNet50 ImageNet pruning targeting inference on dir- ferent bardware.
Fig. 14
As we can see in Fig. 14, our approach consistently outperforms EagleEye, demonstrating the ben- efits of considering the target hardware when performing pruning. HALP leverages the latency characteristics of the target hardware to make a better trade-off between the accuracy and efficiency. Specifically, compared to the original model on the Intel CPU, HALP yields 1.27x speedup with a 0.15% increment in top-1 accuracy: On the NVIDIA Xavier platform, HALP yields a 1.23x speedup with 0.28% increment in top-1 accuracy. Compared to EagleEye, HALP achieves up to 1.45x speedup with similar accuracy. From these results, we can conclude that our method generalizes well to different hardware.
22

---

Under review as a conference paper at ICLR 2022
Q PRUNING FOR INT8 QUANTIZATION
We now focus on results when the target is INT8 inference which is a common require- ment for real-world applications. In particu- lar, we use NVIDIA Xavier the target plat- form as, in this platform, INT8 speedup is sup- ported. We create INT8 latency look-up table for INT8 inference. For comparison, we also create FP32 latency look-up table on the same platform and use both look-up tables for prun- ing a ResNet50 model on ImageNet classifica- tion. After convergence, results quantized into INT8 and the latency and accuracy is mea- sured directly the Xavier platform. We mea- sure latency with a batch size 128, using Ten- sorRT (V8.2.0.1) get INT8 speedup. Results for this experiment are shown in Fig. 15
715
Top! Aop
76.6
73.5
HALP-INTB LUT
HALP-FP32 LUT
Eaglelye
Original model
55
50
70
56
50
100
110
Latency (mi))
Figure 15: The ResNet50 ImageNet pruning targeting INT8 inference NVIDIA Xavier.
As shown, even use a FP32 latency look-up table as the latency guidance, our method HALP out- (2020). These results consistent with
performs the state-of-the-ard method EagleEye Li Sec. 4.2 where we show the HALP acceleration on GPUs with TensorRT. Pruning results of using INT8 look-up table show that our method yields higher accuracy with lower latency on this platform. We obtain a 1.32x speedup while maintaining the original Topl accuracy. Compared to EagleBye, HALP achieves up to 1.26 relative speedup and 0. 15% higher accuracy.
23


Reviews:
[{"review_id": "b60sW6pMebf", "reviewer": "Reviewer_dTnS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\n- This paper is well-written, well-motivated, and clear presentation.\n- The proposed method improves network throughput efficiency with competitive accuracy on classification and detection tasks, outperforming prior pruning approaches.\n- Formulating the hardware-aware structural pruning as a knapsack problem is new.\n\n\nWeaknesses\n\n- The reviewer's main concern is that this work is one of the lookup-table(LUT)-dependent methods that is not new and has critical limitations.\n    - After the emerging MNasNet[1], there are already numerous LUT-based methods improving the efficiency of neural networks under latency-constrained of each target hardwares.\n    - For the case that the correlation between the predicted latencies by LUT and the real latencies is low, the performance of LUT-dependent methods becomes poor. That means the performance of LUT is critical rather than the proposed method. The reviewer guesses that this method also would not guarantee high performance on the target tasks when the performance of LUT is poor.\n    - For considering many hardwares, building LUT process for all hardwares repetitively is not negligible and becomes a heavy burden. We should build LUT for each hardware and we should compile neural networks dependent on device platform types (sometimes it needs domain knowledge of experts).\n    - In addition, the information about LUT such as building process, building time, and correlation between the estimated and the real ones looks omitted in the paper.\n- Generality Issue on multiple hardwares. While this method is 'hardware-aware', the target devices used in the experiments are just two (GPU and Jetson). Recent hardware-aware methods for the efficiency of neural networks under latency constraints have been handled many hardware platforms and device entities. For example, OFA[2] considered 'GPU/CPU/Mobile Phone/Edge GPU/FPGA, BRP-NAS[3] considered 'GPU/CPU/Mobile Phone/Edge GPU/Edge TPU, and HELP[4] considered 'GPU/CPU/Mobile Phone/Edge GPU/FPGA/AISC/Raspberry Pi.\n\n[1] Mnasnet: Platform-aware neural architecture search for mobile, CVPR2019.\n\n[2] Once-for-all: Train one network and specialize it for efficient deployment, ICLR2020.\n\n[3] Brp-nas: Prediction-based nas using gcns, NeurIPS2020.\n\n[4] HELP: Hardware-Adaptive Efficient Latency Predictor for NAS via Meta-Learning, NeurIPS 2021.", "summary_of_the_paper": "This work tackles a latency-constrained structural pruning method by formulating the global resource allocation optimization problem and addressing them via an augmented knapsack solver. During the pruning process, to estimate latency on a target device and accuracy drop, this work uses look up table and global saliency score, respectively. The proposed approach is validated on classification and detection tasks with desktop GPU (Titan V) as the target device.", "summary_of_the_review": "This paper has a clear motivation and introduces a knapsack algorithm for NAS, yet, validates the proposed method on limited devices and has weak novelty since the latency estimation is dependent on LUT.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_-iy2XtEGAp", "reviewer": "Reviewer_ywGD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Contribution**: \n\n1. The authors formulate structural filter pruning as a global resource allocation optimization problem with latency constraint rather than theoretical computation cost (FLOPs).\n2. The authors devise an augmented knapsack to solve the resulting combinatorial optimization problem. \n3. Extensive experiments on image classification and object detection tasks show the promising performance of the proposed method.\n\n**Questions and points needed to be improved**:\n\n1. Algorithm 1 is confusing. Many notations are unclear and not rigorous. For example, what does dp_array denote? What does v_{keep} denote? Moreover, the authors do not provide any explanations regarding Algorithm 1, which makes it hard to follow. More explanations are required.\n\n2. In Section 3.3, the authors state that the proposed method groups the neurons sharing the same channel index from the connected layers in a network with skip connections, which have been proposed in [1-2]. It would be better for the authors to cite the related papers.\n\n3. In Section 3.4, the authors propose to perform pruning for k steps in total, which would improve the performance of the pruned models. It is unclear whether the good performance is resulting from multi-step pruning. It would be better for the authors to provide more ablation studies on the effect of different k.\n\n4. What will happen if the authors replace the latency constraints with FLOPs constraints? Does the performance improvement of the proposed method come from a better metric of computation cost or a better solver? More discussion and experiments are required. \n\n5. The authors conduct experiments on heavyweight networks (e.g., ResNet-50, ResNet-101, VGG-16) and a lightweight network (e.g., MobileNetV1). It would be more convincing for the authors to provide more results on lightweight networks with residual connections (e.g., MobileNetV2 [3]).\n\n6. To demonstrate the effectiveness of the proposed method, it would be better for the authors to provide the detailed configurations (the number of channels of each layer) of the pruned models, which will strengthen the paper.\n\nMinor issues:\n1. Figure 1 is too small to read.\n\n**References**:\n\n[1] Centripetal sgd for pruning very deep convolutional networks with complicated structure. CVPR 2019.\n\n[2] Neural Network Pruning with Residual-Connections and Limited-Data. CVPR 2020.\n\n[3] MobileNetV2: Inverted Residuals and Linear Bottlenecks. CVPR 2018.", "summary_of_the_paper": "The authors propose to formulate structural filter pruning as a global resource allocation optimization problem with latency constraints. To solve the resulting knapsack problem, the authors devise an augmented knapsack solver and further propose neuron grouping to reduce the pruning space and computational cost. Extensive experiments on image classification and object detection tasks show the promising performance of the proposed method. \n", "summary_of_the_review": "The performance is promising. However, some important details of the proposed method are missing, which makes it hard to follow. Moreover, experiments are not sufficient. More experiments are required to show the effectiveness of the proposed method.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ZVw-6WTrLBo", "reviewer": "Reviewer_2vJd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Importantly, the paper works on a very important topic. Model compression is very important when it comes to deployment to ensure QoS of the inference services.\n\nI believe the main benefits of the work are (1) formulating hardware-aware pruning while retaining good performance and (2) doing this efficiently. The formulation of the overall pruning procedure into a Knapsack problem seems to be where the main benefit comes from. Neuron grouping proposed in the paper also seems to be another source of speedup.\n\nQuestions I have are:\n\n* While the paper claims that this can be applied to other platforms too. However, this requires re-generating the look-up table for HALP. Could you provide how long this would take for a new platform? Can this be done efficiently?\n* How would this perform in relation to quantization? And, would similar mechanism for enforcing hardware-awareness in compression work for quantization?", "summary_of_the_paper": "This paper is an effort to perform structured pruning with regard to inference latency on hardware. The paper solves this by creating a look-up table of \"importance score\" and \"latency contribution\" and formulates this into a Knapsack problem, which can be solved efficiently. Evaluation shows that this method effectively reduces the latency with regard to hardware via pruning.", "summary_of_the_review": "Hardware-awareness of pruning, while touched upon by some works, still opens up large potential for inference latency reduction. This paper proposes an interesting direction in optimization where each layer is assigned importance score and measured latency of each layer. Then, the overall pruning problem is formulated as a knapsack problem. Overall performance of the pruned network seems reasonable. I liked reading the paper, and I would like the paper to be in the program.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1rOosSNprtm", "reviewer": "Reviewer_DgPf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n==+== B. Strengths \n\nThe evaluation is strong and the analysis is thorough. The Pareto curves show the effectiveness of the algorithm.\nThe writing is clean. \n\n==+== C. Weaknesses  \n\nIn my opinion, the novelty is not very exciting.\nThere are a lot of design choices (e.g., computing important score) and the alternatives are not discussed.\n\nQuestions:\n\nCan you make a comparison between dynamic channel pruning (e.g., https://arxiv.org/pdf/1810.05331.pdf)? The term `dynamic grouping’ is confusing in your paper, you should clarify that it happens during training but not evaluation. Also, my intuition is that using dynamic pruning during evaluation can push the Pareto curves further (fig.3); so it should be an important baseline. \n\nThere is also a lot of design options in each part of your design. For example: 1) Computing the importance score you can use the method here (https://arxiv.org/abs/1810.02340) by computing the gradient of each neuron. 2) Regarding the latency prediction, you mentioned that latency is not linear to the number of FLOP; however, for a layer-wise model architecture (most of the models in your evaluation are layer-wise), this is not always true. I am wondering in which scenario latency and FLOPs are not linearly related.\n\nIn Table 3, are you sure the baselines (~8000h) are fair compared to yours (30m)? As far as I can tell, you did not include the extra GPU training time. The number reported is confusing, can you possibly show a time breakdown of your compilation process?\n\nMinor:\nYour title is so vague and it looks like a survey paper. You can consider changing the name of your paper.", "summary_of_the_paper": "==+== A. Paper summary\n\n\nThis paper proposes a method to do channel pruning. By building up latency and important score for each group (or channel), the paper use an ILP solver to select the channels that can achieve the best trade-off between latency v.s. Accuracy. The evaluation shows that the method achieves better speedup compared to traditional channel pruning with almost no accuracy loss.\n", "summary_of_the_review": "The quality of this paper is good. The experiments are thorough and solid. However, I think the novelty of this paper is limited.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a hardware-aware pruning method which structurally prunes the given deep neural networks to retain their accuracy while satisfying the latency constraints. Specifically, the authors formulate the latency-constrained pruning problem as a combinatorial optimization problem to find the optimal combination of neurons to maximize the sum of the importance scores, and propose an augmented knapsack solver to solve it, as well as a neuron grouping technique to speed up the training. The proposed method is validated for its classification tasks on two devices, namely Titan V and Jetson TX2, and for object detection performance on Titan V, and is shown to achieve superior accuracy/latency tradeoff compared to existing pruning methods, including latency-aware ones.

The paper received split reviews initially, and the following is the summary of the pros and cons mentioned by the reviewers.

Pros
- The proposed formulation of the latency-constrained pruning problem as a constrained knapsack problem is novel. 
- The method achieves competitive performance against existing latency-constrained pruning methods. 
- The paper is written well, with clear motivation and descriptions of the proposed method.

Cons
-  The idea is not very exciting since posing pruning as a combinatorial optimization problem, or a knapsack problem is not new, and the proposed method only adds in additional latency constraints.
- The title “hardware-aware” is vague and misleading since what the authors do are latency-constrained pruning.
- The experimental validation is only done on two devices, which makes the method less convincing as a “hardware-aware” method and how it generalizes to other devices (e.g. CPU, FPGA)
- Use of lookup tables to obtain the latency constraints is not novel, has a limited scalability, and is inefficient.  
- Missing discussion of design choices. 

During the discussion period, the authors cleared away some of the concerns, which resulted in two of the reviewers increasing their scores. However, one reviewer maintained the negative rating of 5, and the positive reviewers were still concerned with limited novelty.

I believe that this is a good paper that proposes a neat solution for latency pruning, which may have some practical impact. However, the novelty of the idea is limited, as pointed out by the reviewers. The use of lookup tables also does not seem to be an efficient solution for adapting to edge devices for which the collection of latency measurements could be slow. The experimental validation on only two devices of the same type (GPU) also seems insufficient, as how the method generalizes to diverse devices is uncertain. It would be worthwhile to consider using a latency predictor (e.g. BRP-NAS [Dudziak et al. 20]), and perform experimental validation on diverse hardware platforms (e.g. CPU and FPGA). Comparing against recently proposed hardware-aware NAS methods could be also interesting, as there has been a rapid progress on the topic recently.

Thus, despite the overall practicality and the quality of the paper, the paper may benefit from another round of revision, since both the method and the experimental validation part could be improved. 

[Dudziak et al. 20] BRP-NAS: Prediction-based NAS using GCNs, NeurIPS 2020
Decision: Reject