Adaptive Activation-based Structured Pruning
Authors: Kaiqi Zhao
Animesh Jain
Ming Zhao
Venue: ICLR 2022 Submitted (2022)
Keywords: model compression
structured pruning

Abstract: Pruning is a promising approach to compress complex deep learning models in order to deploy them on resource-constrained edge devices. However, many existing pruning solutions are based on unstructured pruning, which yield models that cannot efficiently run on commodity hardware, and require users to manually explore and tune the pruning process, which is time consuming and often leads to sub-optimal results. To address these limitations, this paper presents an adaptive, activation-based, structured pruning approach to automatically and efficiently generate small, accurate, and hardware-efficient models that meet user requirements. First, it proposes iterative structured pruning using activation-based attention feature maps to effectively identify and prune unimportant filters. Then, it proposes adaptive pruning policies for automatically meeting the pruning objectives of accuracy-critical, memory-constrained, and latency-sensitive tasks. A comprehensive evaluation shows that the proposed method can substantially outperform the state-of-the-art structured pruning works on CIFAR-10 and ImageNet datasets. For example, on ResNet-56 with CIFAR-10, without any accuracy drop, our method achieves the largest parameter reduction (79.11%), outperforming the related works by 22.81% to 66.07%, and the largest FLOPs reduction (70.13%), outperforming the related works by 14.13% to 26.53%.

Under review as a conference paper at ICLR 2022
ADAPTIVE ACTIVATION-BASED STRUCTURED PRUN- ING
Anonymous authors
Paper under double-blind review
ABSTRACT
Pruning a promising approach compress complex deep learning models order to deploy them on resource-constrained edge devices. However, many ex- isting pruning solutions are based on unstructured pruning, which yield models that cannot efficiently run on commodity hardware, and require users to manu- ally explore and tune the pruning process, which is time consuming and often leads to sub-optimal results. To address these limitations, this paper presents adaptive, activation-based structured pruning approach to automatically and ef- ficiently generate small, accurate, and hardware-efffcient models that meet user requirements. First, it proposes iterative structured pruning using activation-base attention feature maps to effectively identify and prune unimportant filters. Then, it proposes adaptive pruning policies for automatically meeting the pruning ob- jectives of accuracy-critical, memory-constrainee and latency-sensitive tasks. A comprehensive evaluation shows that the proposed method can substantially out- perform the state-of-the-art structured pruning works on CIFAR-10 and ImageNet datasets. For example, on ResNet-56 with CIFAR-10 10, without any accuracy drop. our method achieves the largest parameter reduction (79.1 11 1)), outperforming the related works by 22.81 % to 66.07%, and the largest FLOPs reduction (70. 13%), outperforming the related works by 14. 133% to 26.53%.
INTRODUCTION
Deep neural networks (DNNs) have substantial compute and memory requirements. As deep learn- ing becomes pervasive and moves towards edge devices, DNN deployment becomes harder because of the mistmatch between resource- resource-hungry DNNs and resource-constraiigg
edge devices. DNN Molsc et al. (2016); Theis et al. 2018); Renda et al. (2020)), which identifies the parameters (or weight elements) that do not contribute significantly to the accuracy and prunes them from the network. Recently, works based on the Lottery Ticket Hypothesis (LTH) have achieved great successes in creating smaller and more curate models through iterative pruning with rewinding (Frankle & Carbin (2018)). However, LTH has only been shown to work successfully with unstructured pruning which, unfortunately leads to models with low sparsity and difficult to accelerate on commodity hardware such as CPUs and GPUs (e.g., Hill al. (2017) shows directly applying NVDIA cuSPARSE on unstructured pruned models can lead to 60x slowdown on GPU compared to dense kernels.) Moreover, most pruning methods require users explore and adjust multiple hyper-parameters, e.g., with LTH-based iter- ative pruning, users need to determine how many parameters to prune in each round. Tuning the pruning process is time consuming and often leads sub-optimal results.
pruning a promising approach (Li et al.
|l.] (2016); Han al. (2015): MMochanov
We propose activation-based adaptive, iterative structured pruning to find the "winning ticket" models that are at the same time hardware efficient and to automatically meet the users' model accuracy, size, and speed requirements. First, we propose an activation based structured pruning method to identify and remove unimportant filters in an LTH-based iterative pruning (with rewind- ing) process. Specifically, we properly define an attention mapping function that takes a 2D acti- vation feature maps a filter as input, and outputs a 1D value used to indicate the importance of the filter. This approach is more effective than weight-value based filter pruning because activation- based attention values not only capture the features of inputs but also contain the information of convolution layers that feature detectors for prediction tasks. We then integrate this attention-

---

Under review as a conference paper at ICLR 2022
based method into the LTH-based iterative pruning framework to prune the filters in each round and find the winning ticket that is small, accurate, and hardware-efficient.
Second, we propose adaptive pruning that automatically optimizes the pruming process according to different user objectives. For latency-sensitive scenarios like interactive virtual assistants, we propose PL.Ossguarantted pruning to achieve the best accuracy given the maximum amount of compute FLOPs; For memory-limited environments like embedded systems, we propose model- size-guaranteed pruning achieve the best accuracy given the maximum amount of memory foot- print; For accuracy-critical applications such as those on self-driving cars, we propose accuracy- guaranteed pruning to create the most resource-efficient model given the acceptable accuracy loss. Aiming for different targets, our method adaptively controls the pruning aggressiveness by adjust- ing the global threshold used to prune filters. Moreover, it considers the difference in each layer's contributions to the model's s size and computational complexity and uses a per-layer threshold, cal- culated by dividing each layer's remaining parameters or FLOPss by the entire model's remaining parameters or FLOPs, to prune each layer with differentiated level of aggressiveness.
Our results outperform the related works significantly all cases targeting accuracy loss, parame- ters reduction, and FLOPs reduction. For example, ResNet-56 with CIFAR-10 dataset, without accuracy drop, our method achieves the largest parameter reduction (7), outperforming the related works by 22.81 & to 66.07%, and the largest FLOPs reduction (70.13%) outperforming the related works by 14.13% to 26.53%. In addition, our method enables a pruned model the reach 0.6% 1.08% higher accuracy than the original model but with only 30% 50%of the original's parameters. On ResNet-50 on ImageNet, for the same level of parameters and FLOPs reduction, our method achieves the smallest accuracy loss, lower than the related works by 0.08% to 3.21 %;; and for the same level of accuracy loss, our method reduces significantly more parameters (6.45% to 29. 29.619 % higher than related works) and more FLOPs (0.82% to 17.2% higher than related works).
2 BACKGROUND AND RELATED WORKS
Unstructured vs. Structured Pruning. Unstructured pruning (LeCun et al. (1990); Han et al. 2015);; Molchanov et al.] (207)) is fine-grained approach that prunes individual unimportant ele- ments weight tensors. It has less impact to model accuracy, compared to structured pruning, but unstructured pruned models are hard to accelerate on commodity hardware. Structured pruning is a coarse- graiined approach that prunes entire regular regions of weight tensors according to some rule-based heuristics, such as L.-norrm (Li et al. (2016)), average percentage of zero (Molchanov 2. and other information considering the relationship between neighboring layers (Theis al.
et al. 20088) Lee et al. (2018)) Compared to unstructured pruning, it is more difficult to prune model without causing accuracy loss using structured pruning. because by removing entire regions, it might remove weight elements that are important to the final accuracy (Li et al. (2016)). How- ever, structured pruned models can be mapped easily to general-purper hardware and accelerated directly with off tff-tf hardware and libraries (He et al. (2018b).
One Shot vs. Iterative Pruning. One-shot pruning prunes a pre-trained model and then retrains once, whereas iterative pruning prunes and retrains the model in multiple rounds. Both techniques can choose either structured or unstructured pruning techniques. Recently, works based on the Lot- tery Ticket Hypothesis (LTH) have achieved great successes in creating smaller and more accurate models through iterative pruning with rewinding (Frankle & Carbin (218)). ). LTH posits that a dense randomly initialized network has a sub-network, termed as a winning ticket, which can achieve an accuracy comparable to the original network. At the beginning of each pruning round, it rewinds the weights and/or learning rate of the sub-network to some early epoch of the training phase of the original model to reduce the distance between the sub- network and original model and increase the change of finding the winning ticket. However, most of LTH-based works considered only un- structured pruning, e.g., Iterative Magnitude Pruning (IMP) (Frankle & Carbin 2018);; Frankle et al. 22019), which, as discussed above, is hardware- inttffriinnt....
It is non-trivial to design iterative pruning method with structured pruning. To understand the state of iterative structured pruning, we experimented with IMP's structured pruning counterpart LL- norm structured pruning (ILP) (Li et al. (2166) which removes entire filters depending on their L1- norm value. We observed that ILP cannot effectively prune a model while maintaining its accuracy, e.g., IIP can prune ResNet-50 by most 11.5% of parameters when the maximum accuracy loss
2

---

Under review as a conference paper at ICLR 2022
Algorithm Adaptive Iterative Structured Pruning Algorithm
1: [Initialize] Initialize a network f(; M O W with initial mask M {0. 1 and Threshold [[0]
2: [Save weights] Train the network for epochs, yielding network / (a; M @ WT), and save weights W
3: (Train to converge] Train the network for T k epochs to converge, producing network f(z; M о WT)
4: for pruning round (r 1) do
5: [Prune] Prune filters from w using T[r]. producing mask M and network f(z; M" о OWT (WF
6: [Rewind Weights] Reset the remaining filters to W epoch k. producing network f(x; M" OW
7: [Rewind Learning Rate] Reset the learning rate schedule to its state from epoch k
[Retrain] Retrain the unpruned filters for T k epoch to converge, yielding network f(a; M WF)
9:
[Evaluate] Evaluate the retrained network f(x; M" O W according to the target.
10:
|Reset Weights] If the target is not met, reset the weights to an carlier round
11:
[Adapt Threshold] Calculate Threshold T(r 1] for the next pruning round
12:
end for
Original
Input
filler 1
fille_2
fiter_3 3
fillr
t
4
w
hiter
liter
Titer
fitrr
Conv2d
ReLu
/
Inputs
A
filer
fiter 10
fiter
filtr 12
ni!!!
Outputs
fitor 13
hiter 14
nitor
filler 16
F
Fj
F!
nt Filters
Filter activation outputs
(b) Convolution layer tensors
Illustration of the (a) activation outputs of 16 filters of conv2d layer, and (b) input and output tensors of a convolution layer. In (a), the left portion shows the original image and the image after data augmentation; and the right portion visualizes the activation outputs of each filter.
Figure 1:
limited to 1% on ImageNet. Therefore, directly applying iterative pruning with existing weight- magnitude based structured pruning methods does not produce accurate pruned models. In this paper, we study how produce small, accurate, and hardware-efficien models based on iterative structured pruming with rewinding.
Automatic Pruning, For pruning be useful in practice, is important to automatically meet the pruning objectives for diverse ML applications and devices. Most pruning methods require users to explore and adjust multiple pyper-parameters, e.g., with LTH-based iterative pruning, users need to determine how many parameters to prune in each round. Tuning the pruning process is time consuming and often leads to sub-optimal results. Therefore, we study how to automatically adapt the pruning process in order to meet the pruming objectives without user intervention.
Some works (Zoph et al. 2018); Cai et al.; Ashok et al. (2017) use reinforcement-leaminn algo- rithms to find computationally efficient architectures. These works in the area of Neural Archi- tecture Search (NAS), among which the most recent is AutoML for Model Compression (AMC) (He al. (0188)). AMC enables the model arrive the target speedup by limiting the action space (the sparsity ratio for each layer), and it finds out the limit of compression that offers no loss of accuracy by tweaking the reward function. But this method has explore a large search space of all available layer-wise sparsity, which is time consuming when neural networks are large and datasets are complicated.
METHODOLOGY
Algorithm Iillustrates the overall flow of the proposed adaptive activation- based structured pruning- To represent pruning of weights, we use mask M (0, 1):d for each weight tensor WTeR' eRd, where is the pruning round number and is the training epoch. Therefore, the pruned network at the end
3

---

Under review as a conference paper at ICLR 2022
of training epoch T is represented by the element-wist product M + W7. The first three steps to train the original model to completion. while saving the weights at Epoch kt Steps 4-10 represent pruning round. Step 4 prunes the model (discussed in Section Step (optional) and perform rewinding. Step 7 retrains the pruned model for the remaining 7 - k epochs. Step 8 evaluates the pruned model according to the pruning target. If the target is not met, Step 9 resets the weights to an earlier round. Step 10 calculates the pruning threshold for the next pruning round following the adaptive pruning policy (discussed in Section 3.2
3.1 ACTIVATION-BASED FILTER PRUNING
To realize iterative structured pruning, one can start with the state-of- of-te-of-the-asdd art structured pruning meth- ods and apply it iteratively. The widely used structured pruning method-Li- Ln-morm based structured pruning Renda et al. (2020) removes filters with the lowest LI Ll-norm values, whereas the most recent Polarizatti--------aaa-- based structured pruning Zhuang et al. (2020) improves it by using a regular- izer on scaling factors of filters and pruning filters whose scaling factors are below than a threshold. These two methods both assume that the weight values of a filter can be used as an indicator about the importance of that filter, much like how LTH uses weight values in unstructured pruning. However, we observe that weight-basse els. For example, to prune ResNet-56 CIFAR-10 with no loss in top-1 accuracy, L1-norm based structured pruning can achieve at most only 1.15x model compression, and Polarization-nort based structured pruning can achieve at most only 1.89 inference speedup. The reason is that, some fil- ters, even though their weight values are small, can still produce useful non-zero activation values that are important for learning features during backpropagation. That is, filters with small values may have large activations.
structured pruning methods cannot produce accurate pruned mod-
We propose that the activation values of filters are more effective in finding unimportant filters to prune. Activations like ReLu enable non-linear operations, and enable convolutional layers to act as feature detectors. If an activation value is small, then corresponding feature detector is not important for prediction tasks. So activation values, i.e., the intermediate output tensors after the non-linear activation, not only detect features of training dataset, but also contain the information of convolution layers that act as feature detectors for prediction tasks. We present visual motivation in Figure The figure shows the activation output of 16 filters of a convolution layer on one input image. The first image on the left is the original image, and the second image is the input features after data augmentation. We observe that some filters extract image features with high activation patterns, e.g., the 6th and 12th filters. In comparison, the activation outputs of some filters are close to zero, such as the 2nd, 14th, and 16th. Therefore, from visual inspection, removing filters with weak activation patterns likely to have low impact on the final accuracy of the pruned model.
There is a natural connection between our activation-based pruning approach and the related attention-based knowledge transfer works (Zagoruyko & Komodakis (2016)). Attention is a map- ping function used to calculate the statistics of each element in the activations over the channel dimension. By minimizing the difference of the activation- based attention maps from intermediate layers between teacher model and a student model, attention transfer enables the student to imi- tate the behaviour of the teacher. Our proposed pruning method builds upon the same theory that activation -based attention is good indicator of filters regarding their ability to capture features, and addresses the new challenges in using activations to guide automatic structured pruning.
In the following, we describe how to prune filters based on its activation feature maps in each round of the iterative structured pruning process. Figure [[b] shows the inputs and outputs of a 2D convolution layer (referred to as conv2d), followed by an activation layer. For the ith conv2d layer, let XteRR" ---x== xh-' x="-i denote the input features, and FfeRP"' "k" xk" be the jth filter, where 1- and wi are the height and width of the input features, respectively, n n is the number of input channels, n' is the number of output channels, and k is the kernel size of the filter. The activation of the jth filter F after ReLu mapping is therefore denoted by A), xw"
The attention mapping function takes 2D activation AfeR'* 'wo of filter F7 as input, and outputs a ID value which will be used as an indicator of the importance of filters. We consider three forms of activation-based attention maps, where p I/ 1 and af, denotes every element of A): 1) Mean of
18 2) Max of atten-
attention values raised to the power of p: Fmenn (At) = n
Eh Σn 0x.1
4

---

Under review as a conference paper at ICLR 2022
Algorithm 2 Accuracy-guarantes Adaptive Pruning
1: Input: Target Accuracy Loss AccLoss'I argee
Output: small pruned model with an acceptable accuracy
3: Initialize: 7 0.0, 0.005.
4: for pruning round r (r > 1) do
5: Prune the model using [rr] (Refer to Algorithm 3
6: Train the pruned model, evaluate its accuracy Acc[r]
Calculate the accuracy loss AccLoss[r]: AccLoss[r] Ace[0] Acefr] Acc < AccLossT Trrgel then if the changes of model size are within 0.1 % for several rounds then Terminate
8:
9:
10:
else
12:
A[r
+ 1] Xrr + 1 + A[r 1]
13:
14:
end
if
15: else
Find the last acceptable round
17:
if has been used to roll back for several times then
18:
Mark k as unacceptable
19:
Go to Step 15
20:
else
21:
Roll back model weights to round
22:
A/r 1|
+1) (N is the number of times for rolling back round k)
23:
T = [[k A(r
24:
end if
25:
end if
26:
end for
tion values raised to the power of Fma (A)) = maz -
a and 3) Sum of attention
values raised the power of p. (A)) Σn Σ a We choose Fmean (Aj) with p equals to 1 as the indicator to identify and prune unimportant filters, and our method removes the filters whose attention value is lower than the pruning threshold. See Section 5 for an ablation study on these choices.
3.2 ADAPTIVE ITERATIVE PRUNING
Our approach to pruning is to automatically and efficiently generate a pruned model that meets the users' different objectives. Automatic pruming means that users do not have to figure out how to configure the pruning process. Efficient pruning means that the pruning process should produce user-desired model as quickly as possible. Users' pruning objectives can vary depending on the usage scenarios: 1) Accuracy-critical tasks, like those used by self-driving accuracy requirement, which is critical for safety, but do not have strict limits on their computing and storage usages; 2) Memory-constrained tasks, like those deployed on mierocontrollens have very limited available memory to store the models but do not have strict accuracy requirement; and 3) Latency-sensitive tasks, like those employed by virtual assistants where timely responses are desirable but accuracy is not hard constraint.
have stringent
In order achieve automatic and efficient structured pruning, we propose three adaptive pruning policies to provide 1) securacy-guaranter Adaptive Pruning which produces the most resource- efficient model with the acceptable accuracy loss; 2) Memory-constrrined Adaptive Pruning which generates the most accurate model within a given memory footprint; and 3) FLOPs-constrained Adaptive Pruning which creates the most accurate model within a given computational intensity. Specifically, our proposed adaptive pruning method automatically adjusts the global threshold (T), used in our iterative structured pruning algorithm (Algorithm 1) to quickly find the model that meets the pruning objective. Other objectives (e.g., limiting model's energy consumption) can also be supported by simply plugging in the relevant metrics into our adaptive pruming algorithm. We take Accuracy-guaranter guaranteed Adaptive Pruning, described in Algorithm as an example show the pro- cedure adaptive pruning. cemoory-onstraiine and FLOPs-constraineed Adaptive Pruning algo- rithms included in Appendix and A.4 respectively.
5

---

Under review as a conference paper at ICLR 2022
Algorithm 3 Layer-aware Threshold Adjustment
1: for the layer in pruning round do
2: Calculate the number of parameters unpruned filters the current layer and the whole model, respec-
tively: ['[r] = n [r- x x [r]x [r]] N [r] E n (2t">((r [r]] k
x [x] [r]xn''[] [r],
where M is the number of convolution layers, and for other notations refer to Section 3.1
3: Calculate the threshold [r] of the current layer: [']r] [r] - T[r] x [
N*[r]
4: For each filter Fj in F', calculate its attention: Fmean (mean(A)) = xxx=" Σx-1 Σn-1 o
5: Prune the filter if its attention is not larger than threshold T [r]] i..., set Mj = 0 Pmeaa (A3) & [r]:
Otherwise, set Mj = 1
6: end for
In Algorithm/ 2, the objective is to guarantee the accuracy loss while minimizing the model size (the algorithm for minimizing the model FLOPs is similar). In the algorithm, T controls the ag- gressiveness of pruning, and À determines the increment T' each pruning round. Pruning starts conservatively, with T initialized to 0, so that only completely useless filters that cannot capture any features are pruned. After each round of pruning, if the model accuracy loss is below than the target accuracy loss, it is considered "acceptable", and the algorithm increases the aggressiveness of pruning by incrementing T by As pruning becomes increasingly aggressive, the accuracy eventually drops below the target accuracy at certain round which is considered "unacceptable". When this happens, our algorithm rolls back the model weights and pruning threshold to the last acceptable round where the accuracy loss is within the target, and restarts the pruning from there but more conservatively- it increases the threshold more slowly by cutting the À value by half. If this still does not lead to acceptable round, the algorithm cuts À by half again and restarts again. If after several trials, the accuracy loss is still not acceptable, the algorithm rolls back even further and restarts from an earlier round. The rationale behind this adaptive algorithm is that the aggressiveness of pruning should accelerate when the model is far from the pruning target, and decelerate when it close the target. Eventually, the changes of model size converges (when the changes of the number of parameters is within 0.1 % for several rounds) and the algorithm terminates.
3.3 LAYER-AWARD THRESHOLD ADJUSTMENT
While adapting the global pruning threshold using the above discussed policies, our pruning method further considers the difference in each layer's contribution to model size and complexity and use differentiated layer-specific thresholds to prune the layers. As illustrated in Figure [[b] in terms of the contribution to model size, the number of parameters of layer i can be estimated as N n (1-1) x k x k x n': in terms of the contribution to computational complexity, the number of FLOPs of layer i can be estimated as x h' w² x N². A layer that contributes more to the model's size or FLOPs is more likely to have redundant filters that can be pruned without affecting the model's accuracy. Therefore, to effectively prune a model while maintaining its accuracy, we need to treat each layer differently at each round of the iterative pruning process based on its current contributions to the model size and complexity. Specifically, our adaptive pruning method calculates weight for each layer based on its contribution and then uses this weight to adjust the current global threshold and derive a local threshold for the layer. If the goal is to reduce model size, the weight is calculated each layer's 's number of parameters divided by the model's 's total number of parameters; the goal is to reduce model computational complexity, the weight is calculated as each layer's s FLOPS divided by the model's total FL.OPs. These layer-specific thresholds then used to prune the layers in the current pruning round.
Assuming the goal is reduce model size, the procedure in a pruning round r for the ith conv2d layer is shown in Algorithm 3 (the algorithm for reducing model FLOPs is similar). For pruning, we introduce a mask Mfc (0, 1)ad x x for each filter Fj. The effective weights of the convolution layer after pruning is the element-wise product of the filter and mask-F" FF O M3. For a filter, all values of this mask is either 0 or 1, thus either keeping or pruning the entire filter."

---

Under review as a conference paper at ICLR 2022
Table 1: Comparing the proposed method with oate-of-thhe-rt structured pruning methods the CIFAR-10 dataset. The baseline Top-1 accuracy of ResNet-56 and ResNet-50 on CIFAR-10 is 92.84%, 91.83% (5 runs), respectively. The numbers of the related works are cited from their papers.
Model
Target
Target Level
Method
Acc. (
Params. (%)
FLOPs. (S)
SCOP
0.06
56.30
HRank
0.09
42.40
50.00
0%
II.
0.00
13.04
PPR
-0.03
47.00
EagleEye
-1.4
50.41
Acc. (9t)
Ours-1
-0.33
79.11
Ours-2
-0.08
70.13
RTR
1:00
50.00
1%
GAL
0.52
44,80
48.50
Ours-1
0.86
88.23
Ours-2
0,77
81.19
DCP
-0.01
70.30
47.10
R6
70%
IRRank
2.38
68.10
74.10
Params. 4 (%)
Ours
-0.6
71.57
CP
1.00
50.00
SFP
1.33
50.60
$2.60
50%
FPOM
0.10
50.60
$2.60
Ours
-1.08
53.3
HRank
2.38
74,10
DH
2.54
71.00
PPR
1.17
71.00
Ours
0.3
71.44
FLOPS. (%)
CP
50.00
1.33
55%
FRGM
0.10
50.60
52.60
AMC
0.90
50.00
Ours
-0.63
52.92
AMC
-0.02
60.00
ResNet-50
Params. J. (%)
60%
Ours
PFS
40.9
SCP
0.06
66.23
VCNNP
0.07
60.9
Acc. (%+
0%
Hinge
0.43
39.07
VGG-16
Ours
-0.16
72.85
61.17
GDP
-0.1
69.45
Params. J. (98)
70%
Ours
-0.29
70.54
41,72
Eigendamage
0.19
78.18
37.13
Acc.
NN Slimming
VGG-19
Ours
-0.26
85.99
56.22
FLOPs. (%6
85%
Eigendanage
1,88
86.51
Ours
1.11
89.02
GDP
-0.26
46.22
Acc.
MDP
MobileNerV2
Ours
-0.54
60.16
38.93
SCOP
0.24
Params. J. (%)
40%
DMC
-0.26
40
Ours
-0.8
43.77
78.65
EVALUATION
4.1
CIFAR-10 RESULTS
Table I lists the results from CIFAR-I0 on ResNet-56, ResNet-50, VGG-16, 19G-19, and Mo- b22eNetV2. We compare our method with the state-offthe works, including SCOP (Tag et al.
2020 HRank (Lin et al. (2020)) ILP (Renda et al.] (2020). PPR (Zhuang et al. (2020)) RFR
(H et al. (2017)) GAL (Lin et al. (2019)), DCP (Zhuang et al. (2018), GBN You et al. 2019 ).
CP (He al. (2017)), SFP (He al. (2018 8a)), FPGM (He et al. (019)). DeepHoyer (DH) (Yang
et al. (2019). AMC (He et al. (0)8))), PFS (Wang et al. (2020)). SCP (Kang & Han 2020 )
VCNNP (Zhao et al. (2019)), Hinge (Li al. (2020)). GDP Guo al. (202 Eigendamage
Wang et al. (2019). NN Slimming (Liu et al. (007)). MDP Guo et al. (2020) ), and DMC (Gao)
al." (0201). Note that negative accuracy drop means that the pruned model achieves better accu-

---

Under review as a conference paper at ICLR 2022
Table 2: Comparing the proposed method with state-of-the-art structured pruning methods on ResNet-50 with ImageNet dataset, and VGG-19 on Tiny-ImageNet. The result of ILP is from our implementation, and all the other numbers of the related works are cited from their papers.
Model
Targets
Targets Level
Mothod
Acc. 1 (%)
Params. 1 (8)
FLOPs (%)
PEP-A
0.22
18.10
10.80
Ours-1
Acc. (%6
Ours-2
0.18
11.62
SSS-41
0.68
15.06
ILP
1,00
11.5
Ours-1
0.64
30.39
ReaNet-50
Ours-2
0.83
32,26
Hrank
1.17
36,70
43.70
SSS
4.30
Params. J (%)
Ours
1.09
37.17
PFP-B
0.92
30.10
44.00
30%
Ours
0.48
29.42
FL.PPs. U (%)
30%
SSS 32
1.94
27.06
31.08
Ours
0.57
31.88
DagenDanage
3.36
61.77
66.21
Acc.
3%
Ours
2.61
78.97
VGG-19
NN Slimmin
10.66
60.14
Params. J. (%)
60%
Ours
-0.34
60.21
racy than its unpruned baseline model. "Ours-1" and "Ours-2" denote the cases where the pruning objective is set to minimize model size and FLOPs, respectively, while meeting the accuracy target.
In all cases targeting accuracy, model size, and compute intensity, our method significantly outper- forms the recent related works. For example, on ResNet-56, without accuracy drop, our method achieves the largest parameter reduction (79.11%) outperforming the related works by 22.81% to 66.07%, and the largest FLOPs reduction (70.13%), outperforming the related works by 14.13% to 26.53%. With 70% of parameter reduction, method achieves the smallest accuracy loss (-0.6%), outperforming the related works by 0.59% to 2.98%. Also, with 75% of FLOOPs reduction, our method achieves the smallest accuracy loss (0.3%), outperforming the related works by 0.87% to 2.24%. Note that the proposed method also produces a pruned model that reaches 0.6% or 1.08% higher accuracy than the original model but with only 30% or 50, respectively, of the original's parameters. Such small and accurate models are useful for many real-world applications.
On VGG-16, without accuracy drop, our method achieves the largest parameter reduction (72.85%), outperforming the related works by 6.62% to 33.78%; With 70% of parameter reduction, the accu- racy loss achieved by our method is lower than GDP by 1.19% On VGG-19, with no accuracy loss, our method achieves the largest parameter reduction (85999% which outperforms the related works by 5.92% to 7.81% and the largest FLOPs reduction (56.22%) which outperforms the related works by 13.57% 19.09%; With 85% of FLOPs reduction, the accuracy loss achieved by our method is lower than EigenDamage by 0.07%.
On MobileNet. without accuracy drop, our method achieves the largest parameter reduction (60.16%), outperforming GDP and MDP by 13.94% and 31.45%, respectively; With 40% of pa- rameter reduction, the accuracy loss achieved by our method is lower than SCOP and DMC by 1.04% and 0.54%, respectively.
The results validate that activation- pruning is more effective than weight magnitude based pruning in finding unimportant filters, as discussed in 3.. For example, on ResNet-56, with 1% of accuracy loss, the amount of parameter reduction achieved by our method is higher than that of ILP (Renda et al. (2020) by 34.51%; %;; and with 75% FLOPs reduction, the accuracy achieved by our method is higher than PPR (Zhuang et al. (2020) by 0.87%.
The results confirm that our method can achieve better results that the state-of-the-art automatic pruning methods, AMC (He al. (2018b)) For example, when targeting about 55% of FLOPs reduction on ResNet-56, our method achieves 1.53% higher accuracy than AMC; and when tar- geting at about 60% of parameters reduction on ResNet-50, our method achieves 0.84% higher accuracy than AMC. Our activation-based adaptive pruning can directly target unimportant filters in the model, whereas AMC has to use reinforcement learning to explore the whole network.
4.2 IMAGENET RESULTS
Table 2 shows the results of ResNet-50 on ImageNet dataset and VGG-19 on dataset, comparing method with PFB (Liebenwein al. (2019).. HRank (Lin al. 2020
8

---

Under review as a conference paper at ICLR 2022
Table 3: Results of one-shot pruning with differ- ent types of attention mapping functions and L1 Norm for VGG-16 on CIFAR-I0 dataset.
Table 4: Results of the adaptive pruning strategy and non-adaptive iterative pruning strategies for VGG-16 on CIFAR-I0 dataset.
Method
Acc. (%)
Attention Mean (p=1)
0.38
Method
Acc. 4 (%)
Params. (%)
Attention Mean (p=2)
0.42
IAP
0.9
69.83
Attention Mean (p=4)
0,47
ILP
1.1
69.83
Attention Sum (p=1)
0.63
Ours
-0,29
70.54
Attention Max (p=1)
0,51
LI Norm
0.48
ILP (Renda et al. (2020)) Sparse Structure Selection (SSS) (Huang & Wang (0118). ) NN Slimming LLu et al. (017)) and Eigendamage (Wang et al. (219)). "Ours-I" and "Ours-2" denote the cases where the pruning objective is set to minimize the number of parameters and FLOPs, respectively, while meeting the accuracy target. On ImageNet, for the same level of accuracy loss, our method reduces significantly more parameters (6.45% to 29.61% higher than the related works) and more FLOPs (0.82% to 17.2% higher than the related works). For the same level of parameters or FLOPs reduction, our method achieves the smallest accuracy loss, lower than the related works by 0.08% to 3.21%. % On Tiny-ImageNet, for the same level of parameters reduction, our method achieves significantly lower accuracy loss than NN Slimming by 1 11; For the same level of accuracy loss, our method achieves higher parameter reduction (72.58% vs 61.87%) and higher FLOPs reduction (78.97% vs 66.21%) 1%) than EigenDamage.
ABLATION STUDY
The effect of attention mapping functions. Table shows the Top-1 accuracy loss of the VGG-16 pruned by one-shot pruning with different types of attention mapping functions on CIFAR-10. The parameter reduction is 57.73% and the accuracy of the original model is 93.63% (5 trails). First, analyze three forms of activation-basse attention maps with p equal to 1 (discussed in Section 3.T..
noted as Attention Mean (p- 1), Attention Sum (p 1), Attention Max leads to the lowest accuracy loss, lower than Attention Sum and Attention Max by 0.25% and 0.13%, respectively. Second, we analyze Attention Mean with difference values of p (p 1,2, 4). When p is set to 1, it leads to the lowest accuracy loss (0.38% vs 0.42% and 0.47%). Also, all the forms of Attention Mean outperforms LI-Norm, which validates that the proposed activation- baaedd attention mapping function is more efficient than weight magnitude based methods to evaluate the importance of filters.
1). Attention Mean
The effect of adaptive pruning. We compare the proposed adaptive pruning strategy with non- adaptive iterative pruning strategies that prune fixed percentage of filters in each pruning round. One baseline prunes fixed percentage of filters with lowest weight values, noted as Iterative L1- Norm Pruning (ILP); the other prunes a fxied percentage of filters using proposed attention mapping function with equal to 1, noted as Iterative Attention Pruning (IAP). Table compares the Top-1 accuracy loss of the VGG-16 model with sparsity of 70% pruned by IAP, ILP, and the proposed adaptive pruning strategy
10. For IAP and ILP, the pruning rate is set to 5%, that is, 5% of filters are pruned in each pruning round. With 70% reduced parameters, the proposed adaptive pruning strategy leads to higher accuracy than IAP and ILP by 1.19% and 1.39%, respectively.
CONCLUSIONS
This paper proposes an adaptive, activation based, iterative structured pruning approach to automat- ically and efficiently generate small, accurate, and hardware-efficient models that can meet diverse user requirements. We show that activation- attention value is a more precise indicator to identify unimportant filters to prune than the commonly used weight magnitude value. We also of- fer an effective way to perform structured pruning in an iterative pruning process, leveraging the LTH theory to find small and accurate sub-networks that are at the same time hardware efficient. Finally, we argue that automatic pruning is essential for this method to be useful in practice, and proposes adaptive pruning method that can automatically meet user-specified objectives in terms of model accuracy, size, and inference speed but without user intervention. Our results confirm that the proposed method outperforms existing structured pruning approaches with large margin.
9

---

Under review as a conference paper at ICLR 2022
REFERENCES
Anubhav Ashok, Nicholas Rhinchart, Fares Beainy, and Kris M Kitani. N2n learning: Net- work to network compression via policy gradient reinforcement learning. arXiv preprint arXiv:/709.0600) 2017.
H Cai, T Chen, W Zhang, Y Yu, and J Wang. Reinforcement learning for architecture search by network transformation. corr 1S//1707.0477 (2017).
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint 1033vv1803..3635, 2018.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Stabilizing the lottery ticket hypothesis. arXiv preprint arXiv: (9Xiv:9903.011111 2019.
Shangqian Gao, Feihu Huang, Jian Pei, and Heng Huang. Discrete model compression with resource constraint for deep neural networks, In Proceedings of the IEEE/CVT Conference on Computer Vision and Puttern Recognition, pp. 1899-1908, 2020.
Jinyang Guo, Wanli Ouyang, and Dong Xu. Multi-dimensional pruning: unified framework for model compression. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1508-1517, 2020.
Yi Guo, Huan Yuan, Jianchao Tan, Zhangyang Wang, Sen Yang, and Ji Liu. Gdp: Stabilized neural network pruning via gates with differentiable polarization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5239-5250, 2021.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning. trained quantization and huffman coding. arXiv preprint arXiv: 151.000199 2015.
Yang He, Guoliang Kang, Xuanyi Dong. Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. arXiv preprint arXiv: 1808. 06866, 2018a.
Yang He, Ping Liu, Ziwei Wang. Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4340 2019.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. In Proceedings of the IEEE international conference computer vision, pp. 1389--197,, 2017.
Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. Ame: Automl for model compression and acceleration on mobile devices. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 784-800, 2018b.
P. Hill, A. Jain, M. Hill, B. Zamirai, C. Hsu, M. A. Laurenzano, S. Mahlke, L. Tang, and J. Mars. Deftnn: Addressing bottlenecks for dnn execution on gpus via synapse vector elimination and near-compute data fission. In 2017 50th Annual IEEE/ACM International Symposium on Mi- croarchütecture (MICRO), pp. 786-799, 2017.
Zehao Huang and Naiyan Wang. Data-driven sparse structure selection for deep neural networks. In Proceedings of the European conference on computer vision (ECCV). pp. 304-320, 2018
Minsoo Kang and Bohyung Han. Operation-aware soft channel pruning using differentiable masks. In International Conference on Machine Learning, pp. 5122-5121111 5131. PMLR, 2020.
Yann LeCun, John S Denker, and Sara Solla. Optimal brain damage. In Advances in neural information processing systems, pp. 598-605, 1990.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based on connection sensitivity. arXiv preprint arXiv: 1xXiv:1800.033444 1810.02340, 2018.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. ICLR, 2016.
10

---

Under review as a conference paper at ICLR 2022
Yawei Li, Shuhang Gu, Christoph Mayer, Luc Van Gool, and Radu Timofte. Group sparsity: The hinge between filter pruning and decomposition for network compression. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 8018-8027, 2020.
Lucas Liebenwein, Cenk Baykal, Harry Lang, Dan Feldman, and Danicla Rus. Provable filter 2019.
pruning for efficient neural networks. arXiv preprint urXiv:1911.0PP
Minghao Lin, Rongrong Ji, Yan Wang, Yichen Zhang, Baochang Zhang, Yonghong Tian, and Ling Shao. Hrank: Filter pruning using high-rank feature map. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1529-1538, 2020.
Shaohui Lin, Rongrong Ji. Chenqian Yan, Baochang Zhang, Liujuan Cao, Qixiang Ye, Feiyue Huang, and David Doermann. Towards optimal structured enn pruning via generative adver- sarial learning. Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pp.
2019.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang. Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. Proceedings of the IEEE international conference computer vision, pp. 2736-2744, 2017.
Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural networks. In International Conference on Machine Learning, pp. 1498-2507. PMLR, 2017.
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06444 2016.
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. arXiv preprini urXiv:2003. 02389, 2020.
Yehui Tang, Yunhe Wang, Yixing Xu, Dacheng Tao, Chunjing Xu, Chao Xu, and Chang Xu. Scop: Scientific control for reliable neural network pruning. arXiv preprint arXiv:2010. 2020.
Lucas Theis, Iryna Korshunova, Alykhan Tejani, and Ferene Huszár. Faster gaze prediction with dense networks and fisher pruning. arXiv preprint arXiv: XX1v:1801.00787.. 007.8000.7777.11.....1 2018.
Chaoqi Wang, Roger Grosse, Sanja Fidler, and Guodong Zhang. Eigendamage: Structured pruning in the kronecker-factored eigenbasis. International Conference on Muchine Leurning, pp- 6566-6575. PMLR, 2019.
Yulong Wang, Xiaolu Zhang, Lingxi Xie, Jun Zhou, Hang Su, Bo Zhang, and Xiaolin Hu. Pruning from scratch. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp-
12273-12280, 2020.
H Yang, W Wen, and H Deephoyer Li. Learning sparser neural network with differentiable scale- invariant sparsity measures. arXiv preprint arXiv: /908. 0099908889777 2019.
Zhonghui You, Kun Yan, Jinmian Ye, Meng Ma. and Ping Wang. Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks. arXiv preprint 1xX1v:1909.08174. 2019.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention attention: Improving the perfor- mance of convolutional neural networks via attention transfer. arXiv preprint arXiv: 1XXi:1612.00928, 2016.
Chenglong Zhao, Bingbing Ni, Jian Zhang, Qiwei Zhao, Wenjun Zhang, and Qi Tian. Variational convolutional neural network pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2780-2789, 2019.
Tao Zhuang, Zhixuan Zhang, Yuheng Huang, Xiaoyi Zeng, Kai Shuang, and Xiang Li. Neuron-level structured pruning using polarization regularizer. Advances in Neural Information Processing Systems, 33, 2020.
Zhuangwei Zhuang, Mingkui Tan, Bohan Zhuang, Jing Liu, Yong Guo, Qingyao Wu, Junzhou Huang, and Jinhui Zhu. Discrimination-wwan channel pruning for deep neural networks. arXiv preprint arXiv: 1810.1 1809, 2018
11

---

Under review as a conference paper at ICLR 2022
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8697-8710, 2018.
A
APPENDIX
A.1 IMPLEMENTATION DETAILS
We implemented our proposed method on PyTorch version 1.6.0. For ResNet models on CIFAR- 10, the learning rate is set to 0.1 initially and decays with rate of 0.1 at the epoch 91 and 136. Weight decay is set to 0.0002. For ResNet-50 on ImageNet, the learning rate increases to 0.256 in warmup mechanism during the first epochs, and decays with a factor of 0.1 at epochs 30, 60, 80 (Renda et al. 2020); Frankle et al. ). Nesterov SGD optimizer is used with a momentum of 0.9 for all models. The simple data augmentation (random crop and random horizontal flip) is used for all training images. Learning rate rewinding is used for iterative pruning, which rewinds to the epoch roughly 60% of the total training duration (see Appendix |A.5 for an analysis of the effect of rewinding epochs the accuracy of the pruned models). For adaptive pruning, I and À are initialized to 0.0 and 0.005, respectively.
A.2 THE EFFECT OF ATTENTIONS
Figure shows the attention of each filter of the first convolution layer of ResNet-50 on ImageNet with different values of p (p=1, 2, 4). The setting where is equal to tends to be best, since iit promotes the effectiveness the pruning by enabling the gap between the mean values of the useful filters and useless filters to be large.
(a) p=l
(b) p=2
(c) p=4
Figure 2: Attentions of each filter of the first convolution layer of ResNet-50 on ImageNet with different values of p (p = 1, 4).
A.3 MENORY-CONSTRAINNDD ADAPTIVE PRUNING
The Memory-constrainedd Adaptive Pruning Algorithm is shown in Algorithm
A.4
FLOPS-CONSTRAINED ADAPTIVE PRUNING
The FIOOssconstrained Adaptive Pruning Algorithm is shown in Algorithm
A.5 THE EFFECT OF REWINDING EPOCH
In order to understand how the rewinding impacts accuracy of the pruned models, we analyze sta- bility to pruming as proposed by Frankle et al.) (2019), which is defined as the distance between the masked weights of the pruned network and the original network at the end of training. We val- idate the observations proposed by Frankle et al. (2019) that for deep networks, rewinding to very early stages is sub-optimal very late training stages is also sub-optimal because there is not enough time to retrain. Specifically, Figure ((a) shows the Top-1 1 test accuracy of the pruned ResNet-50 with 83.74% remaining param- eters when learning rate is rewound to different epochs, and Figure 3(b) shows the stability values the corresponding rewinding epochs. We observe that there region, 65 80 epochs, where
the network has not learned considerably by then; and rewinding to
12

---

Under review as a conference paper at ICLR 2022
Algorithm 4 Memory-consttrined Adaptive Pruning
1: Input: Target Parameters Reduction Paraml arget
2: Output: small pruned model with acceptable model size
Initialize: 7 0.0, 0.005.
4: for pruning round r (r >> 1) do
5: Prune the model using [r] (Refer to Algorithm
6: Train the pruned model, calculate its remaining number of parameters Param[[]
Calculate the parameters reduction: ParamRed[r]]: ParamRedd[]] Param[ ( Puram[r] Param Red(r] < Param7 arget then if the changes of model size are within 0.1 % for several rounds then Terminate
9:
10:
else
A[r
1] Xr] =
13:
+[
A[r 1]
14:
end if
15:
else
Find the last acceptable round
17:
if has been used to roll back for several times then
18:
Mark & as unacceptable
19:
Go Step 15
20:
else
21:
Roll back model weights to round
22:
A[r 1| Mr]/2.0(N+ (N is the number of times for rolling back round k)
23:
T 1] = [[k +
24:
end
25:
end if
26:
end for
Algorithm 5 FLOPs-constrained Adaptive Pruning
I: Input: Target FLOPs Reduction FLOPsTarget
Output: small pruned model with an acceptable FLOPs Initialize: 0.0, 0.005.
4: for pruning round r (r 3/ 1) do
5: Prune the model using [[r] (Refer to Algorithm 3)
6: Train the pruned model, calculate its remaining FLOPs FLOPs(r]
Calculate the parameters reduction: FLOP Redrr: FLOPsHed[r] = - Red]r] < FLOPs1 arget then if the changes of FT.OPs are within 0.1% for several rounds then
8:
9:
10:
Terminate
else
Air 1] Tír 1]
X T A[r + 1]
13:
14:
end if
15:
else
Find the last acceptable round
17:
if has been used to roll back for several times then
18:
Mark &s as unacceptable
19:
Go to Step 15
20:
else
21:
Roll back model weights to round
22:
X[r 1] / (N is the number of times for rolling back round k)
23:
T [ Aʃr
end
25:
end if
26:
end for
13

---

Under review as a conference paper at ICLR 2022
the resulting accuracy is high. We find that L2 distance closely follows this pattern, showing large distance for early training epochs and small distance for later training epochs. Our findings show that rewinding to 75%-90% of training time leads to good accuracy.
75
20.0
74
17.5
Accuracy
15.0
73
J Distance
72
10.0
7.5
5.0
2.5
0.0
25
55
65
80
88
25
55
65
85
88
Rewinding Epoch
Rewinding Epoch
Top-1 Test Accuracy
(b) Stability Pruming
The effect of the rewinding epoch (x-axis) (a) Top-1 test accuracy, and (b) pruning stability, for pruned ResNet-50, when 83.74% of parameters are remaining.
Figure 3:
A.6 THE EFFECT OF ITERATIVE PRUNING
To illustrate the effectiveness of our proposed adaptive pruning. in Figure 4 we show how the accuracy loss and parameter reduction change over the pruning rounds as the pruning threshold adapts according in the experiment pruning ResNet-56 on CIFAR-10, given the target of 1% accuracy loss. From Round to Round 24, the accuracy loss the pruned model is lower
Acc. Loss
Algorithm 2,
15 20
25
35
Param. Reduction
Round
than the target accuracy loss (1%), so the algorithm in- creases the pruning aggressiveness gradually by increas- ing the threshold. At Round 25, the accuracy loss exceeds the target, so the algorithm rolls back the model weights and the pruning threshold Round 24, and restart the pruning from there more conservatively. The above pro- cess repeats until after Round 39, the model size con- verges, and the algorithm terminates at reducing totally 88.23% parameters.
40
0
5 10
15 20 Round
25 30 35
Figure 4: Illustration of the changes the accuracy loss and parameter re- duction over the pruning rounds as the pruning threshold adapts when pruning ResNet-56 with CIFAR-10, given the target of 1% Top- accuracy loss.
14


Reviews:
[{"review_id": "cLdBWCliQcN", "reviewer": "Reviewer_XNxE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength: \n+ The proposed method is simple and easy to follow.\n+ This paper proposes three modes for structured pruning, including accuracy-guaranteed adaptive pruning, memory-constrained adaptive pruning, and FLOPs-constrained adaptive pruning. \n\nWeakness:\n- The contribution of the evaluation for the importance of filter, since activation-based attention maps are similar to attention score in [Sergey Zagoruyko, ICLR 16] and the activation-based score is also proposed in many related works, such as APoZ [1] and HRank. Please clarify the difference between these works and the proposed methods.\n- The rationale of calculating the threshold T in each layer. T in layer-aware threshold adjustment is directly related to the percentage of each layer on parameters or FLOPs. Moreover, the typo in line 3 of algorithm N^i[r]->T^i[r].\n- Lack of some important experimental evaluation: 1. The models selected to compress are all based resnet architectures, how about other architectures, especially light backbones (e.g. mobilenets); 2. The effect of attention score and pruning strategy should be discussed in ablation study; 3. The actual speedup of the pruned models should be evaluated if minimizing FLOPs.\n\n[1] Network trimming: A data-driven neuron pruning approach towards efficient deep architectures, arXiv preprint arXiv:1607.03250, 2016.\n", "summary_of_the_paper": "This paper proposes iterative structured pruning methods using activation-based attention feature maps and an adaptive threshold selection strategy. Inspired by attention transfer, Activation-based attention feature maps are constructed as the important evaluation of filters in each layer. Adaptive threshold selection strategy decides the number of removed filters, which satisfies one of three adaptive pruning policies. Experimental results on CIFAR-10 and ImageNet with ResNet architectures show performance gains over several state-of-the-art methods.", "summary_of_the_review": "Although the proposed method is simple yet effective on ResNets, the contribution is limited and experimental evaluation is not comprehensive. I tend to reject this paper in this version.\n\n\n-------------------POST-REBUTTAL COMMENTS-------------------\n\nI thank the authors for the response and the efforts in the updated draft. After this rebuttal, the authors well answer my comments about the comparison to SOTA methods. However, I still believe that the proposed measurement of filter importance is a limited novelty, as it is a simple revision of attention score from [Sergey Zagoruyko, ICLR 16]. Moreover, the rationale of threshold T is based on the assumption that a layer is more likely to have redundant filters to prune if it contains more remaining parameters. I think it is not a correct assumption, as the entire filters of a layer with a smaller number of parameters may be redundant to be removed safely, especially for ResNets. Thus, I still keep my original rate to reject it.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "747N5kDrz7", "reviewer": "Reviewer_YdeV", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I did not like the way that this paper is written. It is extremely waffley, and is not upfront about what is going on. It is not completely clear (to me at least) what the overall strategy for pruning will be in this work until half way through page 4 (!). The paper needs to be (completely) rewritten to match the expected format in the computer science literature.\n\n**Strengths**\n- I am not a total expert on the pruning literature, but I do find the idea of pruning based on the activation maps rather than the weights intuitive and sensible.\n- The results are compelling relative to other works they compare to. NOTE: again, I will not claim to be an absolute expert on the state of the pruning literature.\n- The automated approach is relatively simple, and far more interpretable than approaches such as AMC. This matters a lot in many real-world applications.\n- ImageNet results are included, which is relatively rare for pruning papers.\n\n**Weaknesses**:\n- The writing of this paper is unacceptable, as I mentioned earlier.\n- A few really important ablation studies are missing. Firstly, there is no direct reason to couple the pruning technique, with the method for iteratively pruning; can these not be compared separately. It is misleading to conflate the two. Secondly, there is no comment on runtime for this method, which is important if this is an automated technique.\n- Another important issue is that the FLOPs reduction doesn't seem that impressive. Works like Eigendamage (Wang et al.) achieve far more impressive reductions in FLOPs, which is arguably far more important than parameters in the real world. As an aside, would you be able to compare to Eigendamage?\n- The work only focuses on ResNets. I know that they're harder to prune, but I'd also like to see some experiments on VGG models, for example.", "summary_of_the_paper": "This work proposes a technique for iterative structured pruning, without necessarily requiring too much manual human intervention. There are two parts to this paper that are important:\n\n1. It is argued that we should prune channels based on the activation maps generated, rather than focusing on the weights of the channel.\n2. They propose an iterative procedure that automatically backtracks if it has made a poor pruning decision.", "summary_of_the_review": "I think this work has some merit, but it is not ready for publication at this time. I can be swayed at rebuttal time if there is convincing new experimental evidence.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0edlJDXef-_", "reviewer": "Reviewer_EiLE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThe proposed method has relatively good results on cifar10 and imagenet.  The method can automatically meet users' several requirements generally, like accuracy, latency, memory, and so on. \n\nWeakness:\n1. Each part of the technique is not that novel. It is like a combination of several existing tricks to perform comprehensive pruning, not enough analysis of the convergence or correctness. For example, how to guarantee the convergence of algorithm 3? \n2. The comparisons on Imagenet are too weak. There are many SOTA pruning methods on Imagenet. The paper only lists four of them. I suggest comparing the paper with more recent SOTA pruning papers and comparing on more benchmark models besides Res50. \nfor example, DMCP: Differentiable Markov Channel Pruning for Neural Networks\neagleeye: fast sub-net evaluation for efficient neural network pruning. \ngdp: neural network pruning via gates with differentiable polarization.\nand so on. \n\n", "summary_of_the_paper": "This paper proposed an activation-based, adaptive threshold, iterative structured pruning method, combining several existing techniques together to perform a comprehensive pruning. ", "summary_of_the_review": "Due to the concerns on the novelty and the weak comparisons on Imagenet. I temporarily think this paper is marginally below the acceptance threshold. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a strategy for incrementally pruning deep learning models based on activation values. The approach can satisfy different kinds of requirements, trading off between accuracy and sparsity.

The approach seems promising and seems to have competitive performance. However, the method is described by reviewers as a combination of ideas that have been proposed in the literature, and the experimental evaluation relies too much on a dataset considered too small to be reliable in such experiments --- CIFAR10. We do not expect substantial experiments within the rebuttal period: such comparisons with relevant SOTA methods should have been present in the submission. Moreover, the strategy proposed for selecting a threshold seems to rely on some doubtful assumptions, and there are no benchmarks on actual runtime.

The writing has improved based on reviewer input, and the reviewers are satisfied with this aspect. I would still add that I would prefer some clarity in the method presentation: is there a quantity being optimized? is there a value we can monitor to ensure our reimplementation is correct? etc. In addition I would like to ask authors in the next revision to be mindful to the difference between `\citet` and `\citep` in author-year citations -- see e.g. the first two ones in 3.1.
Decision: Reject