Pruning by Active Attention Manipulation
Authors: Zahra Babaiee
Lucas Liebenwein
Ramin Hasani
Daniela Rus
Radu Grosu
Venue: Submitted to ICLR 2023 (2023)

Abstract: Structured pruning of a CNN is typically achieved by applying discrete masks on the CNN's filter weights or activation maps, post-training. Here, we present a new filter-importance-scoring concept named pruning by active attention manipulation (PAAM), that sparsifies the CNN's set of filters through a particular attention mechanism, during-training. PAAM learns continuous filter scores from the filter weights by optimizing a cost function regularized by an additive term in the scores. As the filters are not independent, we use attention to dynamically learn their correlations. Moreover, by training the pruning scores of all layers simultaneously, PAAM can account for layer inter-dependencies, which is essential to finding a performant sparse sub-network. PAAM can also train and generate a pruned network from scratch in a straightforward, one-stage training process without requiring a pre-trained network. Finally, PAAM does not need layer-specific hyperparameters and pre-defined layer budgets, since it can implicitly determine the appropriate number of filters in each layer. Our experimental results on different network architectures suggest that PAAM outperforms state-of-the-art structured-pruning methods (SOTA). On CIFAR-10 dataset, without requiring a pre-trained baseline network, we obtain 1.02% and 1.19% accuracy gain and 52.3% and 54% parameters reduction, on ResNet56 and ResNet110, respectively. Similarly, on the ImageNet dataset, PAAM achieves 1.06% accuracy gain while pruning 51.1% of the parameters on ResNet50. For Cifar-10, this is better than the SOTA with a margin of 9.5% and 6.6%, respectively, and on ImageNet with a margin of 11%.

Under review as a conference paper at ICLR 2023
PRUNING BY ACTIVE ATTENTION MANIPULATION
Anonymous authors
Paper under double-blind review
ABSTRACT
Filter pruning of a CNN is typically achieved by applying discrete masks on the CNN's filter weights activation maps, post-training. Here, we present new cillc-inpotannce-ccring concept named pruning by active attention manipulation (PAAM), that sparsifies the CNN's set of filters through particular attention mechanism, during-training. PAAM learns analog filter scores from the filter weights by optimizing a cost function regularized by an additive term in the scores. As the filters are not independent, we use attention to dynamically learn their correlations. Moreover, by training the pruning scores of layers simultaneously, PAAM can account for layer inter-dependenccis, which is essential to finding a performant sparse sub-network. PAAM can also train and generate a pruned network from scratch in straightforward. one-stage training process without requiring a pre-trained network. Finally, PAAM does not need layer-specific hyperparameters and pre-defined layer budgets, since it can implicitly determine the appropriate number of filters in each layer. Our experimental results on different network architectures suggest that PAAM outperforms state-of-the-ar structured- pruning methods (SOTA). On CIFAR-10 dataset, without requiring a pre-trained baseline network, we obtain 1.02% and 1. 1.19% accuracy gain and 52.3% and 54% parameters reduction, on ResNet56 and ResNetl 10, respectively. Similarly, on the ImageNet dataset, PAAM achieves 1.06% accuracy gain while pruning 51.1 1% of the parameters on ResNet50. For Cifar- 10, this is better than the SOTA with a margin of 9.5% and 6.6%, respectively, and on ImageNet with a margin 11 %.
INTRODUCTION
Pruning by Active Atlention M Manipulation (P APM))
Convolutional Neural Networks (CNNs) LeCun (1989) are used nowadays in wide variety of computer- tasks. Large CNNs in par- ticular, achieve considerable performance levels, but with significant computation, memory, and energy footprints. respectively Sui al. (2021). As a consequence, they cannot be effectively em- ployed in resource-limited environments such as mobile or embedded devices. is therefore essential create smaller models, that can per- form well without significantly sacrificing their accuracy and performance. This goal can be accomplished by either designing smaller, but performant, network architectures Lechner et al. (2020): Tan & Le (0199) or by first training an over- parameterized network, and sparsifying thereafter, by pruning its redundant parameters Han et al. (2016); Lieben wein et al. (2020) 2021). Neural-network pruning is defined as systematically removing parameters from an existing neural network [oeffer et al. (202T). D. It is popular technique to reduce growing energy and performance costs and to support deployment in resource-constrained environments such as smart devices. Various pruning approaches have been developed, and this has gained considerable attention over the past few years Zhu & Gupta (2017); Sui et al. (202T) Liebenwein et al. (202)):
loss
spoth
Warm-up on dense network Train spanse network's wrights
Train PAAM
sparse network
Figure 1:
Sensitivity-based filter pruning schedule.
Peste et al. (2021): Frantar et al.] (2021): Deng et al. (2020).
Pruning methods are categorized into either unstructured structured. The first, remove individual only (Hn et al. (2016). The second, remove entire groups. by pruning neurons, filters, channels, respectively Anwar al. (201)) Li al. 2019); He al. (2018b) Liebenwein

---

Under review as a conference paper at ICLR 2023
PAAM
Scores
Linear Map
Linear Map
Linear Map
Scaled Dot
Product Attention
Leaky Expo
Leaky Expo
H
CxXxX
Feature
Pruned
Filters
Maps
Feature Maps
Vanilla PAAM
PAAM
Figure 2: PAAM learns the importance scores of the filters from the filter weights.
al. (2020). As modern hardware tuned towards dense computations, structured pruning offers a more favorable balance between accuracy and performance Hoefler et al. (202T). A very prominent family of structured- prruningggnngggggg methods is filter pruning. Choosing which filters remove according to a carefully chosen importance metric (or filter score) is an essential part of any method in this family.
Data-fres free methods solely rely on the value of weights, or the network structure, in order to determine the importance of filters. Magnitude pruning, for example, is one of the simplest and most common of such methods. prunes the filters that have the smallest weight-values in the /1 norm. Data-informed methods focus on the feature maps generated from the training data (or subset of samples) rather approaches (which consider the
than the filters alone. These methods vary from sensitivity- statistical sensitivity of the output feature maps with regard to the input data MMalik & Naumann (2020) Liebenwein et al. () to correlation- methods (with an inter-channel perspective, to keep the least similar (or least correlated) feature maps Sun et al. (2015); Sui et al. (202T))
Both data-free and data-informed methods generally determine the importance of a filter in a given layer, locally. However, filter-importance is global property, as it changes relative to the selection of the filters in the previous and next layers. Moreover, determining the optimal filter-budget for each layer (a vital element of any pruning method), is also a challenge, that all local, importance- methods face. The most trivial way to overcome these challenges, is to evaluate the network loss with and without each combination of & candidate filters out of N. However, this approach would require the evaluation of (>) subnetworks, which is in practice impossible achieve.
Training-aware pruning methods aim to learn binary masks for turning on and off each filter, A regularization metric often accompanies them, with a penalty guiding the masks to the desired budget. Mask learning, simultaneously for all filters, is an effective method for identifying globally optimal subset of the network. However, due to the discrete nature of the filters and binary masks, the optimization problem is generally non-convex and NP-hard. A simple trick of many recent works Gao et al. 2020 2021); ; Li et al. (202)) is to use straight-through estimators [Bengio et al.) (2013) to calculate the derivatives, by considering binary functions as identities. While ingenious, this precludes learning the relative importance of filters among each other. Even more importantly, the on-off bits within the masks are assumed to be independent, which is gross oversimplification.
This paper solves the above problems by introducing PAAM, a novel end-to-end pruning method, by active attention manipulation. PAAM also employs /1 regularization technique, encouraging filter- score decrease. However, PAAM scores are analog, and multiply the activation maps during score training. Moreover, a proper score spread is ensured through a specialized activation function. This allows PAAM to learn the relative importance of filters globally, through gradient descent. Moreover, the scores are not considered independent, and their hidden correlations are learned in a scalable fashion, by employing an attention mechanism specifically tuned for filter scores. Given global pruning budget, PAAM finds the optimal pruning threshold from the cumulative histogram of filter scores, and keeps only the filters within the budget. This relieves PAAM from having to determine per layer allocation budgets in advance. PAAM then retrains the network without considering the scores. This process is repeated until convergence. PAAM pipeline is shown in Figures 2 Our experimental results show that PAAM yields higher pruning ratios while preserving higher accuracy.
In summary, this work has the following contributions:
1. We introduce PAAM, an end-to-end algorithm that learns the importance scores directly from the network weights and filters. Our method allows extracting hidden correlations in
2

---

Under review as a conference paper at ICLR 2023
the filter weights for training the scores, rather than relying only on the weight magnitudes. The feature maps are multiplied by our learned scores during training. This way our method implicitly accounts for the data samples through loss propagation, enabling PAAM to enjoy the advantages of both data-free and data-informed methods.
2. PAAM automatically calculates global importance scores for all filters and determines layer-specific budgets with only one global pppee-prramtter..
3. We empirically investigate the pruning performance of PAAM in various pruning tasks and compare advanced state-of-the- pruning algorithms. Our method proves to be competitive and yields higher pruning ratios while preserving higher accuracy.
2 PRUNING BY ACTIVE ATTENTION MANIPULATION ALGORITHM
In this section, we first introduce our notation and then incrementally describe our pruning algorithm.
2.1 Notation, The filter weights of layer are given by the tuple Fi E FRFxX where F is the number of filters, C' the number of input channels, and K the size of the convolutional kernel. The feature maps of layer are given by the tuple A IR R x where H and W are the image height and width respectively. For simplicity, we ignore the batch dimension in our formulas.
2.2 Score Learning. The score-learnigg function of PAAM, for layer of the CNN to be pruned, can be intuitively understood as a single-layer independent network, whose inputs are the filter weights Fi of layer I, and the outputs are the scores S associated to the filters. The network first transforms the input weights Ft to a score vector Fi W whose length equals the number F of filters of layer 1, and then passes the result through an activation function é, properly spreading the scores within the [0, 1 + c] interval. The resulting scores are then used by an /1 regularization term of the cost function. The choice á and regularization term is discussed in the next sections. Formally:
S = (FFFF.
(1)
where @ is the activation function and W E | is the weight matrix. This fransfor-
mation, through W and & (vanilla PAM), is similar to additive attention Bahdanau et al. 20144). We will therefore refer in the following to the score-lcarning network as the attention network (AN). Intuitively, the AN weight matrix - captures the hidden correlations among filters. Unfortunately, for layers with large number of filters, as for ImageNet, this matrix will become too large train.
To capture the correlations in a scalable fashion, we would like to first partition the input in chunks, compute the correlations within each chunk, and then compute the correlations among chunks. But this is exactly what a scaled dot- dot-product attention achieves Vaswani et al. (2177)). First, PAAM reformats the input as binary matrix Fi E (Fx xx)xx) x)) where the chunks are its rows, Second, it obtains the correlations within each chunk as the queries Qi= Fi WQQ and keys K₁- = F.WKK Here, are much smaller, as they consider
the query and key weight matrices WQ, W E !Cxxxxx x)) only a chunk, and dj F is the hidden dimension of the layer. The queries and keys are of shape Q1, K₁ E ((d)) Third, PAAM obtains the cross correlations among chunks by multiplying the query matrix Qi with the transpose K. of the key matrix. Finally, the scores are obtained by averaging and normalizing the result, and passing it through the activation function c. Formally:
mean(Qi X KF) /
S d( =
(2)
where a is a scaling factor that we tune. Note that PAAM does not need to learn the value-weight matrix, compute the values, and multiply them with the above result, as it is only interested in the correlation among filters. Learning filter weights is left to CNN training. However, the scores Si learned by PAAM, are then pointwise (0) multiplied by the feature maps A of the same layer:
1 = S C A
The closer a filter score is to 1, the more the corresponding feature map is preserved. Note also that by using an analog value for scores while training the AN, allows PAAM to compare the relative importance of scores later on. In particular, this is useful when PAAM employs the globally allocated budget, and the cumulative score distribution, to select the filters be used during CNN training.
3

---

Under review as a conference paper at ICLR 2023
2.3 Activation Function. SoftMax is the typical choice of activation function in additive attention when computing importance Vaswani et al. (2017b). However, SoftMax is not a suitable choice for filter scores. While the range of its outputs is between 0 and 1, the sum of its outputs has to be 1, meaning that either all scores will have a small value, or there will be only one score with value 1.
In contrast to SoftMax, More formally, the scores should have the following three main attributes: 1) All filter scores should have a positive value that ranges between 0 and 1, as is the case in SoftMax. 2) All filter scores should adapt from their initial value of 1, as we start with a completely dense model. 3) The filter-scores activation function should have non-zero gradients over their entire input domain.
would intuitively want that many filter scores possibly close 1.
Sigmoidal activations satisfy Attributes 1 and 3. However, they have difficulties with Attribute 2. For high temperatures, sigmoids behave like steps, and scores quickly converge to 0 or 1. The chance these scores change again is very low, as the gradient is close to zero at this point. Conversely, for low temperatures, scores have a hard time converging to 0 or 1. Finding the optimal temperature needs an extensive search for each of the layers separately. Finally, starting from a dense network with all scores set to 1 is not feasible.
mm00
1-2x
0.8
0.2
To satisfy Attributes 1-3, designed our activation function, as shown in Figure 3 First, in order to ensure that scores are positive, we an exponential activation function and leamn its logarithm value. Second, we allow the activation to be leaky, by not saturating it at 1, as this would result in 0 gradients, and scores getting stuck 1. Formally, our leaky-exponential activation function is defined as follows, where a is a small value (an ablation study in provided in Appendix):
Figure 3: The leaky-exponen- tial activation function.
E 11.0 ar
if 0
(xz)
if 20 0
2.4 Optimization Problem. The PAAM optimization problem can be formulated in Equation where L is the CNN loss function, and f is the CNN function with inputs x, labels y. and parameters W. modulated by the AN with inputs the filter weights, parameters V. and outputs the scores S. Let (g((S) p)ll denote the 4--orm of the binarized filter scores of layer I, F1 the number of filters of layer the number of layers, and p the pruning ratio. Then PAAM should preserve only as many filters as desired by p, while minimizing at the same time the loss function.
min L(y, (,; S))
S.L.
Σ (l((),))) - Σ R 0 1=0
(5)
1=0
Function g(S..p) p) is discussed in next section. The budget constraint is addressed by adding an /1 regularization term to the loss function, while training the weights of the AN. This term employs the analog scores of the filters in each layer, which are computed as described above. Formally:
((()) Cly, f(z; W, + A
Σ 1=
|
(6)
where À is global hyper-parameter controlling the regularizer's effect the total loss. Since the loss function consists now of both classification accuracy and the I nn-orm cost of the scores, reducing filter scores decrease I cost, directly influences accuracy as scores multiply the activation maps.
In more detail, by incorporating the classification and the nn-mr of the scores in the loss function, the effect of the score of each filter is accounted for in the loss value in two ways:
1. When multiplied by small scores, feature maps have less influence on classification. This may result larger loss if these maps are important and vice versa.
2. On the other hand, the part of the loss function consisting of the 4n-norm of the scores, decreases the loss value when there are more scores with small values.
In some sense, this duality mimics the way synapses are pruned or strengthened through habituation or conditioning in nature. If the use of a synapse does not lead to a reward it is pruned, otherwise, is strengthened. Moreover, pruning is essential in order avoid saturation and enable learning.
4

---

Under review as a conference paper at ICLR 2023
The value of À plays very important role in keeping the balance between the two factors, and in controlling the pruning ratio. When is O, all scores will stay close to 1, and when it is large, all scores will converge 0. During training of the AN layers, we freeze all CNN parameters. The scores are directly learned from the filter weights, reflecting attributes such as the filter-weights magnitude and the correlation among the different filters of the same layer. A globally pruned network is obtained by simultancously training the scores of all layers. This allows propagating the scored influence of the feature maps of one layer to the feature maps and scores of the next layers.
2.5 Training schedule. PAAM starts training the CNN by a number of warm-up epochs. During this phase, trains the dense CNN with all filter scores fixed to and the AN weights frozen. After the warm-up up phase, PAAM starts training both the AN weights and the CNN weights.
As discussed, PAAM multiplies the CNN fea- ture maps by the AN analog scores. This way, when score is 1, the corresponding feature map is fully preserved. As the score gets smaller, its feature map is getting weaker, since it is multiplied by value smaller than I, and fi- nally almost completely pruned, as the score gets closer to O. However, while the scores are getting trained, the network weights can learn to adapt the feature-map intensity. This means that, although filter may have a low score, the loss of the feature map intensity can be compen- sated by magnifying the feature map itself. In order to prevent this, PAAM uses an alternating training approach Peste et al. (2021).
Algorithm 1 PAAM
Inputs: Mini-batches B; CNN N after warm- up. parametrized by W: AN parametrized by V: Regularization hyper-parameter X; PAAM- training epochs Ep; Training mini-epochs E..
Output: Pruned CNN for fine-tuning.
for i in Ep do for j in E, do for b in B do miny Lly, f(:; W,S) +>>))>>>+))> ||//||
calculate gradients w.r.t
Update by the optimizer
end for end for for in Eg do for in do miny Lly, fæ; S)))
First, PAAM trains the AN for a few epochs, while freezing all CNN parameters. Then it trains the sparse CNN for a few epochs, while freezing all AN parameters. Sparsity is obtained by multiplying feature maps Ar with a binary version ( p) of scores learned far. Func- tion g is described in the next section. AN train- ing and CNN training alternate for predefined number of training cycles. After the training phase is completed, PAAM finishes the pruning procedure, by fine-tuning the sparse model. Note that PAAM does not need a pretrained dense model to find the importance of filters or feature maps. Even when starting from scratch, after warm-up phase, can find optimal sparse sub-network by training the AN and CNN weights, respectively, as discussed above.
calculate gradients w.r.t W Update W by optimizer end for end for end for Return N with parameters W
2.6 Binarization function. As mentioned above, during CNN training. PAAM fixes the scores to for all filters retained according to their score magnitude and pruning ratio. All other filters have score O. The malog-to-diiing transformation accomplished by the function g(s, p), as follows:
g(s.p) - -(0 if 2 ((Sp))
otherwise.
where s is the maalog-score value of filter, and ((S,p) the optimal pruning threshold that is computed, by considering the cumulative distribution of all scores s S, and the pruning ratio p. We will discuss how PAAM has computed 0(S,p) p) for Cifar10 and ImageNet, in Section 3
2.7 Complete Algorithm Algorithm I describes PAAM's intermediate stage. While training the AN parameters, it uses Equations While training the CNN parameters, uses Equation
EXPERIMENTAL EVALUATION
In this section, we first present implementation details and then discuss experimental results. Baselines. We compare PAAM to numerous standard and advanced pruning methods. The models include: L.1 norm Li al. (017)) neuron-importance score propagation (NISP) Yu et al. (2017), soft
5

---

Under review as a conference paper at ICLR 2023
Table Results the Cifar10 dataset with ResNet56 and ResNetl 10 for medium (top part) and for large (bottom part) pruning ratios, respectively. The quantity A shows the difference in the accuracy (Aec) between the baseline dense model used, and the resulting pruned network. Note that PAAM does not use a pretrained baseline. Numbers are taken from the reported results of the cited papers.
ResNet56
Methud
Buseline Acc(%)
Pruned Acc(%)
3(%)
[Params(5 (%)
[ioops
Narm (2017 Li al. 2017
93.04
93.06
+0.02
13.7
27.6
NISP (2017) Ya e: mi GUL
NIA
N/A
-0.03
42.2
35.5
(2018)
Zhianp of mi 2118
93. 80
93.59
-0.31
NIA
50.0
DCP-Adapt 12018 Zhiming
2018
93. 80
93.81
+0.01
N/A
47.0
CCP
47.0
GAL (2019) et 2019
93.28
93.38
+0.12
11.8
37.6
IRank (2020)
2020
93.26
93.52
+0.26
16.8
29.3
DMC (20220
40.07
50.0
NPPM (202]11a8
al
1024
93.04
93:40
+0.36
NIA
50.0
CHIP (2021 Sur
93.26
40.90
47.4
PAAM(Vanila) PAAM(KO)
94.28
+102
52.3
49.3
94.01
+0.75
$4.0
$2.3
(2018) He
00
50.0
GAL (2019) (an als
2019
93:26
91.58
-1.68
65.9
60.2
HRank (2020)
2020
93.26
90.72
-2.54
68.1
74.1
CHIP (2021 PAAM(aandI PAAMIK))
93.26
72.3
92.42
-0.84
83.0
70.5
93,10
-0.16
70.0
74.7
ResNetl 10
21 Norm (2017) ( al NISP (2018)
2017
93.53
93.30
-0.23
32.4
38.7
NIA
SFP (2018) Heaera
KDr
93 68
93.86
+0.18
NIA
40.8
GAL (2019)
2015
93.55 $3.50
93.59
+0.09
18.7
4.1
HRank (2020)
2020
+0.73
41.2
CHIP (2021 (St al
2121
93.50
94.44
+0.91
48.3
$2.1
PAAM(Vanilla) PAAM(KQ)
94.69
+1.19
$4,9
$1.3
+1.13
40.9
GAL (2019)) et HRank (2020)
2019
93.50
92.74
-0.76
44.8
48.5
93
68.6
CHIP (2021) Sur PAAMiVandla PAAM(K))
93.50
93.63
+0.13
68.3
71.3
93.68
+0.18
79.3
74.8
77.2
filter pruning (SFP) He et al. (2018a), discrimination-awanl channel pruning (DCP) Zhuang al.
(2018), DCP-adapt Zhuang et al. (2018), collaborative channel pruning (CCP) Peng et al. (2019). generative adversarial learning (GAL) Lin al. (2019), filter-pruning using high-rank feature maps (HRank) Lin et al. (2020). discrete model compression (DMC) Gao et al. (2020), network pruning performance maximization (NPPM) Gao et al. (2021), channel independence-bent based pruning (CHIP) (Sui et al. (202I), and auto-ML for model compression (AMC) He et al. (2018d).
3.1 IMPLEMENTATION DETAILS
Training Procedure. Our first experiments are on CIFAR-10 with two different models: ResNet56 and ResNerl 10. For both models, PAAM does not use fully trained network the baseline to prune. We train each model for 50 warm-up epochs. During warm-up (see Figure [ we use a batch size of 256 and stochastic gradient descent (SGD) as optimizer, with 0.1 as the initial learning rate, 0.9 as momentum, and 0,0005 for the weight decay. We then use PAAM to train the scores.
After warm-up, PAAM trains the AN and the CNN in alternation, with the other-network parameters frozen. PAAM trains the AN weights for 3 epochs, and uses the regularized loss defined in Equa- tion (1O). Then PAAM switches to training the CNN weights for 6 epochs, and uses the classification loss. In this phase, PAAM employs Equation equation 7 as the analog-to-digee di-iggaal score-conversion function. When PAAM trains the AN, the CNN feature maps get multiplied by the analog scores. PAAM repeats these two phases for 10 times, summing up to training for 90 epochs. PAAM uses the ADAM optimizer with learning rates of 10 6 10-3 for training the AN/ CNN parameters.
After training, PAAM fine-tunes the CNN. In this stage, it removes the AN, and keeps only the CNN filters with the binary score one. PAAM uses Equation equation] 7 to compute the binary scores from their analog counterparts, which is in fact PAAM's ultimate goal in pruning. Each feature map either removed entirely (having score zero), or is completely preserved (having score one). PAAM uses the SGD optimizer, with the same parameters in warm-up, and tunes the CNN for 300 epochs.

---

Under review as a conference paper at ICLR 2023
Table 2: Experimental results ImageNet with ResNet50. A is the difference in accuracy between baseline and pruned networks, The numbers are taken from the reported results in the cited papers.
Topl Acc(%)
Top5 Acc(%)
(0%
Method
Baselime
Pruned
Baseline
Pruned
A
Params
Flops
SEP (2018) et al. 80
76.15
74.61
-1.54
92.87
92.06
-0.81
N/A
41,8
DCP 20018/2humn
2018
76.01
74.95
-1.06
92.93
92.32
-0.61
50.9
55.6
FPGM (2019) He el al 201
76.15
75.59
-0.56
92.87
92.63
-0.24
37.5
42.2
CCP (2019) Peng
76.15
75.5
-0.65
92.62
-0.25
N/A
48.8
GAL (2019
2019
76.15
71.95
-4.2
92.87
90.94
-1,93
16,9
43.0
PFP (2020) Jichenwein el 2020
76.13
75.21
-0,92
92.87
92.43
0.44
30.1
44,0
AutoPruner (2020]]L00
76.15
74.76
-1.39
92.87
92.15
-0.72
NIA
48.7
HRank (2020)
et al
2020
76.15
74.98
-1.17
92.87
92.33
-0.54
36,6
43.7
(202)) Tans a1.
2020
76.15
75.95
-0.20
92.87
92.79
-0.08
42,8
45.3
DMC (2020)
76.15
75.35
-0.80
92.87
92.49
-0.38
N/A
55.0
NPPM (2021
Gao et 2021
76.15
75.96
-0.19
92.87
92.75
-0.12
NIA
56,0
CHIP (2021) Sui ct all
2021
76.15
76.30
+0.15
92.87
92.91
40.04
40.8
44,8
PAAM(KQ)
76.15
77.21
+1.06
92.87
93.55
+0.68
51.1
31,9
Our second experiments are ImageNet with ResNet50. As ResNet50 has many layers with very large number of filters, PAAM (Vanilla) runs out of memory, whereas PAAM (KQ) has no problems to scale up. In this experiment, PAAM starts with the pretrained ResNet50 model, in order save time. PAAM alternates the training of the AN/CNN networks for 54 epochs, with 3 epochs for the AN and epochs for the CNN in each iteration. PAAM then fine tunes the pruned CNN for 200 epochs. PAAM sets the batch size, momentum, weight decay and initial learning rate to 128, 0.875, 2e-05 and 0.5, respectively, and uses a aesine-annealing learning-rat scheduler with 5 as warm-up.
Balancing the Pruned Parameters and Flops. PAAM uses while training the AN, the regularized loss to guide the scores to the desired filter budget. However, the number of parameters of each CNN layer is different from the number of computation flops required for that CNN layer:
(8) As the number of flops also depends on the image size, early convolutional layers, before the max- pooling layers, require more flops the image size is larger. To properly balance pruning flops, we multiply the 4 norm of each layer by the fraction of the image sizes of that layer and the last layer.
(Params)) Ci F Ki Ki, (Flops)) C. x Fi K2 KI x W H1
For experiments on ResNet56, PAAM used À 10 and À 15 10-4 4 for pruning ratios of 52.3% and 83.0% respectively. On ResNet 10, PAAM used X 2 x 10-4 and X A= 5.5 x 10-4 to prune 54.9% and 79.3% of the network parameters. On ResNet50, PAAM used = 10-6.
3.2 PERFORMANCE EVALUATION
Experimental results. Table compares the performance of PAAM (Vanilla) and PAAM (KQ) the sate-of-the-art filter-pruning methods (SOA) on Cifar10. Both versions of PAAM outperform SOA, achieving higher accuracy while pruning more parameters. Specifically, in an accuracy increase, even compared to the dense baselines, while sigmificantly pruning parameters and flops. The same true for ResNetl 10. For high pruning ratios. PAAM outperforms CHIP Sui et al. (202T). the next best method, with better accuracy while pruning more parameters and flops on ResNet56. Similarly, on ResNetl 10, PAAM outperforms CHIP in both accuracy and pruning.
ResNet56 PAAM results
Table the performance of PAAM (KQ) to the SOA on ImagNet. Starting from baseline of 76. accuracy, PAAM achieves a higher accuracy of 77.21% even when compared to the baseline (an increase of 1.06%), while pruning 51.1% of the parameters and 31.9% of the flops.
Per-layer Budget Discovery. A remarkable feature of PAAM it that it finds the optimal sparse sub-networks in a fully-automated pipeline and does not require budget-allocction schedule per layer. shows the discovered networks in our experiments from ResNet56 and ResNet 10.
In each residual block of the sub-networks learned by PAAM, the first layer has lower number of filters remaining after pruning, compared to the second layer. This structure similar to the bottle-neck architecture used in ResNets with large number of layers. It enables the network to concentrate on the most important features with less capacity, which is exactly what we are looking for with pruning. Many existing pruning algorithms similar bottle-neck structures, when manually
7

---

Under review as a conference paper at ICLR 2023
Number of fiters
original fileer
Number of fiters
original filter pruning
filters after pruning
Layers
Layers
Nurber of filors
original filters
Number filter
original filters filters pruning
after pruning
Leyers
Layers
Figure 4: CNNs discovered by PAAM from ResNet56 (a) and ResNetl 10 (b), with medium (l) and high (II) pruning ratios. PAAM finds the optimal per-layer budgets automatically.
defining layer budgets for pruning Sui et al. pattern automatically, without supervision. Moreover, for high pruning ratios on ResNetl 10, there are blocks emerging with 0 remaining filters in the first layers. This shows that PAAM can also remove entire layers when required, having more freedom in the possible sparse sub-networks search space.
(202T) : Lin et al. (020).. PAAM is able to discover this
The threshold value. PAAM computes the op- timal threshold 0(S, p) of Equation (7). based on the pruning ratio p. and the cumulative dis- tribution of filter scores, as shown in Figure for the Cifar10 experiments on ResNet56 and ResNetl 10. This resulted in ((S. = 0.5.
Table 3: Ablation results on Cifar10 dataset with ResNet56 pruned with different threshold values.
Thresbold(0)
1[%ooolo
Accuracy())
0.2
16.0
36.7
94.30
0.3
50.2
42.4
94.32
0.4
51.6
46.6
94.29
52.3
49.3
As one see, the leaky-exponential activation function and the regularized- PAAM, ensure that the majority of the scores are pushed values close zero and one. Hence, most of the scores lie very close to the two ends of the [0, 1] interval, and 0.5 is in its center. Moreover, as it is shown in Figure the cumulativeddistiibution plot of scores is almost horizontal before 0.5.
0.6
58.2
57.6
93.97
0,7
64.7
65.9
93.66
functions of
In order to provide more insight on the proper choice for the value of the threshold, we conducted an ablation experiment on Cifarl0 with the ResNet56 network. After the scores training epochs, we pruned the filters with different thresholds. This process is then followed by fine-tuning epochs. Table shows the accuracies of the pruned network with each threshold value. As the table shows, the pruned parameters and flops percentages change slowly with changing the score threshold. As one can see, a threshold 0 of 0.5 ensures best accuracy for most pruning of parameters and flops.
RELATED WORK
Attention, recent years, attention has achieved great success in many computer-vision tasks Guo
et al. (2022), from attentional CNNs to transformers Dosovitskiy et al. (2020). These works mainly focus on improving the accuracy of vision networks by learning to what (channel) or where (spatial) to pay attention to. The squeeze-and-excitatio network Hu et al. (2017) is one of the pioneers in channel- attention networks which calculate channel scores from feature maps to enhance classification.
PAAM. To the best of our knowledge, this paper is the first to employ attention, tailored to computing filter-importance scores from filter weights, for pruning CNNs. PAAM (KQ) only uses the key and the query matrices to compute the filter correlations within layer, and disregards the value matrices.
Pruning. Filter pruning is a very popular structured -pruninn- method for sparsifying CNNs, which supports storage reduction and processing efficiency, without requiring any special library. One can roughly classify existing filter-pruning methods into three main categories, based on their filter- selection approaches: data-free, data- intta-informd,, and training- aware methods, respectively.
Data-free filter pruning. Following up on the weights- magnitude- maagnitud-pruning method, where the weights with the smallest absolute values considered the least important, [L et al. (2017) uses the sum of
8

---

Under review as a conference paper at ICLR 2023
(a)
(b)
[c]
Scores
ResNet110 Scores
Cumulative ResNet56
Number Filters
Number Filters
Number of Filters
m013
19
Score
Score
Score
Figure 5: Filter-score histograms for Cifarl0 experiments. Most filters have zero and one scores.
absolute values of the weights in a filter (the /1 norm) to prune the filters with the smallest weight values. He et al. (2018a) dynamically prunes the filters with the smallest /2 norm in each epoch by setting them to zero and repeating this process in each epoch during training. He et al. (2019) uses the geometric median of filters as the pruning criterion. In summary, although data-free methods can gain acceptable performance levels, several works have later shown that considering the training data during the pruning process, will notably improve pruning precision Hoefler et al. (2021).
Data-informed filter praning. Many pruning methods focus feature maps they provide rich information from both data distribution and filters. [in et al. (2020) prunes filters whose feature sensitivity measure to prune filters with lowest effect on the outputs, giving provable sparsity guaranties. Motivated by the importance of inter-channel perspective for pruning. (Sui al. (2021) the nuclear norm of feature maps independence metric to prune the filters whose feature maps are the most dependent on the others.
maps have lowest ranks, and Liebenwein et al. (2020)
Training-on aware filter pruming. These methods training to learn filter-importane metric guide the network to a sparse structure. Exploiting magnitude pruning, some add regularization factors to the loss, guide filters values towards zero. |Wen al. (2016) and Louizos et al. (2018) use group-lasso and 10 regularization, respectively. Instead of solefy relying on the weight magnitudes, [Zhuang et al. (2018) proposes discrimination-awe aware channel channel-pruning method, by defining per-layer classification losses. Gao et al. (2020) trains binary gate functions with straight-through estimators and Gao et al. (2021) tocuses on training binary gates, by directly maximizing the accuracy of subnetworks.
PAAM. To the best of our knowledge, none of the above methods, is able to automatically learn the filter-importance scores from the filter weights, extract hidden correlations among filters, automati- cally calculate global importance scores for all filters, and determine layer-specific budgets, all at the same time during training, and thus taking advantage of both data-free and data-informed methods.
CONCLUSION
We proposed a novel, end-to-end, attention- based, filter-pruning algorithm called PAAM (pruning by active attention manipulation). PAAM learns the filter-importance scores, through gradient descent the CNN equipped with a filter-attention network (AN), and a classification loss-function regularized with the /1 norm of the filter scores. The AN computes the filter scores from the filter weights and automatically finds the correlations among filters within each layer. The /1 regularization, encourages the decrease of scores and finds the filter correlations across layers. In contrast to large spectrum of advanced pruning algorithms, PAAM does not necessarily require a pretrained baseline CNN to prune from. can rather sparsify dense networks from scratch, through cycles gradient descent.
We showed on comprehensive set of experiments on Cifar10 and ImageNet, that much better compression rates are achievable through the use PAAM for ResNet50, ResNet56, and ResNetl 10, while even surpassing baseline accuracy. PAAM is able to compute the global flter-importance scores and to automatically associate pruning budgets to CNN layers, without layer-specific hyperparameters. In this way, PAAM is taking advantage of both the data-free and the data-informed pruning methods.
is our hope that future work is going to start using PAAM in resource- -hnuurc applications domains, Mearch such those of neural-arde wawhall-chhhhttreee search Mellor et al. (202T). The application of PAAM promises to also result compressed neural networks endowed with salient features, such as accuracy and small compute footprint, automatically distilled during training for resource-constrained environments.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Sajid Anwar, Kyuyeon Hwang, and Wonyong Sung. Structured pruning of deep convolutional
neural networks. J. Emerg. Technol. Comput. Syst., 13(3), feb 2017. ISSN 1550-4832. doi:
10.1145/300534, URLAttts://aoi..... wrt..o..............o........... 10. 10..114/3005340
Dzmitry Bahdanau. Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate, 2014. httphttts://axiii.oo
absp /ttas//1440 1409. 0473)
Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR, abs/1308.330 2013. URL
httt:://axxv.oog/ absa 3432
Lei Deng, Guoqi Li, Song Han, Luping Shi, and Yuan Xie. Model compression and hardware acceleration for neural networks: A comprehensive survey. Proceedings of the IEEE, 108(4): 485-532, 2020.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x words: Transformers for image recognition at scale, 2020. URL https: /awpvv.ivv...////////a/////aaav. orq abs/ /bt///bb////////// 2010.11929
Elias Frantar, Eldar Kurtic, and Dan Alistarh. Efficient matrix-fres approximations of second-orde information, with applications to pruning and optimization. arXiv preprint 11X1::2007003356 2021.
Shanggian Gao, Feihu Huang, Jian Pei, and Heng Huang. Discrete model compression with resource constraint for deep neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Shangqian Gao, Feihu Huang, Weidong Cai, and Heng Huang. Network pruning via performance maximization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2270-9280, June 2021.
Klaus Greff, Rupesh K Srivastava, and Jürgen Schmidhuber. Highway and residual networks learn unrolled iterative estimation. arXiv preprint arXiv:1612.07771. 2016.
Meng-Hat Hao Guo, Tian- -ing Xu, Hlang-Jiang Liu, Zheng-Ning Liu, Peng-Tap Tao Jiang, Tai- Tai-Jiang Mu, Song-Hai Zhang, Ralph R. Martin, Ming Ming-Ming Cheng, and Shi-Min Hu. Attention mechanisms in computer vision: survey. Computcational Visual Media, 8(3):331-36,, mar 2022. doi: 10. .rgg /0/. 007 8 &2F441 001111007459900022222 092-
$41095-022-0271-y Uttptttps:/
Song Han, Huizi Mao, and William J. with pruning, trained quantization and huffman coding, 2016.
Dally. Deep compression: Compressing deep neural networks
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference computer vision, pp. 630-645. Springer, 2016.
Yang He, Guoliang Kang. Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. In Proceedings of the Twenty-Sevens International Joint Conference on Artificial Intelligence, IJCAI-18, pp. 2234-2240. International Joint Conferences on Artificial Intelligence Organization, 7 2018a. doi: 10.24963/ijcai.201 2018/309 URL https
7/doi.org/10
www.oo/.........1.1... 24933/ 2018/309
Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks, 2018b.
Yang He. Ping Liu, Ziwei Wang, and Yi Yang. Pruning filter via geometric median for deep convolutional neural networks acceleration. CoRR, abs/1811.00250, 2018c. URL /ttp:://
faxxiv.org. abs
Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
10

---

Under review as a conference paper at ICLR 2023
Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang. Li-Jia Li, and Song Han. Ame: Automl for model compression and acceleration on mobile devices. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018d.
Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training neural networks, 2021.
Jie Hu, Li Shen, Samuel Albamic, Gang Sun, and Enhua Wu. geueeze--and-xciin networks, 2017. 01507]
URL
Stanistaw Jastrzebski, Devansh Arpit. Nicolas Ballas, Vikas Verma, Tong Che, and Yoshua Bengio. Residual connections encourage iterative inference. In International Conference on Learning Representations (ICLR), 2018.
Mathias Lechner, Ramin Hasani, Alexander Amini, Thomas A Henzinger, Daniela Rus, and Radu Grosu. Neural circuit policies enabling auditable autonomy. Nature Machine Intelligence, 2(10): 642-652, 2020.
Y. LeCun, B. Boser, J.S. Denker, D. Henderson, R.E. Howard, W. Hubbard, and L.D. Jackel. Backpropagation applied to handwritten zip code recognition. Neural Computaaiion 1(4):511-55, 1989.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets, 2017.
Yawei Li. Shuhang Gu, Luc Van Gool, and Radu Timofte. Learning filter basis for convolutional neural network compression, 2019.
Yunqiang Li, Silvia L. Pintea, and Jan C. van Gemert. Equal bits: Enforcing equally distributed binary network weights. Proceedings of the AAAI Conference on Artificial Intelligence, abs/21 12.03466, 2022. URL
orq/ abs/2112.0332
Lucas Liebenwein, Cenk Baykal, Harry Lang, Dan Feldman, and Daniela Rus. Provable filter pruning for efficient neural networks, 2020.
Lucas Liebenwein, Ramin Hasani, Alexander Amini, and Daniela Rus. Sparse flows: Pruning continuous-depth models. Advances in Neural Information Processing Systems, 34, 2021.
Mingbao Lin, Rongrong Ji, Yan Wang. Yichen Zhang, Baochang Zhang, Yonghong Tian, and Ling Shao. Hrank: Filter pruning using high-rank feature map. In Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Shaohui Lin, Rongrong Ji, Chenqian Yan, Baochang Zhang. Liujuan Cao, Qixiang Ye, Feiyue Huang, and David Doermann. Towards optimal structured cnn pruning via generative adversarial learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),
June 2019.
Christos Louizos, Max Welling, and Diederik P. Kingma. Learning sparse neural networks through 10 regularization. In International Conference on Learning Representations, 2018. URL https
net. 7/ ooterrewioo..ne//ooooomaaaaaaa11
Jian-Hao Luo and Jianxin Wu. Autopruner: An end-to-end trainable filter pruning method for efficient deep model inference. Pattern Recognition, 107: 2020. ISSN 0031-3203. doi: http:://oi.org//00 101.:...0..10.........0.... 2024..0741..... URL httpp:://ww...iirreeetee http///www.inneedreetttct.cc com/
Sher Afghan Malik and Uwe Naumann. Interval Adjoint Significance Analysis for Neural Net- works. In Computational Science - ICCS 2020 20th International Conference, Amsterdam, The Netherlands, June 3-5, 2020, Proceedings. Part III / edited by Valeria V. Krzhizhanovskaya, Gábor Závodszky, Michael H. Lees, Jack J. Dongarra, Peter M. A. Sloot, Sérgio Brissos, João Teixeira, volume 12139 of Lecture notes in computer science, pp. 365-378, Cham, Jun 2020. 20th International Conference on Computational Science, Amsterdam (Netherlands), 3 Jun 2020 5 Jun 2020, Springer International Publishing. doi: 100000007783-000--555 wwth-aachent de wwe./eeeooeeo wecord//1677. Weitere Reihe: Theo- retical Computer Science and General Issues ; 12139. Springer eBook Collection.
27. URL
11

---

Under review as a conference paper at ICLR 2023
Joe Mellor, Jack Turner, Amos Storkey, and Elliot J Crowley. Neural architecture search without training. In International Conference on Machine Learning, pp. 7588-7598. PMLR, 2021.
Hanyu Peng, Jiaxiang Wu, Shifeng Chen, and Junzhou Huang. Collaborative channel pruning for deep networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of Machine Learning. volume 97 of Proceedings of Machine Learning Research, pp. 5122-1122. PMLR, 09-15 Jun 2019. URL https: httptttss///occcddnnggg
the 36th International Conference
mlr. milr.preas/997/peng 9c. html
Alexandra Peste, Eugenia Iofinova, Adrian Vladu, and Dan Alistarh. Ac/de: Alternating com- presecd/decoopressedd training of deep neural networks. Advances in Neural Information Process- ing Systems, 34, 2021.
Yang Sui, Miao Yin, Yi Xie, Huy Phan, Saman Zonouz, and Bo Yuan. Chip: Channel independence- based pruning for compact neural networks, 2021.
Yi Sun, Xiaogang Wang, and Xiaoou Tang. Sparsifying neural network connections for face recognition, 2015.
Mingxing Tan and Quoe Le. EfficientNet: Rethinking model scaling for convolutional neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 6105-6114 PMLR, 09-15 Jun 2019. URL https::/proceeddnggs mlr
press/v97/tan1 hhmmml
Ychui Tang, Yunhe Wang. Yixing Xu, Dacheng Tao, Chunjing XU, Chao Xu, and Chang Xu. Scop: Scientific control for reliable neural network pruning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 10946-10947 Curran Associates, Inc., 2020. URL httpttts::/pprcceeeeeeee http://pptceeee.cccoccopoo neurips cc/ pwtppp/ppp//iieeeeeppp
1ea.aa/2000002201111000110011110 fbbbb Paper.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, & ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gamett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017a. URL 0itee22434477eee110003e3e3e
www.eieeuerpp...ee. wwwerips.ccppaaeeee cctpp////////////////e//// Paper. pdf
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017b.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured spar- sity in deep neural networks. In D. Lee, M. Sugiyama, U. Luxburg, Guyon, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 29, Curran Asso- ciates, Inc., 2016. URL htttpa:/ www.ris....pp.... httpsnnnsssccc..pppe/o..... 6/file/ 41bfd20a3 8bb wtt.bbbbbbaaaaaaaaa1333000111110
Ruichi Yu, Ang Li, Chun-Fu Chen, Jui-Hsin Lai, Vlad Morariu, Xintong Han, Mingfei Gao, Ching- Yung Lin, and Larry S. Davis. NISP: pruning networks using neuron importance score propagation. CoRR, abs/1711.05903 2017. URL
05908
Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression, 2017.
Zhuangwei Zhuang, Mingkui Tan, Bohan Zhuang, Jing Liu, Yong Guo, Qingyao Wu, Junzhou Huang, and Jinhui Zhu. Discrimination-waare channel pruning for deep neural networks. In S Bengio, H Wallach, H Larochelle, K Grauman, N Cesa- Aasa-Bianchi, and R Garnett (eds.), NIPS Proceedings, volume 8-eecembee of Advances in Neural Information Processing Systems, pp. 875-886. Neural Information Processing Systems (NIPS), 2018 Advances in Neural Information Processing Systems 2018, NIPS 2018: Conference date: 02-12-2018 Through 08- 12-2018.
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 PAAM'S S EFFECT ON ITERATIVE INFERENCE IN RESNETS
In this section, we proceed with an analysis of the effect that PAAM has on the iterative feature- refinement process in residual networks Greff et al. (2016).
ResNets He al. 2016) are known to enhance representation learning in deeper layers via an iterative feature- refimment scheme Greff et al. (2016). This scheme suggests that input features to ResNet do not create new representations. Rather, they gradually and iteratively refine the learned features of the initial residual blocks Jastrzebski et al. (2018). Iterative refinement of features was shown to be necessary for obtaining attractive levels of performance, while their disruption hurts performance.
As AN modifies the underlying model structure and the feature maps of residual block, it is very important to investigate if PAAM preserves the iterative feature-refinement property of ResNets.
To make this analysis precise, let us first formalize iterative inference as discussed in Jastrzebski et al. (2018) A residual block in a ResNet with M blocks, performs for the input feature x; the following transformation: X + fx()). ). Hence, the following loss function L can be recursively applied to the network
(aattreeaastzebskkk Jastrzebski et al. (2018):
[(xM) [(XM-1 fM-1(XM-1)).
A first-order Taylor expansion of this loss, while ensuring that ff's magnitude is small, is a good approximation to formally investigating the iterative inference @aastzzeskii al. (2018). Thus:
= [(x,)=((xx) [(x;)+ EOO)) al(x,) Xj
((f((x)))
(10)
This approximation reveals that the i-th residual block, modifies features x; with roughly the same amount of (xx,, as that of @C(x.) This implies a moderate reduction of loss as we transition from xi the i-th to the MMth block, as an iterative refinement scheme Greff al. (2016); Dastrzebski al.
(2018). The refinement step for vanilla residual block can be computed by the squared norm of fx::), and can be normalized to the input feature as: [x. ||x:122.
Any modification to the structure of the residual network (c.g., in PAAM) causes a change in the refinement step. This step has to be investigated if it does or it does not modify the iterative inference.
Lemma A.1. The iterative with parameter e from the leaky integrator and < <1+c:
scale is bounded for ResNets with PAAM as follows,
(xx(x)) ||x:12
||S, の f (xx)) ||S, 12
+ e [[
S
(11)
1+c
5
|
Proof. A ResNet block i that is equipped with an SbF-Pruner layer, transforms the features with the following expression: x,+1 = S, & (x. + fx(x)) )), where 5; stands for the score vector computed by PAAM. The refinement step is given by ||S, @ fxx,))1 and its input-normalized representation is ||S @ x,
||S. fxx))
Deriving the upper bound: Assuming that every element in S, is between d and 1+c, for 0 < 0655++
we have:
||S o fxx)) || < || [xxx] < (1 + (ff((l)
(12)
||S C x, Z || |x||| 2 8 ||x:|
(13)
As consequence, the following -ouer- inequality holds for the iterative inference:
|S の f(xx)
< (x,
1+
(14)
|| x; 2
5
| 2
Deriving the lower bound: Assuming that every element in S, is between 5 and 1 + c, we have:
||S; f((x))l 2 (xx)
(15)
13

---

Under review as a conference paper at ICLR 2023
1.0
0.8
exp(x)
0.4
1+ax
2x
0.2
sigmoid(()
sigmoid((x
-3
2
Figure 6: Activation functions.
the anutig
30
of ROME
Figure 7: Network discovered by PAAM with Sigmoid activation function.
o (1 +) ||x||
(16)
As consequence, the following lower-bound inequality holds for the iterative inference:
SS f
8 |
(17)
S 1
1
|
Inequalities equation 14 and equation
17 prove the the stated lemma.
Lemma A. has profound implications in practice. It indicates that the cetaatvveinnfrenne property of
the ResNets equipped with PAAM is both lower and upper bounded. These ResNets not only get compressed in size, but also preserve the representation learning capabilities of ResNets between these two bounds. The bounds themselves can be fine tuned with the parameter À of Equation equation 6
A.2 ABLATION ON ACTIVATION FUNCTIONS
In this section, we provide additional experiments on the choice of the activation function for PAAM layers. We compare the performance of the vanilla PAAM on CIFARIO with ResNet56 with four different activation functions presented in Figure
We start with the Sigmoid function. As already mentioned in the paper, Sigmoid favourable function, satisfying the first and third conditions we look for in our activation function. In our experiment with Sigmoid, the scores did converge to values close to 0 and and mostly stayed at values close to 0.5, making it hard to control the pruning ratio by the loss regularization factor. We further tried the Sigmoid function with higher temperatures, o (bxr) with the scores converged to ( and 1. As discussed in the manuscript, high temperature helps the Sigmoid to converge. But the scores converge at a fast rate to or 1, and hardly change in future epochs, resulting in uniform random pruning budget allocation scenario. Figure 7 shows the network discovered by PAAM with high temperature Sigmoid activation function. Compare this to the layer specific pruming results shown in the manuscript in Figure 4(a). Table 4 shows that high temperature Sigmoid activation function results in poor accuracy for the pruned network. order to be able to Sigmoid activation function, extensive per-layer tuning for the temperature is required.
100. In this experiment,
14

---

Under review as a conference paper at ICLR 2023
To improve the results with Sigmoid, experimented with activation function similar to leaky-expo, which we call leaky- 2sigmod.. 2sigm-d is initialized by ones with output range in [0, 2] and has affine linear component for outputs greater than 1 as follows:
(2) 1.0
i <0 0
<(x)
(18)
if 2 0
As one in Figure 6 this function is very similar leaky-expo. The results shown in Table illustrate that PAAM with leaky-2sigmoid and leaky-expo perform competitively, with leaky-expo resulting in better compression rate. This indicates that starting the scores from 1, which is starting from a fully dense network, plays an important role in the results obtained by PAAM.
Table 4: Ablation study with different activations
Sedivation-Functiion Pruned Acc(%)
Parrameter((%)
Sigmoid(x)
not converged
not converged
Sigmoid(bx)
89.68
51.5
Leaky-22igmoid
94.10 10
48.2
Leaky-expo
94.28
52.3
A.3 EXPERIMENTAL DETAILS
In this section, provide the details of the experiments conducted on Cifar10 and ImageNet datasets. Table shoows the training epochs and À value for the scores loss regularizer for our experiments. In all experiments, during pruning epochs, train the PAAM layers for 3 epochs and then switch to training the network weights for epochs and repeat this set of 9 epochs.
Table 5: Hyperparameters used in Cifar 10 experiments.
Model
Warm-up
Pruning
Fine-tuning
PParams(%)
(Flops(%)
5e
52.3
49.3
ResNet56
50
90
300
1.5c 33
83.0
70.5
2e
54.9
51.3
ResNetl 10
50
90
300
5.5c 44
79.3
74.8
ResNer50
pre-trained
54
200
le
51.1
23.1
Cifar10. We used Vanilla and KQ PAAM for Cifar10 datasets. We didn't fully pre-trained networks, but rather did 50 warm-up epochs before pruning.
ImageNet. We used KQ-PAAM for ImageNet experiments. We the hidden dimension di of each layer to number of filters of that layer, devided by the layer expansion parameter, which is 4 in ResNet50. We normalised the scaling factor by 1/a, set to the sum-of-squares root the number of filters in the last layers of each block. We used the pre-trained pytorch model for ResNet50, For pruning and fine-tuning we augment the data by random-resized crop, random-horizontal flip, and trivial augmentation. We also normalized the data samples. For pruning epochs, use Adam, and set the learning rates to 10 and 10 for score training and network training phases, respectively.
15


Reviews:
[{"review_id": "ah3tNTNPkrR", "reviewer": "Reviewer_12a5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose a filter-importance-scoring concept named pruning by active attention manipulation. The proposed method is a one-stage training process training network from scratch without requiring a pre-trained network.", "summary_of_the_review": "There are many activation-aware pruning methods, whose idea is very similar to the proposed method. Thus, I am afraid that this work lacks novelty. The experiments are not solid to demonstrate the effectiveness and efficiency of the proposed method.", "strength_and_weaknesses": "Strength:\nThe authors proposed a one-stage training process training network from scratch without requiring a pre-trained network. The proposed method could achieve good performance.\n\nWeaknesses:\n1. There are many activation-aware pruning methods, whose idea is very similar to the proposed method. Thus, I am concerned about the lack of novelty in this paper.\n2. The proposed method cannot compare with the latest SOTA methods. In fact, the proposed method cannot achieve the best performance compared with some recent works.\n3. From the experimental results, one could see that the activation attention of the proposed method cannot gain much improvement.\n4. The ablation study is actually a study of different activations. It is better to add the experimental results of the variant without activation.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ZZaJSJYCzKm", "reviewer": "Reviewer_wYHx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to compute the importance of filters with attention weights. Although the authors utilize existing attention mechanisms (i.e., additive attention and scaled dot-product attention), they properly design a nonlinear activation that considers the range of values and gradients, an alternating training process of model parameters and attention weights, a histogram-based analysis to determine the pruning threshold. Experiments on CIFAR and ImageNet with ResNets show the superiority of the proposed method.", "summary_of_the_review": "In general, I believe this work did a good job in terms of applying attention mechanisms for structured pruning. Although there exists a concern regarding the reproducibility, I believe it can be resolved if the authors can release their codes. The experimental results and analyses are quite good and thorough.", "strength_and_weaknesses": "Strength\n- The idea that utilizes attention mechanisms to compute the importance scores of filters while considering the correlations between them is well-motivated.\n- Although the authors used the existing attention mechanisms of previous literature, they introduce several interesting elements to properly exploit them for structured pruning. \n- I think the experimental validation is quite good. The baselines are strong, and the results of the proposed method are either SOTA or competitive.\n- I appreciate the visualization of Figure 4 that shows the selected filters after pruning and the related discussions. \n- The paper is well backed-up by a comprehensive supplementary material that contains the in-depth discussion and ablation study on the proposed method as well as the details required for reproducibility. \n- The paper is generally quite thorough and generally feels complete: I believe it is generally ready for publication if it is decided by the reviewers that significance is sufficient to warrant publication.\n \nWeakness\n- The proposed method was validated with only ResNets. Is this method also effective for small-size or other networks, including MobileNets and VGG? It would be helpful to present additional results with other architectures.\n- I think this paper includes only the performance after finetuning, and it is difficult for me to identify whether good accuracy comes from the power of finetuning or the effectiveness of PAAM. It would be good to add the performance before finetuning (but after channel pruning) and compare it with at least one baseline method. \n- My major concern is that the proposed method requires numerous hyperparameters including the number of warm-up epochs, the number of PAAM-training epochs, the number of alternating training epochs, the regularization weighting, and different optimizer schedules for AN/CNN parameters. How did the authors set these values? Although the paper includes quite detailed numbers, it would be very helpful to provide some guidelines to determine hyperparameters.\n- I would highly recommend the authors to make their codes publicly available for the research community. Implementing the proposed method with attention networks seems not to be simple, and releasing the codes would be very helpful for reproduction and future studies.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DZ5OsMgQUC", "reviewer": "Reviewer_rXzi", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The work proposes a novel method for network pruning by co-training a network to predict filter importance given model weights. This allows the model learn hidden correlations between the model weights and channel importance. They achieve strong empirical results on pruning resnet networks on CIFAR-10 and Imagenet. The method naturally learns E2E and doesn't require per-layer sparsity values and is tuned by a global cost-weighting/function. To satisfy the requirements of their attention network, they propose a novel activation function.", "summary_of_the_review": "This work proposes a novel method for structured pruning which achieves significantly better SOTA results on Imagenet and Cifar10. Some aspects may of be limited novelty however, due to similarities to methods explored in other NAS and channel pruning methods. Since the empirical results are quite important, the work would benefit from more exploration of it's robustness and more detailed analysis of the method and training cost. I believe this work is currently marginally below the acceptance threshold, but would be willing to increase my score with additional  analysis, experimental results, and details of the hyperparameter search", "strength_and_weaknesses": "Strengths:\n- Achieves very strong possibly SOTA empirical results on CIFAR-10 and Imagenet.\n- Proposes novel method for channel pruning using an attention mechanism that allows accounting of layer interdependencies.\n\nWeaknesses:\n- Little to no discussion of the stability or robustness of the approach.\n- Possibly limited novelty due to the similarities to existing work in pruning and NAS and would benefit from significantly more analysis and ablation of their method and the cost.\n\nQuestions:\nThe paper would benefit from discussion of the cost of pruning compared to competing methods. Do other methods prune/fine-tune the network for as many epochs? How expensive is the score model?\nSince the training method is quite lengthy and approaching NAS, it would be quite useful to run the pruning results multiple times to give a better idea of how stable the method is.\nHow much hyperparameter tuning did you conduct and do you have results from different weightings of the flops/parameter cost? The work would benefit from more discussion of how controllable the sparsity is.\nIs a single score model trained used for every layer or are separate networks used per layer?\nI don't understand the section on the threshold value. Table 3 shows that you get a more accurate, and sparser network with a value of 0.2 or 0.3. Why was 0.5 chosen?\n\nNits:\nOn page 7: \"Table 2 compares the performance of PAAM (KQ) to the SOA on ImagNet\" is missing Ts\nQuestion, but a bit separate from the paper. Can the channel importance estimator transfer between models or during training or does it only give useful signal when co-trained with the network? Would be a useful area to explore if possible.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject