IDP: Iterative Differentiable Pruning based on  Attention for Deep Neural Networks
Authors: Minsik Cho
Saurabh Adya
Devang Naik
Venue: Submitted to ICLR 2023 (2023)
Keywords: pruning
deep learning
attention

Abstract: Deep Neural network (DNN) pruning is an effective method to reduce the size of a model, improve the inference latency, and minimize the power consumption on DNN accelerators, at the risk of decreasing model accuracy. In this paper, we propose a novel differentiable pruning scheme, Iterative Differentiable Pruning or IDP which offers state-of-the-art qualities in model size, accuracy, and training cost. IDP creates attention-based pruning masks for a given sparsity target to achieve the state-of-the-art trade-offs between model accuracy and inference compute with negligible training overhead. We evaluated IDP on various computer vision and natural language processing tasks, and found that IDP delivers the state-of-the-art results. For MobileNet-v1 (which is a challenging DNN for pruning), IDP can achieve 68.2% top-1 ImageNet1k accuracy with 86.6% sparsity which is 2.3% higher accuracy than the latest state-of-the-art pruning algorithms. For ResNet18, IDP offers 69.5% top-1 ImageNet1k accuracy with 85.5% sparsity at the same training budget and 0.8% better top-1 accuracy than the state-of-the-art method. Also, IDP demonstrates over 83.1% accuracy on Multi-Genre Natural Language Inference with 90% sparsity for BERT, while the next best from the existing techniques shows 81.5% accuracy.

Under review as a conference paper at ICLR 2023
IDP: ITERATIVE DIFFERENTIABLE PRUNING FOR NEU- RAL NETWORKS WITH PARAMETER-FREE ATTENTION
Anonymous authors Paper under double-bling review
ABSTRACT
Deep Neural Network (DNN) pruning is an effective method to reduce the size of model, improve the inference latency, and minimize the power consumption on DNN accelerators, at the risk of decreasing model accuracy. In this paper, we propose a novel differentiable pruning scheme, Iterative Differentiable Pruning or IDP which offers Mate-offthe-ane qualities in model size, accuracy, and training cost. IDP creates soft pruning masks based on fixed-point attention for a given sparsity target to achieve the state-of-the-art trade-offss between model accuracy and inference compute with negligible training overhead. We evaluated IDP on various computer vision and natural language processing tasks, and found that IDP delivers the state-of-the-ar results. For MobileNet-vl, 1, IDP can achieve 68.2% top-I I ImageNet k accuracy with 86.6% sparsity which is 2.3% higher accuracy than the latest state-of-the-ard pruning algorithms. For ResNet18, IDP offers 69.5% top-I ImageNet accuracy with 85.5% sparsity the same training cost which is 0.8% better than the state-of-the- method. Also, IDP demonstrates over 83. 33.1% accuracy on Multi-Genre Natural Language Inference with 90% sparsity for BERT, while the next best from the existing techniques shows 81.5% accuracy.
INTRODUCTION
While advanced deep neural networks (DNN) have exceeded human performance on many complex cognitive tasks (Silver et al.) 2018), their deployment onto mobile/edge devices, such as watches or glasses, for enhanced user experience (i.e., reduced latency and improved privacy) is still challenging. such on-device systems are battery-powered and are heavily resturce-cnnstrained, thus requiring
DNNs to have very high power/comppttessorage efficiency (Wang et al.) 2019 Wu et al. 2018
Howard et al. 2017 Vasu et al. 2022 Wang et al. 2020b).
Such efficiency can be accomplished by mixing and matching various techniques, such as designing efficient DNN architectures like MobileNe//Mobile MobileOne (Sandler et al. 20188] Mehta
Rastegari 2022; Vasu et al. 2022), distilling a complex DNN into smaller architecture (Polino et al.
2018),
the weights of DNNs (Cho et al. [022] Han et al. 2016 D. Lee 2021
Park & Yoo, 2020 Li et al. 2019. Zhao al. 2019), ,, and pruning near-zero weights (Peste al.
2usupausupati et al [2020] Liu et al. 2021]] Zhang et al. 2022 Sanh et al. 2020 /atrir et al. 2021
Zhu & Gupta 2018 Wortsman al. (2019). Also, pruning known be highly complementary to
(Wang et al. (020b) when optimizing a DNN model. Training a larger model and then compressing it by pruning has been shown be more effective in terms of model accuracy than training a smaller model (Li et al. (220)) from the beginning. However, pruning comes the cost of degraded model accuracy, and the trade-off is not straightforward (Kusupati et al. 2020).
ceantizaatio/comprrssttt
Hence, desirable pruning algorithm should achieve high accuracy and accelerate inference for various types of networks without significant training overheads in costs and complexity. In this work, we propose simple yet effective pruning technique, Iterative Differentiable Pruning or IDP based on an parameter-free attention mechanism (Bahdana et al. 2015 Xu et al. 2015) that satisfies all the above criteria. Our attention approach allows pruning mask to be differentiable, and lets training- loss decide whether/how each weight will be pruned. Therefore, such a loss-driven differentiable pruning mask will help capture the interactions among weights automatically without expensive mechanisms (Liu et al. 202I). D. Also, IDP requires neither additional learning parameters (Zhang et al. 2022) nor complicated training flows (Peste et al. 2021), yet offers precise control on the target

---

Under review as a conference paper at ICLR 2023
STR
GraNet"
Optor
ACDC
MVP
POPA
IDP
Accuracy on Conv-Nets Accuracy on Transformers
?
?
Inference speed (MAC)
?
Training speed
?
Training stability
VV
VTV
VVV
Training flow
simple
complex
complex
complex
complex
complex
simple
Extra training parameters few none
many
none
many
none
none
Soft threshold weight re- parameterization for learnable sparsity (Kusupati et al. 2020
Sparse training via boosting pruning plasticity with neuro-regeneration (Lau al. 2021
Optimizing gradient-driven criteria in network sparsity: Gradient all you need (Zhang et al
2022
4 AC/DC: alternating compressedddeooppessed training of deep ncural networks (Pette ct al.
2021
Movement Pruning: Adaptive Sparsity by Fine-Tuning (Sanh et al. 20200.
Prune Once for All: Sparse Pre-Traine Trained Language Models (Zalrir
2021
state-of offthheartt pruning schemes and IDP: IDP can explore a good trade-off bewteen accuracy and inference speed without introducing new Icarnable parameters and with a simple/fast training flow.
Table I: Comparison
sparsity level (Kusupati et al. 2020) and pushes the state-of-thee -at in pruning. Table compares IDP with the latest state-of-the -fe-th--rtttt art pruning schemes, and our major contributions include:
I
A differentiable and parameter-free pruning algorithm based on attention.
Efficiently pruning to offer a high-quality model for given pruning target.
The state-of-the-art results both computer vision and natural language tasks.
2 RELATED WORKS
Pruning in DNN incurs complex trade-off between model accuracy and inference speed in terms of MAC (mult-add operations) (Kusupati et al.] 2020). weight can contribute differently the model accuracy, depending on the number of times it used for prediction (i.e., weight in convolution filter for a large input) and the criticality of the layer it belongs to (i.e., a weight in bottleneck layer). Please even if two models are pruned to the same level, the accuracy and inference speed of each can be vastly different, which makes exploring the best trade-off challenging yet crucial in DNN pruning.
Trade-off's in Pruning:
Fig. 7 and Section B in Appendix for details. Therefore,
Unstructured schemes make individual and independent pruning decision for each weight maximize the flexibility and minimize the accuracy degradation. Simple and gradual/iterative pruning based on the weight magnitude has been studied extensively (Zhu & Gupta 2018: Gale al. 2019 Frankle Carbin 2019 Han et al.] 2015).
Unstructured Pruning:
these schemes, once a weight pruned, it does not have the second chance to become unpruned and improve the model quality. To address such challenges, RigL. (Evei et al.] 2020) proposes to grow sparse network by reallocating the removed weights based on their dense gradients. Applying brain-inspiee neurogeneration (i.c.. unpruning some weights based on gradients) and leveraging pruning plasticity is proposed (Liu et al. 2021). Altering the phase of dense and sparse training to accomplish co-training of sparse and dense models is studied, which results in good model accuracies on vision tasks (Peste et al.. 2021). Unlike other magnitude-driven pruning, supermask training (Zhou et al. 2019) integrated with gradient-driven sparsity is proposed in (Zhang et al. (002)), where accumulated gradients are used to generate binary masks and straight-through estimator (Bengio al. 2013) used for backward propagation. Motivated by the lottery hypothesis (Frankle & Carbin 2019), pruning in one-shot
based on heuristics (Tanaka et al., 2020) or gradient-driven metrics (Wang al. 2020a) is explored.
Structured Pruning:
Unstructured pruning limits inference latency speedup as it takes significant overhead to fetch irregular non-zero index matrices, suffers from poor memory access performance,
and does not fit well on parallel computation (Anwar et al.) 2017] Liu et al. (022). Therefore, recent research extends unstructured pruning by imposing particular sparsity pattern during pruning at the cost of lower model predictive power, but increases the hardware performance during inference.
One popular and effective form of structured pruning is channel pruning, where some channels with negligible effects the model accuracy discarded (He et al. 2017 [i al.) [017]] Kang & Han 2020). Using regularization to prune weights in a block is proposed in (Lagunas et al. 2021 N:M
pruning enforces that there are N zero weights out of every consecutive M weights (Zhou
2021)
2

---

Under review as a conference paper at ICLR 2023
0.18
0:19
0.19
0.20
0.21
0.22
0.25
0.30
0.15
0.19
0.39
0.20
0.21
0.22
Wl
W5
W
Wis
Wi
chreshold: 0.205
0..005
0.20
0:25
0.00
0.90
702
716
Weight Update
Weight Update
0.18
0.19
0.19
0.20
0.19
0.21
0.23
0.31
0.18
0.20
0.22
0.22
0.29
0.21
0.23
0.31
Mis
Mey
W
W9
Mig
We's
Mig
Wy
threshold:: 0.202
0.19
0.50
/0g
m
Mig
my
Hllp
Training Completed
Traliingg Completted
0.21 1Fg
0.23
0.31
0.22
0.22
0.23
0.31
W9
W3
176
Wo
W9
W2
Wy
P6
(a) Using trainable threshold.
(b) Using soft-mask by attention.
Figure I: Unlike learning threshold parameter for differentiable pruning (Kusupati et al. 2020 Ning et al. 2020) in (a), IDP generates soft-mask using parameter- attention to accomplish differentiable pruning (b). The moment a weight is pruned (a), weight will not get any update the mask zeros out the gradient. depriving the opportunity for better model training. However, the schemes in (a) ensure the training and test behaviors are identical, yet makes the sparsity control difficult. While IDP reveals different behaviors during trainin/tesst times because the test needs hard-masks our attention-based soft mask allows weight to recover from being pruned over the time, which yields higher model accuracies according to our results.
Such hard constraints make possible have a compact memory layout and efficient inferences on hardware (Jeff Pool 2021 Mishra et al. 2021 Zhou et al. (021).
Differentiable Pruning: Using differentiable techniques for pruning gained popularity due the advances in network architecture search methods, as well as network compression techniques (Liu
al. 2019 Chang et al., 22199] Cho et al. (2022). Using the first derivative of foothill function (Bel- bahri et al. 2019) as mask function and applying L1 regularization on the pruning thresholds has been proposed for differentiable pruning (Ramakrishnan et al. 2020). However, it is not clear why the foothill function is the right choice (i.e., the mask value can exceed 1.0 and is not monotonic). DSA (Ning et al., 2020) finds the layer-wise pruning ratios in differentiable fashion by computing a channel-wise keep probability drawn from the Bernoulli distribution. DSA requires solving sub- problem in order to keep the expected sparsity ratio satisfied. Applying a differentiable technique for channel pruning was proposed (Kang & Han (020) based on the operation-sseeciicc observation that feature map with a large and negative batch mean will get deactivated (or zeroed) by ReLU activation function. A differentiable Markov process studied for channel pruning, where a represents
channel and the transitions from states accounts for the probability of keeping one channel unpruned under the condition that the other channel is also retained (Guo et al. 2020). AutoPrune (Xiao et al. 2019) introduces trainable auxiliary parameters (controlled by a regularizer) to generate a hard pruning mask differentiably using STE. Optimizing and training per-layer pruning threshold with ReLU based on a dynamic re- parameteriization method was proposed to allocate the sparsity across all layers (Kusupati et al. 2020), which shows competitive model accuracy and low inference overhead. IDP is novel kind of differentiable pruning powered by attention-based soft-mask to optimize individual weight masks per task-loss.
3 IDP: ITERATIVE DIFFERENTIABLE PRUNING
In this section, we introduce IDP by first providing the key ideas and insights behind IDP in Section 3.1 explain the details in Section 3.2 followed by the training flow with IDP in Section 3.3
3.1 OVERVIEW
Existing differentiable schemes focus differentiable pruning budget allocation by training differentiable pruning threshold based on marginal loss or Ll-regularization (Kang & Han 2020 Ning et al. 2020). Meanwhile IDP directly generates soft pruning mask based parameter-free
3

---

Under review as a conference paper at ICLR 2023
(a) IDP: epoch 20
(b) IDP: epoch 30
(c) IDP: epoch 40
(d) IDP: epoch 50
(e) IDP: epoch 60
(f) IDP: epoch 70
IDP: epoch
(h) IDP: epoch 90
STR: epoch 20
(j) STR: epoch
(k) STR: epoch
(I) STR: epoch 50,70,80,90
The effects of IDP and STR (Kusupati et al. 2020) (from Table for the first. 3x3 Conv2d layer with 32 filters in MohileNet-vl ImageNettk where each small rectangle represents one 3x3 kernel: For IDP, during entire end-2-end training, we temporarily round soft-mask value make the pruning decisions. The white cell indicates such pruning decision for the corresponding weight has been flipped at least once during the particular epoch, and the black cell indicates the other case.
Figure 2:
attention for each weight to make the pruning process differentiable. Fig. illlstrates the differences with an example. Learning pruning threshold useful for global pruning budget allocation, as the threshold gets adjusted per the task loss in Fig. (a), but has the following drawbacks:
Difficult to control a sparsity level, the pruning threshold sparsity because masks and the thresholds are not directly co-optimized. In Fig. [ (a). after one weight update, the threshold is reduced from 0.205 to 0.202, but the sparsity increased from 50% to 66% as W4 becomes smaller than the threshold.
explicitly related the
Once weight gets pruned during training, it does get updated as the gradient is masked out. The bottom 4 weights W(u.1.2,3) get no weight update due to the zero masks, and can remain likely pruned through the end of training.
In the contrary, IDP allows all the weights to get updated through attention-based soft-mask as in Fig. (b). providing higher flexibility and recovery from undesirable pruning (Guo (0166). For example, consider the weight W2 of value 0.19 which would have been permanently pruned in (a). IDP discourages weight from being permanently pruned through the weight update, as still scaled-down gradient is available. If near-zero weight continues to get negative gradients over the time (even it is scaled down by the soft-mask), it can eventually get unpruned at the end of training. In the same way, if a very large weight gets positive gradients many times enough to be near-zero, it will get pruned eventually, even if was not pruned in the early stage. Hence, such gradual pruning decision over the time during training allows IDP to make better pruning decisions w.r.t. the task loss.
Such soft masking allows pruning decisions to be flipped multiple times during the entire training process, and such effects are compared between IDP and STR (Kusupati et al. 2020) (from Table in Fig. where the pruning decisions for the first Conv2d (3x3 kernels with 32 output channels) in MobileNet-vi for the end-to-end ImageNet training are captured. The white cell means that the pruning decision has been flipped (from to prune to not-to-prune or vice-versa), while the black cell means the decision has never changed, during the given epoch. Then, observe from Fig.
4

---

Under review as a conference paper at ICLR 2023
Algorithm I Computing pruned weight matrix with IDP
1: procedure PRUNIMASK(N)
2:
W, = lop (abs W) - -
3: W LopK ubs (W) r. (rnn(W))
4:
= detach_ 0.55
+ marr(Wi))) Ildtacched from autograd
5:
ZZ M] soʃ Imaax W
/element-wiss operation
6: return M
7: end procedure
F-W(W)
(1 (1-r)-m(W)
10
W/
W/
10
Counts
Attention
10
0.7
m(w)
10²
0.2
z(w)
0.0
0.6
0.8
1.0
Mask values
(a) Attention plot by w Ho with 0.6
(b) soft-mask distribution example
Figure 3: Computing z(w), m(w) for the attentions
M with the equal attention to and M.
that while IDP in (a)-(h) continues to flip the decisions until the very late training stage, STR (i)-(I) mostly finalizes the pruning decisions early (i.e., pruning decision gets static after about 30 epochs).
3.2 SOFT PRUNIG MASK WITH PARAMETER-FREE ATTENTION
Instead of hard-mask, IDP uses soft-mask generated by attention the two symbolic states, "to-prunc" noted as Z or "not-to-prune" noted as M for given weight Unlike the existing attention-based model optimizations such as (Cho et al. 2022), where the attention anchors are numerically defined, the attention to symbolic states is not straightforward to compute (i.e., Euclidean distance to is not available directly), and thus requires a diffcrent approach.
Let us imagine continuous and diffcrentiable functions, 2, [jj, + Ro,,1 where and the attentions to and M, respectively. Then, z and m must satisfy the following conditions to be used attention-bai based soft-mask for the magnitude-driver weight pruning:
2(la)) < 2((bl) for any [a] > [b]: a weight with smaller magnitude has more attention to Z.
mm[aa) > m((bl) for any |a| > [b]: weight with larger magnitude has more attention M.
2(w) m(w)
for any the total attention is 1.
Accordingly, due to the monotonicity and continuity, there must exist E Rzo such that z(1) = m(t) = 0.5 for the equal attention Z and M, which leads the following boundary conditions:
1
if 0
(ww)
0
w
One can use any function that satisfies all of the above conditions, and take m(w) as soft-mask of w for train-time pruning. In IDP, we uniquely identify for a given prune ratio r E [o, 1) for layer with weight matrix W follows based on the the pruning context:
Wn = topK (abs (W), (1 r) n(W)
(1)
Wi (-bs((bbbsW..... (W))
(2)
5

---

Under review as a conference paper at ICLR 2023
Algorithm 2 Training flow for IDP
1: procedure TRAIN(C, s, W [Wo, Wi....)
2:
for epoch e in [0, 2, 8) do
3:
for each mini-batch do forward with [W, -..,, backward and weight update
4:
5: end for end for
6:
7:
W topK (bs(Wa)s))))))) W) r n ( (1.... = (W,, (Ww) W)) (W))
n,,, ,W,
8:
9:
for epoch e [s, + 1,
2,... do
10:
ro. f fi.... mimminn11 1, (e (s)= [ro. r....
for each mini-batch do
12:
for i E {0, 1,... doz
13:
M,= "ls=PruneMask(WWr
14:
W = M; W; //element-wise operation
15:
end for forward with (Wo W..... backward and weight update end for
16:
17:
18:
end for
19:
W; [M] W,, Vi E 1,,
20:
end procedure
where topK( (X, 1e) is selecting the largest & elements from a matrix X, abs(X) is an element- wise absolute operation, and nxX) returns the number of elements. Then, we compute the following:
0.5 (55min(Wi, + mar(Wi))
Note that Eq. (1,2,3) compute outside the autograd (i.e., do not participate the backward- scalar is required for the next step. Then, we propose the following differentiable parameter-free attention to compute z(m), m(w), and the soft-pruned weight w. w.r.t. which represents the current weight distribution.
using torch.no (oroggrr())) )), as only
T m(w) [z(w), mm m[w] [0, w|
so ftmar (t2 w²) [0, w] ftmar (1,,0²]
10, w]
(4)
where is the temperature parameter (see Section C in Appendix for more discussions) and m(w) is directly used as a pruning mask during training. Fig. (a) illustrates how t is obtained IDP and moft- is computed: essentially, t in gray, is in the exact middle of the largest pruned weight in black and the smallest unprumed weight in white when a hard-mask is applied for a given sparsity ratio thus giving the equal attention to Z and M. Fig. (b) shows the soft-mask distribution of the word-embedding layer of BERT (Devlin et al. 2019) Therefore, Eq. 44 satisfies all the constraints
for and m. Further, the softmax is fast enough process large number of weights. The soft-mask matrix generation by IDP is stated in Algorithm which has O(W) runtime complexity.
In comparison scaled-dot product attention (or QKV attention) in Transformer (Vaswani al. 2017), our parameter-free attention has the following similarity, differences, and advantages:
Our parameter-free free attention in Eq. (4) has structure- wise similarity with QKV attention: [0, w| corresponds to V. and and w maps to Q and K, respectively, except that we use a simple concal instead of dot product to get the attention scores from softmax.
Our attention does not have learnable attention parameters, based on the structure of pruning. Instead, let weights learn and improve their distribution generate better attention scores (through weight update) in the presence of our soft-mask for the subsequent iterations.
Having no extra parameters and using concar makes IDP very efficient in terms of training speed and keeps training flow simple, yet our experimental results show its effectiveness.
In short, using a soft-mask by enables the pruning decision for the weights around the pruning boundary (i.c., t) to be driven by a task loss through back propagatton rather than the weight value itself. On the contrary, a pruned weight will get zero gradient with hard-mask once pruned.

---

Under review as a conference paper at ICLR 2023
550
Infererncc MAC (e6)
490
520
GraNet
ACDC
Inference MAC (e6)
800
GraNet
ACDC
460
720
430
400
370
STR
IDP-hase
560
340
310
IDP-str
480
Pstr IDP-optg
OptG
IDP-opig
OptG
220
66%
67%
68%
69%
70%
72%
73%
74%
75%
Top-1 Accuracy
Top-1 Accuracy
(a) ResNet18 (sparsity: 85.5%)
(b) ResNet50 (sparsity: 89.8%)
110
Inference MAC (e6)
GraNet
ACDC
Inference MAC (e6)
100
GraNet
ACDC
90
IDP-base
IDP-opts
OptC
70
80
OptG
IDP-optg
70
IDP-base
60
60
50
-TR
IDP-sir
STR
IDP-str
61%
63%
65%
67%
69%
56%
60%
64%
68%
Top-1 Accuracy
Top-1 Accuracy
(c) MobileNet-vl (sparsity: 86.6%)
(d) MobileNet-v2 (sparsity: 80.2%)
Figure 4: IDP-peeered pruning (in white box markers) delivers the Pareto superiority the other schemes (i.e., the top-bottom corner is best trade-off) ResNet18, ResNet50, and MobileNet-vI/v2 on ImageNetlk, The size of markers indicates the relative training overheads. The detailed numbers are Table. 2
3.3 IDP ALGORITHM AND TRAINING FLOW
In order to obtain t in Eq. 3), IDP needs a target pruning ratio The pruning ratio can be computed by selecting the top weights with larger magnitudes across all the layers and then instantly convert the selections into the per-layer ratios. Another way is to handcraft per-layer ratios, or reuse an existing configuration. Also, IDP is using the softmax operation which makes the softness concentrated over the weights around the / (as shown in Fig. (3). Hence, we gradually increase the target pruning ratio from to r so that all low magnitude weights in the pruning range have a chance to use a soft-mask and settle down smoothly. For that purpose, we introduce a scaling step to let each weight have opportunities to leverage a soft-mask at least once, which leads to the training flow in Algorithm 2
In lines 2-6, normal training is performed for the first epochs. Then, in lines 7 and the per-layer target pruning ratio is computed by selecting the bottom r nWW weights globally in terms of the magnitude. Then, in the remaining epochs, use IDP to generate soft-masks as in the line 13, while gradually increasing the target ratio as in lines 9 and 10. Once the entire training is over, we binarize the last mask for each weight to output the fully pruned weight for inference in the line 19.
EXPERIMENTAL RESULTS
We compared our IDP with state-of-the-ar unstructured pruning schemes on various computer vision and natural language models. We used two x86 Linux nodes with eight NVIDIA V100 GPUs each.
Vision Benchmark: We compared the proposed IDP with the latest prior arts, STR (Kusupati et al. 2020), GMP (Zhu & Gupta 2018), DNW (Wortsman al. 2019) GraNet (Liu et al. 2021
7

---

Under review as a conference paper at ICLR 2023
Network Sparsity
Method
Top-1
GPU
MAC
Network Sparsity
Method
Top-1
GPU
MAC
(%)
hours
(xe6)
(%)
hours
(xe6)
Dense
69.8
91
1814.1
Dense
76.1
135
4089.2
GMP
65.2
118
263.5
GMP
73.6
263
419.0
DNW
64.4
112
263.5
DNW
70.7
254
419.0
GraNet
66.0
108
539.6
GraNet
72.5
175
868.0
ResNet18
STR
66.7
93
334.6
ResNet50 89.8%
STR
72.8
227
373.7
85.5%
OptG*
65.5
151
223.7
OptG*
72.1
322
333.0
ACDC
68.7
194
502.8
ACDC
74.7
346
735.6
IDP-base
69.0
92
408.6
IDP-base
74.7
177
502.8
IDP-base+
69.5
183
405.1
IDP-base+
75.3
332
483.0
Itt-str
68.6
92
334.7
IDP-str
74.0
177
373.7
IDP-
68.5
95
223.7
IDP-optg
74.2
181
332.9
Dense
70.9
151
568.7
Dense
71.9
162
300.8
GraNet*
61.4
200
145.7
GraNet
56.3
239
103.4
MobileNet-vi
STR
61.7
96
47.2
MobileNetv-2
STR
60.0
155
40.6
86.6%
OptG*
66.3
185
87.4
80.2%
OptG*
65.4
297
76.8
ACDC
66.5
349
124.5
ACDC
64.1
442
93.9
IDP-base
68.2
153
88.3
IDP-base
66.8
193
95.3
IDP-str 65.3
167
47.2
IDP-str
60.7
167
40.6
IDP-optg 68.2
162
87.3
IDP-optg
66.5
187
76.6
the total GPUs hours on all the GPUs in synchronous distributed training.
used only one with 8 GPUs due to the limitations in the public code.
Table 2: IDP compared with other unstructured pruning algorithms on with ImageNet1 shows the best tradc-off
among accuracy, inference MAC, and training overheads. More results are available in Scction D in Appendix.
OptG (Zhang al. 2022), and ACDC (Peste al. 2022] ResNet18, ResNet50, MobileNet-v 1, and MobileNet-v2 He et al. 2016 Howard et al. 2017 Sandler et al. 2018) with the ImageNet dataset (Deng et 2009p. Since
these schemes have been experimented only with ResNet50 and/or Netvve we reproduced the pruning results in our controlled environment with the identical data augmentations by running the official implementations from the authors (Kusupati) al. 2020; Zhang al. 2022 Peste et al. 2021 Liu et al.) 2021) verified implementations from the prior arts (Wortsman et al. 2019 Zhu & Gupta 2018 8 as Section in Appendix. Since the primary goal of pruning
trade-off the model accuracy with the compute reduction as in Section 2 we measured the accuracies and inference-tims Aultiply-Accumile
Operation (MAC) on each experiment with layer fusion (i.e., BatchNorm folding), and mainly focused on the high-sparsification cases. Note that the MAC is purely theoretical and reported to understand the trade-off among accuracy, size, and compute across various algorithms. In experiments with ImageNet1k, all layers, including both the first and last layers, are pruned without any restriction. Also, we estimated training efficiency wall-clock time by measuring the total GPU-hours (Chen et al. 2022). which is crucial to capture the algorithmic complexity and training flow overhead.
We applied not only the proposed hyper- parameerrssserss from the authors but also set of further fine- tuned hyper-pann parameters for competing methods. Also, since each algorithm used different number of epochs and showed results different sparsity levels, a) we ran STR first to set the target sparsity levels for all the networks for fair comparisons, because all other schemes can control the sparsity level precisely, b) we trained ResNet 8/50 for 100 epochs
MobileNet-vl/ve for 180 epochs
following (Peste al.) 2021 Kusupati et al.) 2020: Liu et al. 2021 Zhang et al.) (022) except STR (which diverged with more epochs for MobileNet-vI/v2), For IDP, we started pruning from the epoch 16 at the rate of 1.5% of the target sparsity per epoch for all the experiments which correspond to s = 16 and c = 0.015 in Algorithm [] For detailed experiment configurations, please refer to Table 4 Every experiment began with randomly initialized model (i.e., no pre-trained model). For IDP, we had the following variants to demonstrate the value of IDP with the same training overhead with the same per-layer pruning budgets.
IDP-base computes the target sparsity level by using weight magnitude globally, as in the
lines 7 and 8 of Algorithm
IDD--ase+ is the same as IDP-base except runs for additional epochs to consumes the similar total GPU hours with ACDC.
8

---

Under review as a conference paper at ICLR 2023
Validation dataset
Methods
Network
Sparsity
Dense
STR*
OptG
GMP
MVP
POFA
IDP
90
75.8
78.5
n/a
81.2
81.5
83.1
matched
84.5
94
74.4
76.9
74.8
80.7
n/a
82.0
Bert-base
90
76.3
78.3
n/a
81.8
82.4
83.0
mismatched
84.9
94
74.1
76.5
75.6
81.2
n/a
82.4
Since STR cannot control the sparsity precisely, we report the metrics with our closest achieved sparsity levels, 85.9% for the 90% case and 8% for the 94% case.
Table 3: IDP delivers the best accuracies on the BERT-based model with the MNLI (Multi-Genre Natural Language Inference) task of the GLUE benchmark.
IDP-str/optg uses the sparsification levels resulted from STR/OptG input to achieve the similar MAC by overwriting [r0, rl,... in the line of Algorithm 2
Our experimental results are highlighted in Fig. where the size of circles indicates the relative training overhead due to pruning. Note that we used only one single node with 8 GPUs due the limitation in the official implementations for GraNet and OptG, thus both have the advantage of not having the note communication cost. Also, each approach imposes different level of training- time overhead, mainly due to the various complexities of training flow and pruning itself as captured Fig. Overall results can be summarized follows:
IDP showed the best the model accuracy: IDP-base on ResNet18 delivered 69% Top-1 accuracy which is superior other schemes but higher MAC than only STR and OptG.
IDP offered the better model accuracy for given pruning target: With the custom sparsifi- cation target for each layer, IDP-str/opig demonstrated the 2-3% higher Top-1 accuracy at the same MAC, demonstrating the effectiveness of the proposed method.
When we use the similar GPU hours for additional epochs with ACDC which is noted IDP-base+, method further improved the Top-1 accuracy from 69% to 69.5% for ResNet18 and from 74.7% to 75.3% for ResNet50 with slight fewer MACs.
We compared IDP with the state-of- the-art art pruning results from MVP Sanh et al. 2020) and POFA (Zafrir et al. 202T) (quoted from the respective papers) in addition to OptG a BERT model (Devlin et al., 2019) for NLP task, MNLI (Multi-Genre Natural Language Inference) of the GLUE benchmark. Following the setups in MVP and POFA, we used the same batch size 32 per GPU (i.e., global mini-batch size of 512), excluded the embedding and the last linear layer from pruning, and trained the bert- serrt-bse--une unccadd from HuggingFace from scratch with self-distillation The teacher for the distillation is from the best checkpoint in the first 40 epochs. We use the weights of 0.95 on the distillation loss and 0.05 on the task loss for IDP, and 0.75 on the distillation loss and 0.25 on the task loss for STR. We could not use distillation for OptG, as it caused the out-of-memory error due to the extra-parameter overhcads from both pruning and distillation. For more details, please refer to Table 4 in Appendix. Our experimental results in Table can be summarized follows:
NLP Benchmark:
and STR (reproduced in our environment)
STR underperforms on all the best cases even though could not achieve the target sparsity.
OptG shows better model accuracy only than GMP and worse than others in our setup.
IDP outperformed all other methods for both MNLI validation datasets.
CONCLUSION
In this work, we proposed differentiable pruning method, IDP which yields the state-of-the-art pruning quality on popular computer vision and natural language models. Our method requires no additional learning parameters, yet keeps the training flow simple and straightforward, making a practical method for real- real-world scenarios. We plan to extend our differentiable pruning into quantization, making both jointly differentiable and optimizable by the task-loss.

---

Under review as a conference paper at ICLR 2023
6 ETHICS STATEMENT
The proposed technique does not incur any concern or penitential issues around ethics.
7 REPRODUCIBILITY STATEMENT
We provide the details hyper-parametery and experimental configurations in Seciton 4 and Table 4 in Appendix. For Imagenet k. we used common data augmentation techniques: RandomResized- 0.456, 0.406], std=[0.229, 0.224,
Crop(224), RandomHorizzone 0.225l) based on the prior arts (Park & Yoo! 2020: J. Lee 2021). During evaluation, we used the following augmentations: Resize(256), CentorCrop(224)) and Normalize(mean=((A4 std=[0.229, 0.224, 0.225]). For the MNLI benchmark, we used the default setup from HuggingFace.
and Normalize
0.456, 0.406),
REFERENCES
Sajid Anwar, Kyuyeon Hwang, and Wonyong Sung. Structured pruning of deep convolutional neural networks. ACM Journal on Emerging Technologies in Computing Systems, 2017.
Dzmitry Bahdana, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations, 2015.
Mouloud Belbahri, Eyyüb Sari, Sajad Darabi, and Vahid Partovi Nia. Foothill: A quasiconvex regularization for edge computing of deep neural networks. In Fakhri Karray, Aurélio Campilho, and Alfred C. H. Yu (eds.), International Conference on Image Analysis and Recognition, 2019.
Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR, 2013.
Jianlong Chang, xinbang zhang, Yiwen Guo, GAOFENG MENG, SHIMING XIANG, and Chun- hong Pan. Data: Differentiable architecture approximation. In Advances in Neural Information Processing Systems, 2019.
Beidi Chen, Tri Dao, Kaizhao Liang, Jiaming Yang, Zhao Song, Atri Rudra, and Christopher Ré. Pixelated butterfly: Simple and efficient sparse training for neural network models. In International Conference on Learning Representations, 2022.
Minsik Cho, Keivan Alizadeh- Vahid, Saurabh Adya, and Mohammad Rastegari. DKM: differentiable k-means clustering layer for neural network compression. In International Conference Learning Representations, 2022.
J. Deng, W. Dong, R. Socher, Li- K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Scale Hierarchi- cal Image Database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2009.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, 2019.
Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, 2020.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. CoRR, 2019.
Shaopeng Guo, Yujie Wang, Quanquan Li, and Junjie Yan. Dmep: Differentiable markov channel pruning for neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 2020.
10

---

Under review as a conference paper at ICLR 2023
Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dans. In Advances in Neural Information Processing Systems, 2016.
Song Han, Jeff Pool, John Tran, and William J. Dally, Learning both weights and connections for efficient neural network. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett (eds.), Advances in Neural Information Processing Systems, 2015.
Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. In International Conference on Learning Representations, 2016.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2016.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In The IEEE International Conference on Computer Vision, Oct 2017.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint orXiv::1744 0700444461, 2017.
B. Ham J. Lee, D. Kim. Network quantization with element-wise gradient scaling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021.
Chong Yu Jeff Pool. Channel permutations for n.m sparsity. In Advances in Neural Information Processing Systems, 2021.
Minsoo Kang and Bohyung Han. Operation-aware soft channel pruning using differentiable masks. In International Conference on Machine Learning, 2020.
Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade, and Ali Farhadi. Soft threshold weight reparameterization for learnable sparsity. In International Conference on Machine Learning, July 2020.
François Lagunas, Ella Charlaix, Victor Sanh, and Alexander M. Rush. Block pruning for faster transformers. In The Conference on Empirical Methods in Natural Language Processing, 2021.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In International Conference on Learning Representations, 2017.
Yuhang Li, Xin Dong, and Wei Wang. Additive powers-of-two quantization: An efficient non-uniform discretization for neural networks. In International Conference Learning Representations, 2019.
Zhuohan Li, Eric Wallace, Sheng Shen. Kevin Lin, Kurt Keutzer, Dan Klein, and Joseph E. Gon- zalez. Train large, then compress: Rethinking model size for efficient training and inference of transformers. In International Conference on Machine Learning, 2020.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: differentiable architecture search. In International Conference on Learning Representations, 2019.
Shiwei Liu, Tianlong Chen, Xiaohan Chen, Zahra Atashgahi, Lu Yin, Huanyu Kou, Li Shen, Mykola Pechenizkiy, Zhangyang Wang, and Decebal Constantin Mocanu. Sparse training via boosting pruning plasticity with neuroregeneration. In Advances in Neural Information Processing Systems, 2021.
Z. Liu, P.W. N. Whatmough, Y. Zhu, and M. Mattina. S2ta: Exploiting structured sparsity for energy- efficient mobile enn acceleration. In 2022 IEEE International Symposium on Higg-Peeformance Computer Architecture (HPCA), 2022.
Sachin Mehta and Mohammad Rastegari. Mobilevit: Light-weight, pereral- and mobile- friendly vision transformer. In International Conference on Learning Representations, 2022.
Asit K. Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu, and Paulius Micikevicius. Accelerating sparse deep neural networks. CoRR, 2021.
11

---

Under review as a conference paper at ICLR 2023
Xuefei Ning, Tianchen Zhao, Wenshuo Li, Peng Lei, Yu Wang, and Huazhong Yang. DSA: more efficient budgeted pruning via differentiable sparsity allocation. In European Conference on Computer Vision, 2020.
Eunhyeok Park and Sungjoo Yoo. Profit: A novel training method for sub-4-bit mobilenet models. In European Conference on Computer Vision, 2020.
Alexandra Peste, Eugenia lofinova, Adrian Vladu, and Dan Alistarh. AC/DC: alternating com- pressed/decompresssed training of deep neural networks. In Advances in Neural Information Processing Systems, 2021.
Antonio Polino, Razvan Pascanu, and Dan Alistarh. Model compression via distillation and quantiza- tion. In International Conference on Learning Representations, 2018.
Ramchalam Kinattinkara Ramakrishnan, Eyyüb Sari, and Vahid Partovi Nia. Differentiable mask for pruning convolutional and recurrent networks. International Conference on Computer and Robot Vision, 2020.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chiee Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, June 2018.
Victor Sanh, Thomas Wolf, and Alexander Rush. Movement pruning: Adaptive sparsity by fine-tuning. In Advances Neural Information Processing Systems, 2020.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419): 1140-11 44, 2018.
Hidenori Tanaka, Daniel Kunin, Daniel L. K. Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. In Advances in Neural Information Processing Systems, 2020.
Pavan Kumar Anasosalu Vasu, James Gabriel, Jeff Zhu, Oncel Tuzel, and Anurag Ranjan. An improved one millisecond mobile backbone. 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, t. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Chaoqi Wang, Guodong Zhang, and Roger B. Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations, 2020a.
Kuan Wang. Zhijian Liu, Yujun Lin, Lin, and Song Han. Haq: Hardware-awar automated quantization with mixed precision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
Ying Wang. Yadong Lu, and Tijmen Blankevoort. Differentiable joint pruning and quantization for hardware efficiency. In European Conference on Computer Vision, 2020b.
Mitchell Wortsman, Ali Farhadi, and Mohammad Rastegari. Discovering neural wirings. In Advances in Neural Information Processing Systems, 2019.
Junru Wu, Yue Wang, Zhenyu Wu, Zhangyang Wang, Ashok Veeraraghavan, and Yingyan Lin. Deep k-means: Re-training and parameter sharing with harder cluster assignments for compressing deep convolutions. In International Conference on Machine Learning, 2018.
Xia Xiao, Zigeng Wang, and Sanguthevar Rajasekaran. Autoprune: Automatic network pruning by regularizing auxiliary parameters. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-But Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, 2019.
Kelvin Xu, Jimmy Ba. Ryan Kiros. Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International Conference Machine Learning, 2015.
12

---

Under review as a conference paper at ICLR 2023
Ofir Zafrir, Ariel Larey, Guy Boudoukh, Haihao Shen, and Moshe Wasserblat. Prune once for all: Sparse pre-trate mre-trained language models. In Advances in Neural Information Processing Systems, 2021.
Yuxin Zhang, Mingbao Lin, Mengzhao Chen, Zihan Xu, Fei Chao, Yunhang Shen, Ke Li, Yongjian Wu, and Rongrong Ji. Optimizing gradient-driven criteria in network sparsity: Gradient is all you need. In CoRR, 2022.
Xiandong Zhao, Ying Wang, Xuyi Cai, Cheng Liu, and Lei Zhang. Linear symmetric quantization of neural networks for low-precision integer hardware. In International Conference on Learning Representations, 2019.
Aojun Zhou, Yukun Ma, Junnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hongsheng Li. Learning n.m fine-grained structured sparse neural networks from scratch. In International Conference Learning Representations, 2021.
Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing lottery tickets: Zeros, signs, and the supermask. In Advances in Neural Information Processing Systems, 2019.
Michael Zhu and Suyog Gupta. To prune, or not to prune: Exploring the efficacy of pruning for model compression, In Workshop, International Conference Learning Representations, 2018.
13

---

Under review as a conference paper at ICLR 2023
other params
init density: 0.5
init value: -3200 3 ::1.0
8 alterations
T: 1e- , :::::::00000 :.0055
le 1e:lecc:0 :::0=:0::0000 0.015
init density: 0.5
init value: -3200
8 alterations
T: : 1e-4 - 0.015
1e 1e-4 cc: 0.015
init density: 0.5 0.5
init value: -12800
8 alterations
T: :: le 1e-4 .. 0.015
init density: 0.5
init value: -12800
8 alterations
T: le 4 *. 0.015
init value: .90
init value: -110
7 le ::: 0.015
3:1.0
1.0
8: 1.0
8:10 1:0
SGD: momentum, weight decay, cosine: leamning rate, Adam W: epsilon, multiplicative learning rate, gamma, #wu: the number of warm-up epochs.
Mask optimizer, scheduler
SGD SGD 0.875 0.066 cosine le-4
S.85 0.065, cosine le-4
n/a same the main same as the main SGD 0.9 0, cosine 0.1 n/a
SGD 0.9 O, cosine 0.1 n/a
SGD 0.9 O,, cosine 0.1 n/a
SGD 0.9 0. cosine 0.05 n/a
SGD 0, cosine le-4 n/a
same as the main same as the main
same as the main
same as the main
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
Refer to Section E and Tables and 6 in (atri et at. 2021
Refer to Section 6 and Table 3 in Sanh el al. 2020
Refer to Scction and Table 3 in Sanh et al. 2020
SGD SGD 0.875 2.2517578 1e--5 cosine 0.256
O.D 3.05115771 13e-4, cosine 0.256
2.5177777777 13--5. cosine 0.256
0.875 3.051757813e 13-133-444 cosine 0.256
0.87. 737757577811--55 cosine 0.256
SGD 3.0517578 11-8-4 cosine 0.256
0.870875 3,751757813.-5 13c-5, cosine 0.256
SGD 3.05175788 13e-4, cosine 0.256
SGD 0.9 le-4 (0.0 for BN), cosine 0.1
SGD 0.9 le-4 (0.0 for BN), cosine 0.1
SGD 0.9 lc-4 (0.0 for BN), cosine 0.1
le-4 (0.0 for BN), cosine 0.1
Adam W 1e-8, multiplicative le.4 0.95
Adam W 1e-8, multiplicative le-4 0.95
SGD 0.875 le-5. cosine 0.256
SGD 0.875 le-5. cosine 0.256
Main optimizer, scheduler
SGD 0.9 lc-4, cosine 0.1
SGD 0.9 1.4e-5, cosine 1.8
SGD 0.9 1. 1.4e-5, cosine 1.8
SGD 0.9 le-4, cosine 0.1
SGD 0.9 1.4e-5. cosine 1.8
SGD 0.9 1..e-5, cosine 1.8
SGD 0.9 4e cosine 0.1
SGD 0.9 1 4c-5, cosine 1.8
SGD 4-e4. cosine 0.05
SGD 0.9 8e-6, cosine 0.8
Adam 1e-8, cosine le-4
Adam W 1e-8, cosine le-4
Table 4: The hyper- pere-parrmeeess Section 4
0.D
SGD 0.9 0.9
Batch #epochs #wu #GPUS #Nodes
2
2
2
2
2
1
2
2
2
1
2
2
1
2
2
2
1
2
2
2
2
2
2
16
16
16
8
16
16
8
16
16
16
8
16
16
8
16
16
16
8
16
16
16
16
16
16
16
100
100
100
100
100
100
200
100
100
100
100
100
100
200
180
100
180
180
180
180
100
180
180
180
140)
140
140
140
128
128
256
256
256
1024
1024
128
128
256
256
256
1024
1024
128
256
256
256
1024
128
256
256
256
1024
32
32
32
32
DoP-base/strroptg
IDP IDP- basc/stt/////tggggggg
IDP IDP- bbbs/////////ttggggggggg
IDP-base/str/optg
IDP-base+
IDP-base. base+
STR (85. 9%
STR (91.7%)
Method
GraNet
GMP
STR
OpiG
ACDC
GraNet
GMP
STR
OptG
ACDC
GraNct
STR
OptG
ACDC
GraNet
STR
OptG
ACDC
GMP
MOV
POFA
OptG
IDP
MobileNet-vi
MobileNet-v2
Network
ResNer18
ResNer50
Bert
14

---

Under review as a conference paper at ICLR 2023
190%
90%
Sparsity
con1
leed hoad
lgped 1ppo
Uppec Lood
Lppert 1pamm
leeck Loom2
Izper Izperk Gooni
layerfn comsarrlll
tppent cooe
byyen Loow
lapor
lepe
Uppel
layed
IDP GraNet
# STR
OptG
ACDC
(a) ResNet18
100%
Sparsity
0%
aper apriinn
Jged) Leed
Syer coed
andrt Sardd
byert bacrk
Jgee Aesd
and andtlfeed fard
byokk
lzeek balnssanccc
Agodt Apodlenn
Spoed boodl Lond
Agest
Sayel
Lepel
lapel
IDP GraNet STR
OptG
ACDD
(b) ResNet50
Sparsity
300
10000000
moki20
modi.100
mabt.003
mobi.11
modu120 120
15
chaifer
IDP GraNet
STR
OptG
ACDC
(c) MobileNet-vl
Sparsity
304
tauns- nanne Lowe
detune 2 aam0
fetms Seuns Sen 2
kauno kauno Aomid
femar tam.
Tunnet tunnect 2
funune 2
batun Satus 7amnn
Seturs Seurs
ksurs kaus Kome. 10
Sauno kauno-lltt-l Ma
homm1
dazze Iilande
foane Som12
Satue 12mm000
Satar B Be
Satur Leme.
fearee 4.m.1
mane 2
faaue fatues 3
faturs fates 17
foars 3
d
fonues
Sattne 11
IDP
GraNet STR OptG
ACDC
(d) MobileNet-v2
Figure 5: Layer-wise sparsity allocation from the experiments in Table
15

---

Under review as a conference paper at ICLR 2023
80
Inference MAC (e6)
con
love Roomi
kged Boomd
Ipper Boowa
leyed Lomm
leced Doam2
Sapprt downaappp
lypert cood
lepd
lyee
Jaye2
layod
IDP
GraNet
# STR
OptG
ACCD
(a) ResNet18
60
Infererence MAC (e6)
40
cmn
Lypt foom
Uper Umper Booms
layet deneraapoo
Uped 1ood
lyee comd
Loond
Lged Joamd
lyc2 lae22aon
layet Aettonooarrp ca
Undr Liter Zomm
Lyaa' Lyye5 homm
tye Zoom
Lyach form
typer comm
tyee Eoom!
Lyed Boom?
lyee Roand
lavch Laann
Lncch Loand
Lypec comntt
Lyed Foomd
Lyper' Foonch
Lypel Dooon
Epes foond
layek Doonnd
layek lapok Loand
Laookaam
land
Lood
Lyyar
Lyyac
lyye'
Laook
Loook
IDP GraNet SIR
OpIG #CCD
(b) ResNet50
20
Inference MAC (06)
10
thet
mobdns
mode1.3
modelK
IDP
GraNet
STR
OptG
#ACDC
(c) MobileNct-vl
Inference MAC (e6)
10
g
[
Salura Istural0 2e E
Bonred coee
Icans Zonn Donny 00
Itarsaami
fornciine
frauee Tootr DD
falund
trana trara Koanr to
mane forr
falundood
fanaro fraar Sour no
Ieams fears Samr
Gunns foor
fenn foar 1d
Rmae Toar1
Rom
frann Brarak ko
formec cooe
frnares Sour 13
tunne Mooo 10
Ioeus Haam
Satmncl 1 co00
funns Thamii
Satural 3oord
fenass Ro
fanmc
funes Hoarr D0
tunne Hoon LO
Mooo
toxes Hoarr to
feoure IRS
dhaifee)
lanns
IDP GraNet STR
OptG
ACDC
(d) MobileNet-v2
Figure 6: Layer-wise Inference MAC distribution from the experiments Table
16

---

Under review as a conference paper at ICLR 2023
Dense
GraNet
STR
OpIG
ACDC
IDP
0.0
1.0
1.3
2.0
2.3
3.0
3.3
4.0
4.3
5.0
5.3
6.3
7.0
7.3
8.0
8.3
9.0
9.3
10.0
10.3
fe
Table The weight histograms in log scale MobileNet-vl in Table
17

---

Under review as a conference paper at ICLR 2023
A TRAINING CONFIGURATIONS AND HYPER-PARAMETERS
Since some techniques in Section 4 require extra training parameters and pruning scheduling as shown in Table we disclose the training configurations and hyper-parameters found the best in Table 4
B TRADE-OFF IN PRUNING.
Pruning for DNN requires exploring good trade-off between model accuracy and inference latency under given pruning target. Such a challenge can be elaborated with the MobiletNet-vl Dense case in Fig. 7 where the following observations can be made:
The earlier layers have significantly fewer parameters than the later layers while still having comparable inference MACS as shown in (a). For example, the final classifier, which is linear layer, has the lowest inference MAC but the 2nd largest parameters.
When per-parameter inference MAC is computed as in (b) (which in log-scale), we can easily that the parameters in the earlier layers get much more involved in the inference than those in the later layers. For example, the MAC-per-paramette for the last classifier is only 1.
Then, with given pruning target, one pruning scheme can favor heavily pruning the classifier, as it is easier to hit target without affecting model accuracy much (i.e., each parameter shows up only once in the forward pass), but this would fail to reduce the inference MAC enough. Then, the other scheme may favor aggressively pruning the earlier layers to significantly minimize the inference latency at much greater risk of degrading the model accuracy. Therefore, it is critical find good balance between accuracy and inference speed. According to our experimental results, IDP can accomplish such balance using differential pruning w.r.t. the task loss.
30%
Inferunce MAC
Paraneter Count
20%
9%
mode:33
Normalized Inference
-Normalized Parameter Count
Normalized inference MAC and parameter for each layer.
16,000
Inference MAC
100
0000 1.8
1.00113
maki22
mokk223
1000333
mo0666
mode
meek
modd lns
mobile
no00113
129
mobiiS
MACiparameter
(b) The inference MAC per parameter for each layer.
Figure 7: Laser-wise Inference MAC and Parameters from the MobileNet-vl Dense case Table
18

---

Under review as a conference paper at ICLR 2023
C ABLATION STUDY: HYPER-PARAMETER 7 SEARCH
In the current IDP implementation, use global control the level of softness in the pruning mask. Therefore, the selection of r affects the model predictive power and should be carefully tuned. In order explore the methodology for the search, we tried various values for MobileNet-v2 training. and the results are plotted in Fig. [ The selection of 7 affects the model predictive power as shown in Fig. where there appears to be an optimal 7. For examples of MobileNet-v2, le the best value and is used for all the experiments in Section
Top-1 Accuracy
66.9%
66.7%
66.6%
66.5%
66.4%
1.0E-06
1.0E-05
1.0E-04
1.0E-03
1.0E-02
T
Figure 8: MobileNet-v2 with varying values.
Since, Fig. showws a concave curve, one could use a binary search to find the best values w.r.t. the top-1 accuracy. Also, it could be possible to cast 7 as a learnable parameter for each layer or apply some scheduling to improve the model accuracy further (as future work), but still both approaches need an excellent initial point which can be found using a binary search technique.
D ADDITIONAL RESULT FOR SECTION 4
Different approaches made different sparsity allocations per the characteristics of the algorithm for given pruning target, which results in complex trade-off's between model accuracy and inference speed. We report the detailed sparsity and inference MAC break-down for each layer in Fig. 5 and Fig. on ImageNet1k and summarize our observations as follows:
OptG prunes the early convolution layers quite aggressively ResNet18 and ResNet50, which leads to very low inference MACs shown in Fig. (a) and (b), yet at the cost of the worse Top-1 accuracy. For example, the inference MAC of ResNet18 from OptG is more than 2x less than that from ACDC,
Interestingly, STR becomes aggressive in pruning the early convolution layers in MobileNet- v1/v2, while OptG does not expose such behavior to MobileNet-vl/v2 (unlike it did for ResNet18/50), Such characteristies also favor the low inference latency over the model accuracy. Also, STR tends not to prune the last linear layer much discussed in (Kusupati ct al. 2020).
Unlike OptG and STR, ACDC does not prune the early convolution layers much for the tested networks, but prunes somewhat actively for the late convolution and linear layers, which leads to high model accuracies at the cost of higher inference latencies.
IDP is somewhat between STR and ACDC and modest across all layers pruning allocation for all the networks, leading to superior accuracy and inference trade-offs. For example, the layers model. 13.3 of MobileNet-vi and features. 1.covvv ..0 of MobileNet-v2 have the most difference among algorithms, and IDP is modest in pruning these two layers.
OptG has very low inference MACs on the earlier layers ResNet18 and ResNet50 due its aggressive pruning on these as seen in Fig. 5 (a) and (b), which leads to the extremely low inference latencies as shown in Fig.
and (b).
GraNet tends to prune the earlier layers less but the later layers more in general which explains why GraNet shows the highest inference MACs in Fig.
19

---

Under review as a conference paper at ICLR 2023
Table. shows the pruned weight histograms of MobileNet-vl from Table 4 We can observe that each algorithm affects the weight distribution in a slightly different way.
STR prefers to split the distribution more widely than others. For the example of the layer 5.0, STR clearly separated the positive and negative weights with wide gap centered at the zero, while others sis not, except IDP created a slight dip around the zero to create mild separation.
IDP tends to spread out the sparsified weight distributions more than others. For the example of the fe layer, the weights from IDP range from -0.5 2.0, while those from others are from -0,5 to 1.5. On the other hand, GraNet tends to keep the weight distributions tight.
We also experimented with varying pruning rates for IDP, OptG and ACDC for MobiletNet-vl and ResNet-18 with ImageNetIk, and Bert with MNLI benchmark under the same configuration Section Overall, all tested schemes delivered higher accuracy with lower pruning rate, yet we can observe that IDP keeps its superiority to other schemes over all the tested pruning rates.
in
Sparsity (%)
Network
Method
80
70
60
50
IDP
69.5
71.0
71.6
71.9
MobileNet-vl
OptG
68.1
69.1
69.6
69.7
ACDC
68.5
69.9
70.9
71.4
IDP
69.8
70.8
71.0
71.3
ResNet-18
ACDC
69.4
70.3
70.6
70.8
Table 6: Top-1 accuracy with ImageNet1 k: IDP outperforms other schemes with various pruning rates.
Validation
Sparsity (%)
Network
Method
dataset
80
70
60
50
matched
83.7
84.0
84.3
84.7
IDP
mismatched
83.4
83.8
84.4
84.5
Bert
matched
80.3
80.7
81.3
81.2
OptG
mismatched
80.1
80.7
80.5
81.0
Table 7: Accuracies with MNLI benchmark: IDP maintains the similar accuracy lead over other schemes.
E PARAMETER- FREE ATTENTION
In this section, discuss insight behind our attention and illustrate how compute the soft-pruned weight matrix following Eq. (4) and Algorithm 3.2
Our attention is parameter-free purpose. In pruning, having learnable masks is quite expensive in a way that not only doubles the number of learnable parameters but also may require some particular learning schedulers and tuning, making the overall training flow cumbersome and complicated. Instead, we let weights learn and improve their distribution (weight update changes the distribution naturally) generate better attention scores for the subsequent iterations the presence of the current soft mask. Essentially, captures the current weight distribution, and what would be learned by extra learnable parameters is fused into the weights and their distribution. This way, we can understand where this weight is in terms of overall weight distribution and let backward- paaaardddssssssaa optimize the weight distribution for better pruning. This also key difference from differentiable NAS including DARTS (Liu et al. 2019) where each sub-network comes with learnable parameter and the value distribution of the alpha vector drives the search process.
This is feasible due to the structure of the pruning problem. Pruning is optimizing weight distribution for a task loss under the pruning constraints. Updating parameters for task-loss also affects the pruning quality at the same time (i.e.. trade-off). Therefore, need to find distribution that is best for task loss and also pruning friendly. Hence, instead of having extra parameters, we can directly optimize the weight distribution for high quality pruning. For example, if we surely need to be
20

---

Under review as a conference paper at ICLR 2023
pruned for some reason, instead of having new parameter to denote 'to-prune', make w itself smaller relatively against other parameters in the same layer (this is why is needed) via gradient and weight update. And, IDP is shown to be effective in achieving that with differentiable soft-mask based on parameter-free attention. Our results confirm that our parameter- attention is effective by showing superior results other state-of-the-an algorithms that have explicit mask parameters.
actually
For example, consider 50% pruning of weight matrix with 0.1.
00.1
0.2
W
0.4
00.6
Then, we can compute represent the current weight distribution from the pruning perspective: f == 0.5(0.2 0.4) = 0.3. With = 0.3, we can compute the attention scores capture the attention to to-prune and not-to-prune for each weight, which is further dot-producted with W get the soft-masked weight W as follows.
00.01
0.04
C0.0at(0.09
0.16
0.36
W- so aomaa
10, W]
(5)
T0.09, 0.017
0.09, 0.04
0.09, 0.16
TO, 0.17
0.09, 0.36
0,0.2
= softmax(
(6)
0.1
0,0.4
0,0.6
T0.69, 0.31
0.1
0.62, 0.38
0.0.2
0.33, 0.67
0,0.4
0.06, 0.94
0.,066
O + 0.031
0 + 0.076
(8)
0 + 0.268
0 + 0.564
00.0317
0.076
0.268
0.564
F
CODE REFERENCES
Dense http://yyyocllaghhhiiivsaallllme
GradNet htnp://ggiibbbccom//rrrAACGGGGGu
OptG hutps//gtthbbcoom///xxxm///ppg
ACDC http://gthhabbcomm////pSSSSsaaa
STR http://gittua.com//AAWWWaaaA/
GMP http://ghhhbbbcom//AAAWWWLaabt
DNW http://gttabbccom/AAllvvLo/////
21


Reviews:
[{"review_id": "jqLtG6i6Ca", "reviewer": "Reviewer_gR7e", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This manuscript studies the problem of network pruning for efficiency. The central technical point is a heuristic function (equation.1-3) that assigns a weight-wise soft mask (this is called attention) to each weight during training. The mask value is related to the target sparsity. Combined with a tailored training algorithm that gradually increases the sparsity ratio, the method is shown to out-perform recent pruning state-of-the-art methods. Code is not promised or provided.", "summary_of_the_review": "Clear technical description (although heuristic), good benchmark results but substantial presentation issues (maybe only to me).", "strength_and_weaknesses": "Strengths:\n+ This method is technically clear.\n+ The performance on standard benchmarks are quite good。\n\nWeaknesses:\n- I think the major issue is presentation. As far as I can see, the method is not iterative, not differentiable and not attention-based. I don't understand what the title means. The presentation makes me so confused and doubt whether I truly understand the technical part. \n- By iterative, I thought there are some multi-stage residual formulations like [B] but it turns out not. What does 'iterative' mean in this method? According to algorithm.2, I can only see some standard SGD iterations. If so, shall we call every method 'iterative'? \n- By differentiable, I though there are some tricks that allow us to differentiate through some non-differentiable operators but it turns out not. Equations 1-3 are used in a non-differentiable manner. Which part is made 'differentiable'?\n- By attention, I though there some learnable attention masks but it turns out that the heuristically assigned mask values are named attention. Of course the authors have the freedom to call it attention. But this is the first time I see the term 'attention' used in this way.\n- Finally, since the motivation is to allow dead weights to be non-zero again, an important reference [A] that addresses this old problem is missing.\n\n[A] Dynamic network surgery for efficient dnns, NeurIPS 2016\n[B] Network sketching: Exploiting binary structure in deep cnns, CVPR 2017", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "iHrxfG4YL7o", "reviewer": "Reviewer_XNYa", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The main contribution of the paper is to devise an iterative differentiable and parameter-free pruning algorithm utilizing attention-based soft pruning masks. They show that the iterative learnable pruning results in improved performance on vision and NLP tasks.  ", "summary_of_the_review": "This is a well-motivated and well-written paper. It clearly explains the issues with previous methods, and how they addresses some of those. ", "strength_and_weaknesses": "The differentiable pruning utilizing soft attention masks results into simplicity and efficiency of training. There is no overhead of auxiliary optimization or additional parameters. As opposed to learning a pruning threshold which is difficult for controlling the sparsity, the proposed method IDP provides higher flexibility and recovery from unwanted pruning in early stages. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RhD6V_bpIA", "reviewer": "Reviewer_Dmgj", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper suggested two major drawbacks on pruning algorithms:\n\n- Difficulties in controlling the sparsity level\n- Weights that are pruned away at an early stage do not have a chance to recover\n\nThe proposed scheme defines two symbolic states ‘to-prune’ and ‘not-to-prune’, then they generate a soft-mask for values $w$ that sits at the pruning boundaries.", "summary_of_the_review": "Although I have mentioned about how the evaluation using MACs is not ideal and soft-mask bsed pruning is an existing idea, I do think the proposed symbolic states and its corresponding softmax formulation presents an interesting angle to this pruning problem. So I vote for an accept for this paper.", "strength_and_weaknesses": "Strengths\n\n- This paper is well-written with many illustrative plots.\n- The design choices are well justified, and the authors have made a detailed comparison to various state-of-the-art fine-grained pruning algorithms.\n\nWeaknesses\n\n- The reported MAC savings are very theoretical, with this pruning granularity, I do not think they transfer directly to run-time performance gains.\n- Using a soft-mask for pruning is an existing idea.  However, this paper did perform a nice  reasoning for the usage of a softmax function on these two symbolic states , the proposed method is still an interesting one.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RGaTaNXf-3", "reviewer": "Reviewer_qrMF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This post introduces a differentiable pruning algorithm.\n\nThe major contributions include:\n+ A differentiable and parameter-free pruning algorithm based on attention.\n+ Efficiently pruning to offer a high-quality model for a given pruning target.\n+ The state-of-the-art results on both computer vision and natural language tasks.", "summary_of_the_review": "Overall, this article introduces a pruning method that can achieve better performance. But I think this method lacks a certain innovation (refer to the Weakness section). There is still a certain distance from a paper that can be received.", "strength_and_weaknesses": "## Strengths\n1. The article is easy to understand.\n2. The method is simple and effective and has achieved good performance.\n\n## Weaknesses\n### Content\n1. In essence, the core contribution of this article is how to choose a threshold t. After the threshold t is obtained, the rest is to use t to generate a mask and then apply the mask to the weight for e2e training. This is actually a very common idea. For example, in the field of NAS, DARTS uses a similar idea. As for gradually adjusting the pruning ratio r, in the field of Detection, it is actually warm up. So these two points are technically feasible, but they lack innovation.\n2. I think an experimental result with a low pruning rate r needs to be provided. In this way, the experimental results are complete, proving that this method can work (or at least not deteriorate) under various pruning rates r.\n\n### Writing\n1. z(w) is missing a label.\n2. Symbol |.| has a different meaning in z(w) and equation 1.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject