PDP: Parameter-free Differentiable Pruning is All You Need
Authors: Minsik Cho
Saurabh Adya
Devang Naik
Venue: NeurIPS 2023 poster (2023)
Keywords: pruning
cnn
transformers
TL;DR: Existing methods might be too complex and require too many extra parameters. Our simple and efficient method can yield the state-of-the-art random/structured/channel pruning results.

Abstract: DNN pruning is a popular way to reduce the size of a model, improve the inference
latency, and minimize the power consumption on DNN accelerators. However,
existing approaches might be too complex, expensive or ineffective to apply to
a variety of vision/language tasks, DNN architectures and to honor structured
pruning constraints. In this paper, we propose an efficient yet effective train-time
pruning scheme, Parameter-free Differentiable Pruning (PDP), which offers state-
of-the-art qualities in model size, accuracy, and training cost. PDP uses a dynamic
function of weights during training to generate soft pruning masks for the weights
in a parameter-free manner for a given pruning target. While differentiable, the
simplicity and efficiency of PDP make it universal enough to deliver state-of-the-art
random/structured/channel pruning results on various vision and natural language
tasks. For example, for MobileNet-v1, PDP can achieve 68.2% top-1 ImageNet1k
accuracy at 86.6% sparsity, which is 1.7% higher accuracy than those from the
state-of-the-art algorithms. Also, PDP yields over 83.1% accuracy on Multi-Genre
Natural Language Inference with 90% sparsity for BERT, while the next best from
the existing techniques shows 81.5% accuracy. In addition, PDP can be applied to
structured pruning, such as N:M pruning and channel pruning. For 1:4 structured
pruning of ResNet18, PDP improved the top-1 ImageNet1k accuracy by over 3.6%
over the state-of-the-art. For channel pruning of ResNet50, PDP reduced the top-1
ImageNet1k accuracy by 0.6% from the state-of-the-art.

PDP: Parameter-free Differentiable Pruning is All You Need
Minsik Cho
Saurabh Adya
Devang Naik
Apple Inc.
{minsik, sadya, naik naik. balk.))@appee.om
Abstract
DNN pruning is a popular way to reduce the size of model, improve the inference latency, and minimize the power consumption on DNN accelerators. However, existing approaches might be too complex, expensive or ineffective to apply a variety of vision/language tasks, DNN architectures and to honor structured pruning constraints. In this paper, we propose an efficient yet effective train-time pruning scheme, Parameter-free Differentiable Pruning (PDP), which offers state- of-the-art qualities model size, accuracy, and training cost. PDP uses a dynamic function of weights during training to generate soft pruning masks for the weights in a parameter- free manner for a given pruning target. While differentiable, the simplicity and efficiency of PDP make it universal enough to deliver state-of satte-of-the-arr theherrt sundom//trructurddcchamee pruning results on various vision and natural language tasks. For example, for MobileNet-vl, PDP can achieve 68.29 top-1 ImageNet accuracy at 86.6% sparsity, which is 1.7% higher accuracy than those from the state-of-the-art algorithms. Also, PDP yields over 83.1 % accuracy
Multi-Genre Natural Language Inference with 90% sparsity for BERT, while the next best from the existing techniques shows 81.5% accuracy. In addition, PDP can be applied structured pruning, such as N:M pruning and channel pruning. For 1:4 structured pruning of ResNet18. 8, PDP improved the top-1 ImageNetlk accuracy by over 3.6% over the state-of fate--of-thearr the-art. For channel pruning of ResNet50, PDP reduced the top-1 ImageNet accuracy by 0.6% from the state-of-the-an
1 Introduction
Deep neural networks (DNN) have shown human performance on complex cognitive tasks [46]. but their deployment onto mobile/edge devices for enhanced user experience (i.e., reduced latency and improved privacy) is still challenging. Most such on-device DNN systems are powwedd and heavily resource-constrained thus requiring high power/computt/storage efficiency
50 53
Such efficiency can be accomplished by mixing and matching various techniques, such as designing efficient DNN architectures like MobileNetMMobileiiTi MobileOne [36] 43 48). distilling a complex
DNN into a smaller architecture [41, quantizing/coppresing the weights of DNNs [8] [8] 23 31
39
58), and pruning near-zero weights 28
34
40
44
52
56
57 61]. Also, pruning is known be
highly complementary to quantization/comprssti
[1 when optimizing DNN model. Training
larger model and then compressing by pruning has been shown be more effective in terms of
model accuracy than training smaller model from the beginning (32] However, pruning comes at
the cost of degraded model accuracy, and the trade-off is not straightforward 28
Hence, desirable pruning algorithm should achieve high accuracy and accelerate inference for various types of networks without significant training overheads in costs and complexity. In this
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

STR" FTWT
CS
GraNer
OptG
ACDC
MVPs
POPA"
PDP
Conv-Net Accuracy
V
?
Transformer Accuracy
?
?
V
VV
Inference speed
4
/
?
2
Training speed
?
14
Training stability
//
NVV
//1
?
Training flow
simple complex complex complex complex complex complex complex simple
Extra parameters few many
many none
many
none
many
none
none
Soft threshold weight re-paramcterization for learnable sparsity (28
Fire Together Wire Together: A Dynamic Pruning Approach with Self-Supervised Mask Prediction
Winning the Lottery with Continuous Sparsification 45
Sparse training via boosting pruning plasticity with neuro-regeneration [4]
Optimizing gradient-driven criteria in network sparsity: Gradient is all you need 57
AC/DC: alternating conprresedddeccomrressse training of deep neural networks 40
Movement Pruning: Adaptive Sparsity by Fine-Tuning 44
Prune Once for All: Sparse Pre-Trained Language Models (56
Table Comparison of the state-of-the-art pruning schemes and PDP: PDP can explore good trade-off
bewteen accuracy and inference speed without introducing new learnable parameters and with a simple/fast
training flow.
work, we propose a simple yet effective pruning technique, Parameter-free Differentiable Pruning or PDP, which uses a dynamic function of weights to generate soft pruning masks for the weights themselves. PDP requires neither additional learning parameters [57] nor complicated training flows [40] yet offers precise control on the target sparsity level [28]. while pushing the state-of-the-
arts in
pruning. The soft masks from PDP make pruning differentiable
and let training loss decide whether/how each weight will be pruned 42 Table [ compares PDP
with the latest state-of-ther of-the-art pruning schemes, and our major contributions include:
PDP outperforms the state-of-the-ar schemes on variety of models and tasks. Being differen- tiable and parameter-free, PDP offers efficient and effective pruning without complex techniques.
PDP offers universal and holistic approach for efficient random//sruccturd/cchmneel pruning, while delivering a high-quality model optimization for a given pruning target.
With dynamic function of weights, PDP generates soft pruning mask which does not need training, and thus does require neither gradient synchronization nor SGD-update.
2 Related Works
Pruning in DNN incurs a complex trade-off between model accuracy and inference speed in terms of MAC (mult-add operations) (28). A weight can contribute differently to the model accuracy, depending on the number of times it is used for prediction (i.c., a weight in convolution filter for a large input) and the criticality of the layer it belongs (i.c., weight in a bottleneck layer). Therefore, even two models are pruned to the same level, the accuracy and inference speed of each can be vastly different, which makes exploring the best trade-off challenging yet crucial in DNN pruning. body of work in random and structured pruning has been proposed to optimize such a trade-off. Please Fig. g and Section B in Appendix for additional reviews and details.
Differentiable techniques gained popularity due to the advances in network architecture search methods as well as network compression techniques 6 g 33, and is a powerful technique for random/structured pruning as drives pruning based on task loss. However, diffcrentiable pruning does not necessarily deliver the best quality pruming (i.e., many state-of-the-ar results are from non-differentiable methods), because the computational overhead and training complexity can limit its benefit. Existing differentiable pruning methods can be classified into few approaches.
Learning pruning budget allocation determine target sparsity for each layer in a differentiable way, rather than computing pruning masks. Optimizing and training a per-layer pruning threshold with ReLU based on dynamic ce-parameterization was proposed to allocate the sparsity across all layers in STR (28) which shows good model accuracy and low inference overhead. DSA [8 finds the layer-wise pruning ratios a differentiable fashion by computing a channel-wise keep probability drawn from the Bernoulli distribution and keeping the expected sparsity ratio satisfied.

---

0.18
0.19
0.19
0.21
0.18
0.199
0.19
0.20
0.21
0:22
0.25
0,30
M4
Pia
W>
Wigg
W/s
Wig
threshold: 0,205
00.005
0.25
0:25
0.51
0.85
0.90
Triining
7lg
114g
1lly
Traniing
710
"02
Weight Update
Weight Update
0.18
0.19
0.19
0.20
0.19
0:21
0.23
0.31
0:18
0.20
0.22
0.22
0.19
0.21
0.23
0.31
M/s
Min
Mio
W2
W2
Ws
Mip
00.115
threshold:: 0.202
0.19
0:45
0:51
0.51
0.24
0.49
0.58
0,90
/
m4
1
Tratning Completed
Training Completed
Teat
0.21
0.23
0.31
0
0.22
0.22
0:25
0.31
W0
W1
16
W4
Hig
(a) Using trainable threshold
(b) Using soft masks with (see Section
3.1
Figure 1: Unlike learning threshold parameter for differentiable pruning 28 33 in (a), PDP generates soft masks without extra trainable parameters to accomplish differentiable pruning as in (b). The moment a weight is pruned in (a). the weight will not get any update as the mask zeros out the gradient, depriving the opportunity for better model training. However, the scheme in (a) ensures the training and test behaviors identical, yet makes the sparsity control difficult. PDP generates such soft masks without extra parameter/training overheads unlike existing schemes [34] 40
(see Section [.1]). While PDP reveals different behaviors during training/test times because the test needs hard masks, our soft mask allows weight to recover from being pruned over time, which yields higher model accuracies efficiently.
45 57)
Learning pruning mask as an extra parameter is popular way for differentiable pruning. In CS [45], new trainable mask parameter is learned to continuously remove sub-networks based on the lottery hypothesis [3]. In detail, the new mask parameter is first multiplied by scheduled temperature parameter, and then LO-regularized inside the Sigmoid function to continuously increase the sparsity. OptG [57] proposed learning mask parameter based the loss changes due pruning a particular weight, which is proportional to the gradient and the magnitude the corresponding masked weight when approximated with STE (5). AutoPrune [54] also introduces extra learnable parameters (controlled by regularizer) to generate a diffferentiabeeeppooimmdee pruning mask using STE (5). In [42, new learnable parameters, a introduced and L,, reguularre enforce sparsity. Then, a is compared with a hyper-param to derive the masks for the model parameters (i.e., prune if a t). As such comparison is not differentiable, comparison is approximated into foothill function [4) Learning channel pruning mask is proposed in SCP (25) based on the operation-speciffe observation that feature map with large and negative batch mean will get zeroed by ReLU.
Generating pruning masks is proposed FTWT 11] where new layers are attached to the exiting main architecture. The attached new layers are fed with activations from the main network, then trained to generate masks for the weights in the main network. Thus, although does not learn masks directly, it still requires new trainable parameters to learn how generate masks, not mention the changes to the model network. Using knowledge distillation to generate pruning masks is explored in NISP (55) and DCP (62), where fully trained teacher network guides the mask generation in way that the distortion to the layer responses can be minimized. A differentiable Markov process is studied to generate channel pruning mask probabilistically, where state represents channel and the transitions from states account for the probability of keeping one channel unpruned [16]
3 PDP: Parameter-free Differentiable Pruning
Complex pruning schemes do not always yield the best quality results, and their complexity and cost can make them impractical and difficult to use. The proposed Parameter-fre Differentiable Pruning (PDP) is a highly effective and efficient scheme that generates soft pruning masks using a dynamic function of weights in parameter-free free and differentiable fashion. Since PDP is differentiable, the task loss can directly guide the pruning decision, offering an effective pruning solution. Simultaneously, being faremeter-free, PDP can be fast and less intrusive to the existing training flow. Overall, PDP finds a weight distribution that is best for task loss and pruning. Instead of having extra parameters, PDP indirectly influences the weight distribution for high-quality pruning. For example, if a weight
3

---

Gradient Aggregation and Weight Update
HEW)
19 WLW)
W/
1/A
W to (n())
Cupture Weight
Distribution
W bottomk(abo(WW).> m(())
1.0
(.5 ()++++++ maxt W))
Probahility
0.7
0.5
mn()
w
0.2
z(W)
Forward/Backward (dw))m(w))
softmaxt
with Masked Weight
0.0
m(w)
0.2
0.4
0.6
0.8
(a) PDP training flow
(b) Probability plot by w E Rio.a] with 0.6
Figure Computing z(w), m(w) for the chances for Z and M with t for the equal chance to be in Z and M.
w is destined to be pruned for some reason, instead of having new parameter to denote "to-prune", PDP lets SGD gradually make itself smaller relatively against other parameters in the same entity, increasing its chance to be pruned over time. We will first present PDP in Section provide in-depth analysis Section 3.2, discuss the benefits of PDP over existing differentiable pruning approaches in Section 3.3
and then show the extension to structured and channel pruning in Section 3.4
3.1 PDP Algorithm
To address the drawbacks of existing differentiable pruning algorithms, we propose PDP. soft mask should idcally represent the chance of a weight w being in one of two symbolic states, "to-prune" (noted Z) "not-to-prune" (noted M), without requiring extra parameters or expensive book- keeping. While the chance of w being in either state is not straightforward to compute, PDP generates a soft mask based on the fact that there exists an equal chance point for both states. Let us consider differentiable functions, 2, R[0, co] + R 1). to compute the chances of being in Z and M, respectively, which must satisfy the following conditions as a soft mask for magnitude-
pruning:
2(la]) < 2((b)) for |a| |b]: weight with smaller magnitude has higher chance to be in Z.
m(la)) > mlb]) for |a] > |b|: weight with larger magnitude has higher chance to be in M.
2(w) + m(w) for any w: the total probability is 1.
Then, by the monotonicity and continuity, there exists t E R~o such that at) m(() 0.5 (the equal chance for Z and M), which leads to the following boundary conditions on the left. Any function that satisfies these conditions can be used to compute mnw) as a soft mask of w for train-time pruning. Let denote that lopK(X (X, k)/ (bolooo (X, is selecting the largest/smallest elements from matrix X, abs(X) is an element-wise absolute operation, and n X') returns the element count. In PDP, we uniquely identify/ / for given prune ratio [0, for layer with a weight matrix W as in Fig. 2 (a) based on the pruning context.
if w
=(w)=
if | Do if w =t
The sparsity for each W can be easily obtained by sorting the weights from the network per magnitude after a few epochs w.r.t the global target sparsity as a one-time task, by pruning manually by user.
budget allocation algorithm [4 28
Right after the SGD weight update, t is computed for the weights W in each layer entity. The role of t is to abstract the current weight distribution of each layer/entity for pruning.
During forward-past paard, a soft mask, m(w) for the weight w is computed and then the masked weight á is applied. is the temperature parameter (see Section E in Appendix for details).
Computing t and generating áé repeat iteratively to adapt to the updated weight distribution.
Figure shows how the value of is obtained in PDP and soft mask is computed. Specifically, t set to the value that halfway between the largest pruned weight and the smallest unpruned weight when a hard mask is applied for given sparsity ratio r. This ensures that each weight has an equal probability of being pruned or kept. result, PDP satisfies all the constraints for and m. More details on PDP training flow are in Section D in Appendix.

---

PDP uses a dynamic function of t to generate soft pruning masks of W without the need for any extra trainable parameters. Instead, PDP lets the weights of the network adjust themselves such that the information that would otherwise be learned by the extra trainable parameters is instead fused into the weights themselves and their distribution. This is possible because each weight w not only a coefficient in layer, but also an indicator of the relative chance of that weight being pruned against the other weights in W. This relative chance determined by the value of t, as shown in ( Overall, PDP looks simple but is shown to be quite effective, which opens to broad applicability.
3.2 PDP Gradient Analysis
To better understand how PDP helps pruning decision, we derive the gradient of the masked weight τ. As in Fig. 2 (a), d is the following:
?
i m(w) w + e
(1)
2
Then, the gradient of w, Aw can be simplified as the next:
Aw = m(w) Au - m(w) mm(w)
(2)
where we can make the following observations:
The 1st term typical gradient in mask-based pruning, and the 2nd term is an additional gradient with a positive factor (i.e, _² mm()) - m(w)} 2 0) from PDP.
If m(w) Rs 0 or hard-prune, Aw ss O, which is true for any pruning algorithm.
If mmw) Rs 1 or hard-not-to-prune. Aw R Au, which is true for any pruning algorithm.
When m (w) x 0.5 (i.c., pruning decision is unclear), my -1-mo ( is maximized and accordingly Aw is too, boosting the movement.
serves inverse scaling factor for the boosted gradient.
Hence, PDP will accelerate the SGD updates for the weights near the pruning boundary (i.e., f) toward loss-decreasing direction, which can encourage the weights to settle with the proper pruning decision at the end. Even if the current gradient is not globally beneficial for the task, many second chances will eventually help recover the damages in an accelerated manner. We conjecture these features enable PDP to make a better pruning decision for the weight on the boundary.
3.3 PDP vs. Existing Differentiable Pruning Strategies
Learning Pruning Budget Allocation focuses on obtaining a pruning threshold in differentiable way based on marginal loss or L--regularization 25 38) Meanwhile, PDP directly generates a soft pruning mask for each weight also a differentiable way. Fig. Iillustaates the differences with an example. Learning a pruning threshold is helpul for global pruning budget allocation, as the threshold gets adjusted per the task loss as in Fig. (a), but has the following drawbacks:
Sparsity level is hard control, the pruning threshold is not explicitly related to the sparsity, as masks and the thresholds are not directly co-optimized. In Fig. | (a), after one weight update, the threshold is reduced from 0.205 to 0.202, but the sparsity is increased from 50% to 66% w4 becomes smaller than the threshold.
Once a weight gets pruned during training, it does not get updated as the gradient masked out. The bottom weights 1,233, get weight update due to the zero masks, and remain pruned.
On the contrary, PDP allows all the weights to be updated through soft masks as in Fig. D (b). providing higher flexibility and recovery from undesirable pruning [17. For example, consider the weight Wy of a value 0.19 which would have been permanently pruned in (a). PDP discourages weight from being permanently pruned through the weight update, w2 still receives scaled-down gradient. If near-zero weight continues to get negative gradients over time (although scaled down by the soft mask), it can eventually get unpruned the end of training. Similarly, if very large weight gets positive gradients many times enough to be near zero, it will get pruned eventually, even
5

---

PDP: epoch 20
(b) PDP: epoch 30
(c) PDP: epoch
(d) PDP: epoch
(c) PDP: epoch 60
PDP: PDPccpoch 70
gg) PDP: epoch 80
(h) PDP: epoch 90
STR: epoch 20
STR: epoch 30
(k) STR: epoch 40
() STR: epoch 50,70,80,90
Figure 3: The effects of PDP and STR (28) (from Table I for the first 3x3 Conv2d 2d layer with 32 filters in MobileNet-vl on ImageNetIk where cach small rectangle represents one 3x3 kernel: For PDP, during the entire end-2-end training, temporarily round the soft mask value to make the pruning decisions. The white cell indicates such prurning decision for corresponding weight has been flipped least once during the particular epoch, and black cell indicates other case.
it was not pruned in the early stage. Hence, such gradual pruning decision over time during training allows PDP make better pruning decisions w.r.t. the task loss.
Such soft masking allows pruning decisions to be flipped multiple times during the entire training process, and such effects are compared between PDP and STR (28) (from Table [T in Fig. 3 where the pruning decisions for the first Conv2d (3x3 kernels with output channels) in MobileNet-v for the end-to-end ImageNet training are captured. The white cell means that the pruning decision has been flipped (from to-prune to not-to-prume or vice-versa), while the black cell means the decision has never changed, during the given epoch. Then, can observe from Fig. 3 that while PDP in (a)-(h) continues to flip the decisions until the very late training stage, STR in (i)-(1) finalizes the pruning decisions early (i.e., in the first 30 epochs).
learning/Generating Pruning Masks with Extra Parameters allows the pruning decision be driven by a task loss through back- propagation rather than the weight value itself (i.e., hard mask will zero out the gradient of a pruned weight), thus addressing problems with differentiable pruning budget allocation, but comes with the following issues.
A pruning mask is (or derives from) learnable parameter. Hence the number of total trainable parameters increases significantly, making the training process slow and complex [[] 45 57
A hard mask is approximated into soft mask using differentiable function, without guaranteeing the key properties of a pruning mask, such as the [0, value range or monotonicity
Table compares parameter-free PDP with differentiable mask pruning scheme, CS 45 from Table (see Table 8 Appendix for detail). The results show that PDP outperforms CS and is 2.8 faster, without adding extra trainable parameters or complicating the training flow. Table 3 also demonstrates being parameter-free can provide the substantial training efficiency gains for very large language models like GPT. When trained and sparsified on 32 GPUs in mixed-precision (to fit OptG into GPU memory) using the recipe lower cost than OptG [57].
II. PDP delivers the best perplexity 75% sparsity with 42%

---

Top-1 (%)
Sparsity (%)
Model
Extra
Runtime
Inference
Method
#param
#param
min
MAC(xe6)
Dense
91.4
0
273k
0
22.1
41.0
CS
89.1
86.3
273k
267k
84.7
5.80
PDP
90.4
86.3
273k
0
30.3
5.79
Table Compared with CS ResNeC00CIFAR10, PDP delivers 2.7x speed up due no extra parameters.
Model #param
Extra #param
GPU cost(S)
Method
Perplexity
Dense
22.4
163M
GMP [T
37.7
163M
6997
OptG
33.7
163M
124M
11210
PDP
33.7
163M
7499
Table 3: With 75% sparsity, being parameter-free tereeetteerreeeeeeee becomes more important for training GPT2 with Open Tex
Weights
0.18
0.20
0.01
0.10
0.07
0.32
0.09
0.12
Channels
Norms
0.35
Masks
PDP
PDP
0.18
0.20
0.01
0.10
0.29
00.29,011. 0.19
0.34
W
[0.18,0.20.0.10]
W [0.32.0.1..0009]
0.15
0.07
0.09
0.02
0.19
PDP
0.19
1:4 masks
W,
[0.01]
0.06
w,= [0.07]
0.08
0.02
0.52
0.39
0.01
0.65
0.81
0,58
0.68
0.1
0.21
0.13
0.88
0.15
0.28
0.22
0.12
0.19
0.26
0.41
0.66
(a)
1:4 pruning (75% sparsity)
(b) Channel pruning (50% sparsity)
Figure PDP simple and universal enough applied directly to structured and channel pruning.
3.4 PDP for Structured and Channel Pruning
The simplicity and non-intrusive nature of PDP make it readily applicable to differentiable structured and channel pruning. As an example of structured pruning, we consider N:M pruning. where only N weights are kept out of every M consecutive weights. N:M pruning is attracting high research and industrial attention because top-of-the-ling GPUs support 2:4 configuration [24). To apply PDP to N:M pruning, apply it every M consecutive weights of the layer, as if the layer were composed of many sub-layers, each with M weights, which is illustrated in Figure target sparsity, we can easily find the threshold t and generate the soft mask, as shown in Figure (a).
(a). Since N:M dictates the
Channel pruning is another type of structured pruming that can be easily applied to PDP with minor modifications. To do this, first compute the L2 norm of each channel in the layer, and then use these norm values (in place of the absolute values of the weights in Figure soft mask for each channel, which is depicted in Figure 4 (b). Using the soft mask to prune all the corresponding weights in the channel will make the channel pruning process differentiable.
to compute a
4 Experimental Results
We compared our PDP with sfate-of-the-ar random, structured, and channel pruning schemes on various computer vision and natural language models. We used two x86 Linux nodes with eight NVIDIA V100 GPUs each in a cloud environment. All cases were trained from scratch. More experimental results and the hyper-parametes are Section and Table Appendix.
Random Pruning for Vision Benchmark: We compared the proposed PDP with the latest prior arts. STR 28). GMP (61, DNW [52]. GraNet (34). OptG [57], and ACDC 40 on R8sNet18.
ResNet50, MobileNet-vil I, and MobileNet-v2 (20) 22 43 with the ImageNetlk dataset [9]. Since all of these schemes have been experimented only with ResNet50 and/or MobileNet-vl. we reproduced the pruning results in our controlled environment with the identical data augmentations by running the official implementations from the authors 28 34 40 [57] or verified implementations from the
prior arts [52 61 as in Section G in Appendix. Since the primary goal of pruning is to trade-off the model accuracy with the compute reduction as Section 2 we measured the accuracies and Multiply-Accumulate Operation (MAC) during inference on each experiment with layer fusion (i.e., BatchNorm folding), and mainly focused on the high-sparsiffiction cases. Note that the MAC purely theoretical and reported understand the trade-off among accuracy, size, and compute across
7

---

550
880
490
GraNet
ACIC
Inferenee MAC (e6)
800
GraNet
ACIC
Inferernce MAC
460
720
PDI--aase"
400
640
PDF-basc+
-TRR
560
340
PDP-basee
O
480
280
400
STR
250
OptG
PDP-oppt
OptG
PDP-appg
320
74%
75%
Top-1 Accuracy
Top-1 Accuracy
(a) ResNet18 (sparsity: 85.5%)
(b) ResNer50 (sparsity: 89.8%
150
Inference MAC (e6)
140
130
GraNet
ACDC
Inference MAC (e6)
GraNee
120
100
130
PDP-sptt
OptG
PDP -base
90
Opt
PDP--ppi
70
PDP-base
60
STR
50
STR
40
61%
63%
65%
67%
69%
36%
60%
64%
68%
Top-1 Accuracy
Top-1 Accuracy
(c) MobileNet-vl (sparsity: 86.6%)
(d) MobileNee-v2 (sparsity: 80.2%)
Figure 5: PDP- PDP-powered pruning (in white box markers) delivers the Pareto superiority to the other schemes (i.e.. the top-bottom corner is best trade-off) ResNet18. ResNet50, and MobileNet-v1//2 on ImageNetIk. The size of markers indicates the relative training overheads. The detailed numbers are in Table
various algorithms, but it still captures the latency benefit on some hardware platforms (see C in Appendix for details). In our experiments with ImageNetIk, all layers, including both the first and last layers, are pruned without any restriction. Also, we estimated the algorithmic overhead and training efficiency by the total monetary cost for GPUs on commercial cloud spot instances [
We applied not only the proposed hyper-parameters parameters from the authors but also a set of further fine- tuned dlyee--paramede each algorithm used a different number of epochs and showed results at different sparsity levels, a) the target sparsity levels for all the networks for fair comparisons, because all
for competing methods (see Table in Appendix for details). Also, since
STR first other schemes can control the sparsity level precisely, b) we trained ResNet 8/50 for 100 epochs and Nebileeet--11/2 for 180 epochs following (28 more epochs for MobileNet-v1/v2) For PDP, we fixed the target sparsity for each layer based on the global weight magnitude the epoch 16 and started pruning at the of 1.5% of the target sparsity per epoch for all the experiments which correspond to 16 and 0.015 in Algorithm ] in Appendix. For detailed experiment configurations, please refer to Table Every experiment began with a randomly initialized model (i.e., no pre-trained model). For PDP. we had the following variants show the value of PDP with the same training overhead or per-layer pruning budgets.
34
40 57] except STR (which diverged with
PDP-base globally computes the target sparsity by abs(W) at epoch 16 across all the layers.
PDP-base+ is the same as PDP-base yet with more epochs to match the GPU cost of ACDC.
PDP-str/optg uses the per-layer sparsity from STR/OptG input to normalize the MAC.
Our experimental results are highlighted in Fig. 5 where the size of circles indicates the relative training overhead due to pruning. Note that we used only one single node with 8 GPUs due the limitation in the official implementations for GraNet and OptG, thus both have the advantage of not having the inter-node communication cost. Also, each approach imposes different level of training-time time overhead, mainly due to the various complexities of training flow and pruning itself as captured Fig. Overall results can be summarized follows:
PDP showed the best the model accuracy: PDP-base on ResNet18 delivered 69% Top-1 accuracy which is superior to other schemes but at higher MAC than only STR and OptG.

---

Network Sparsity
Top-1
GPU
MAC
Network Sparsity
Top-1
GPU
MAC
Method
Method
(%)
cost(S)
(Xe6)
(%)
cost ($)
(Xe6)
Dense
69.8
167
1814.1
Dense
76.1
248
4089.2
GMP
65.2
217
263.5
GMP
73.6
483
419.0
DNW
64.4
206
263.5
DNW
70.7
466
419.0
GraNet
66.0
198
539.6
GraNet"
72.5
321
868.0
ResNet18 85.5%
STR
66.7
171
334.6
ResNer50 89.8%
STR
72.8
417
373.7
OptG*
65.5
277
223.7
OptG*
72.1
591
333.0
ACDC
68.7
356
502.8
ACDC
74.7
635
735.6
PDP-base
69.0
169
408.6
PDP-base
74.7
325
502.8
PDP-base+
69.5
336
405.1
PDP-base+
75.3
610
483.0
PDP-str
68.6
169
334.7
PDP-str
74.0
325
373.7
PDP-optg
68.5
174
223.7
PDP-optg
74.2
332
332.9
Dense
70.9
277
568.7
Dense
71.9
297
300.8
GraNet
61.4
367
145.7
GraNet"
56.3
439
103.4
MobileNet-vl
STR
61.7
176
47.2
MobileNetv-2
STR
60.0
285
40.6
86.6%
OptG*
66.3
340
87.4
80.2%
OptG*
65.4
545
76.8
ACDC
66.5
641
124.5
ACDC
64.1
812
93.9
PDP-base
68.2
281
88.3
PDP-base
66.8
354
95.3
PDP-str
65.3
307
47.2
PDP-str
60.7
307
40.6
PDP-optg
68.2
297
87.3
PDP-optg
66.5
343
76.6
the GPU cost (S) is based commercial cloud spot instance pricing.
used only one with 8 GPUs due to the limitations the public code.
o when PDP applied for MobileNet-v3 (5.5M parameters) with 80% sparsity target, achieved
71.5 11.5% top-1 I accuracy, which is 2.5% down from the dense version.
Table 4: PDP compared with other unstructured pruning algorithms ImageNet1 K shows the best trade-off
among accuracy, inference MAC, and training overhcads. More results are available Section in Appendix.
PDP offered the better model accuracy for given pruning target: With the custom sparsification target for each layer, PDP-str/optg demonstrated the 2-3% higher Top-1 accuracy at the same MAC, demonstrating the effectiveness of the proposed method.
When we use the similar GPU budget for additional epochs with ACDC which is noted as PDP-base+, our method further improved the Top-1 accuracy from 69% to 69.5% for ResNet18 and from 74.7% to 75.3% for ResNet50 with slight fewer MACs.
Random Pruning for NLP Benchmark: We compared PDP with the state-of-the-arr pruning results from MVP 44 and POFA [56] (quoted from the respective papers) in addition to OptG and STR (reproduced in our environment) on a BERT model [10) for the two largest NLP tasks of the GLUE benchmark, MNLI (Multi- Gente Natural Language Inference) with 392,702 samples and Natural Language Inference) with 363,836 samples. Following the setups in MVP and POFA, we used the excluded the embedding and the last linear layer from pruning, and trained the bert--bas-uncased from HuggingFace from scratch with self-distillation. The teacher for the distillation is from the best checkpoint in the first 40 epochs. We use the weights of 0.95 on the distillation loss and 0.05 on the task loss for PDP. and 0.75 on the distillation loss and 0.25 on the task loss for STR. We could not use distillation for OptG, as it caused the out-of-memory error due to the extra- extra-parameter overheads from both pruning and distillation. Our experimental results in Table can be summarized as follows:
QQP (Question-answerii
batch size 32 per GPU (i.e., global mini-batch size of 512),
STR underperforms on the best cases even though it could not achieve the target sparsity.
OptG shows better model accuracy only than GMP and worse than others in setup.
PDP outperformed all other methods for both MNLI and QQP.
Struetured/Channel Pruning for Vision Benchmarks: We compared PDP-driven N:M pruning and channel pruning on the ImageNetlk dataset 99. For N:M pruning, we reproduced the LNM 59 results using the public code base but without color augmentation environment normalized. For PDP, we simply reused the hyper-parameters and configurations as in Table our Appendix, and the top-1 accuracies by various N:M configs with ImageNet on ResNet 8/50 are presented in Table 6 We can observe that PDP outperforms LNM all the test
keep the experimental

---

Methods
Benchmark
Metric
Dense
STR
OptC
GMP
MVP
POFA
PDP
MNLI
90
75.8
78.5
n/a
81.2
81.5
83.1
84.5
matched
94
74.4
76.9
74.8
80.7
n/a
82.0
accuracy (%)
MNLI
90
76.3
78.3
n/a
81.8
82.4
83.0
84.9
mismatched
94
74.1
76.5
75.6
81.2
n/a
82.4
accuracy (%)
90
91.2
n/a
89.8
n/a
90.2
90.9
91.0
QQP
f1
90
88.1
n/a
86.2
n/a
86.8
87.7
88.0
Since STR cannot control the sparsity precisely, we report the metrics with our closest achieved sparsity levels, 85.9% for the 90% case and 91.8% for the 94% case.
Table 5: PDP delivers the best accuracies on the BERT-base model with the MNLI (Multi-Genre Natural Language Inference) and QQP (Quora Question Pairs2) tasks of the GLUE benchmark.
N:M
avg GPU
Network
Method
Batch size
#epochs
2:4
4:8
1:4
2:8
cost (%)
LNM
256
120
69.6
70.2
65.1
68.4
395
ResNet18
PDP
1024
100
70.2
70.1
68.7
69.1
275
LNM
256
120
74.6
75.1
74.1
75.0
812
ResNer50
PDP
1024
100
75.9
75.8
75.0
75.3
380
For LNM, we used the the public dense model from Vision for channel pruning, as the original baseline from [59] (which has higher accuracy) is not available publicly. For the example of ResNee550
ImageNetlk, LNM used dense model with 77.3% top-1 accuracy while the public dense model in Torch Vision) yields 76.1% top-1 accuracy.
Table 6: Structured Pruning: PDP superior results than the latest N:M pruning in 55 at 46% less training cost.
directly do N:M pruning due its generality. PDP delivers the
Method NISP ?
Batch size
#epochs
Top-1 (%)
MAC drop (%)
90
75.3
44.0
DCP
256
60
75.0
55.0
SCP
256
100
75.3
54.3
PDP
1024
100
75.9
54.9
SCP, DCP, and NISP reported only ResNet50 results with MAC drop instead of sparsity. Hence, for PDP, we report the nearest MAC drop we obtained (44 at 57% channel sparsity.
Table 7: Channel Pruning: the generality of PDP helps deliver the state-of-the-ar results without modifications.
cases, with 4x larger batch size in 20 fewer epochs. LNM training cost is also much higher than PDP because of its costly weight regularization and complex back-propagation scheme.
For channel pruning, we compared PDP with SCP 26 NISP (55, and DCP [62] Note that SCP uses the BatchNorm select the channels to prune (i.e., beta S c), hence applicable to limited types of networks only. We again reused the hyper-parameters and configurations in Table 8 in our Appendix PDP, and the top-1 accuracy with ImageNetlk on ResNet50 reported in Table 7 We can see that PDP can be used for channel pruning and show superior performance out of the box.
5 Conclusion
We showed that a simple and universal pruning method PDP can yield the tfate-of-the-as ran- domn/structured/channe pruning quality on popular computer vision and natural language models. Our method requires no additional learning parameters, yet keeps the training flow simple and straight- forward, making a practical method for real-world scenarios. We plan extend our differentiable pruning into quantization, making both jointly differentiable and optimizable by the task-loss.
References
[
https:///apple.githhbbb.. 10/ http://hwggg...oooooooooooolaalp html
[3] Anwar, Hwang, and W. Sung. Structured pruning deep convolutional neural networks.
ACM Journal on Emerging Technologies in Computing Systems, 2017.
10

---

[4] M. Belbahri, E. Sari, S. Darabi, and V. P. Nia. Foothill: A quasiconvex regularization for
edge computing of deep neural networks. In F. Karray, A. Campilho, and A. C. H. Yu, editors. International Conference on Image Analysis and Recognition, 2019.
[5] Y. Bengio, N. Léonard, and A. C. Courville. Estimating or propagating gradients through
stochastic neurons for conditional computation. CoRR, 2013.
[6] J. Chang, x. zhang, Y. Guo, G. MENG, S. XIANG, and C. Pan. Data: Differentiable architecture
approximation. Advances in Neural Information Processing Systems, 2019.
[7] B. Chen, T. Dao, K. Liang. J. Yang, Z. Song. A. Rudra, and C. Ré. Pixelated butterfly: Simple and efficient sparse training for neural network models. In International Conference Learning
Representations, 2022.
[8] M. Cho, K. Alizadeh- Vahid, S. Adya, and M. Rastegari. DKM: differentiable k-means clustering layer for neural network compression. In International Conference on Learning Representations.
2022.
[9] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, 2009.
[10] Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, 2019.
[11] S. Elkerdawy, M. Elhoushi, H. Zhang, and N. Ray. Fire together wire together: A dynamic pruning approach with self-supervised mask prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[12] U. Evci, T. Gale, J. Menick, P. S. Castro, and E. Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, 2020.
[13] J. Frankle and M. Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference Learning Representations, 2019.
[14] E. Frantar and D. Alistarh. SPDY: Accurate pruning with speedup guarantees. In International
Conference on Machine Learning, 2022.
[15] T. Gale, E. Elsen, and S. Hooker. The state of sparsity in deep neural networks. CoRR, 2019. [16] S. Guo, Y. Wang. Q. Li, and J. Yan. Dmcp: Differentiable markov channel pruning for neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 2020.
[17] Y. Guo, A. Yao, and Y. Chen. Dynamic network surgery for efficient dnns. In Advances in Neural Information Processing Systems, 2016.
[18] S. Han, H. Mao, and W. J. Dally. Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. In International Conference on Learning Representations, 2016
[19] S. Han, J. Pool, Tran, and W. J. Dally. Learning both weights and connections for efficient neural network. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, 2015.
[20] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.
[21] Y. He, X. Zhang, and J. Sun. Channel pruning for accelerating very deep neural networks. In The IEEE International Conference on Computer Vision, Oct 2017.
[22] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and
H. Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. 170.. 2017.
arXiv preprint
[23] B. H. J. Lee, D. Kim. Network quantization with element-wise gradient scaling. In Proceedings
of the IEEE Conference on Computer Vision and Pattern Recognition, 2021.
[24] C. Y. Jeff Pool. Channel permutations for n:m sparsity. In Advances in Neural Information
Processing Systems, 2021.
[25] M. Kang and B. Han. Operation- -aaree soft channel pruning using differentiable masks. In International Conference on Machine Learning. 2020.
[26] M. Kang and B. Han. Operation-awee aware soft channel pruning using differentiable masks. In International Conference on Machine Learning, 2020.
[27] E. Kurtic, D. Campos, T. Nguyen, E. Frantar, M. Kurtz, B. Fineran, M. Goin, and D. Alistarh.
The optimal bert surgeon: Scalable and accurate second-order pruning for large language models. arXiv preprint arXiv:2203.072:
2022.
[28] A. Kusupati, V. Ramanujan, R. Somani, M. Wortsman, P. Jain, S. Kakade, and A. Farhadi. Soft threshold weight reparameteriizttion for learnable sparsity. In International Conference on Machine Learning, July 2020.
11

---

[29] F. Lagunas, E. Charlaix, V. Sanh, and A. M. Rush. Block pruning for faster transformers. In The Conference on Empirical Methods in Natural Language Processing, 2021.
[30] H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf. Pruning filters for efficient convnets.
In International Conference on Learning Representations, 2017.
[31] Y. Li, X. Dong, and W. Wang. Additive powers-of-twe quantization: An efficient non-uniform
discretization for neural networks. In International Conference on Learning Representations,
2019.
[32] Z. Li, E. Wallace, S. Shen, K. Lin, K. Keutzer, D. Klein, and J. E. Gonzalez. Train large, then compress: Rethinking model size for efficient training and inference of transformers. In International Conference on Machine Learning, 2020.
[33] H. Liu, K. Simonyan, an, and Y. Yang. DARTS: differentiable architecture search. In International Conference on Learning Representations, 2019.
[34] S. Liu, T. Chen, X. Chen, Z. Atashgahi. L. Yin, H. Kou, L. Shen, M. Pechenizkiy, Z. Wang, and D. C. Mocanu. Sparse training via boosting pruning plasticity with neuroregeneration. In Advances in Neural Information Processing Systems, 2021.
[35] Z. Liu, P. N. Whatmough, Y. Zhu, and M. Mattina. S2ta: Exploiting structured sparsity for energy-efficient mobile enn acceleration. In 2022 IEEE International Symposium on High-
Performance Computer Architecture (HPCA). 2022.
[36] S. Mehta and M. Rastegari. Mobilevit: Light-weight, general-purpose, and mobile-friendly
vision transformer. In International Conference on Learning Representations, 2022. [37] A. K. Mishra, J. A. Latorre, J. Pool, D. Stosic, D. Stosic, G. Venkatesh, C. Yu, and P. Micikevi-
cius. Accelerating sparse deep neural networks. CoRR, 2021.
[38] X. Ning. T. Zhao, W. Li, P. Lei, Y. Wang. and H. Yang. DSA: more efficient budgeted pruning via differentiable sparsity allocation. In European Conference Computer Vision, 2020.
[39] E. Park and S. Yoo. Profit: A novel training method for sub-4-bit mobilenet models. In European Conference on Computer Vision, 2020.
[40] A. Peste, E. Iofinova, A. Vladu, and D. Alistarh. AC/DC: alternating compresseeddeecoppessss
training of deep neural networks. In Advances in Neural Information Processing Systems, 2021. [41] A. Polino, R. Pascanu, and D. Alistarh. Model compression via distillation and quantization. In International Conference on Learning Representations, 2018.
[42] R. K. Ramakrishnan, E. Sari, and V. P. Nia. Differentiable mask for pruning convolutional and
recurrent networks. In International Conference on Computer and Robot Vision, 2020.
[43] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, June 2018.
[44] V. Sanh, T. Wolf, and A. Rush. Movement pruning: Adaptive sparsity by fine-tuning. In Advances in Neural Information Processing Systems, 2020.
[45] P. Savarese, H. Silva, and M. Maire. Winning the lottery with continuous sparsification. In Advances in Neural Information Processing Systems, 2020.
[46] D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre,
D. Kumaran, T. Graepel. et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419): 140-1144, 2018.
[47] H. Tanaka, D. Kunin, D. L. K. Yamins, and S. Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. In Advances in Neural Information Processing Systems, 2020.
[48] K. A. Vasu, J. Gabriel, Zhu, O. Tuzel, and A. Ranjan. An improved one millisecond mobile backbone. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2023.
[49] C. Wang, G. Zhang, and R. B. Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations, 2020.
[50] K. Wang, Z. Liu. Y. Lin, J. Lin, and S. Han. Haq: Hardware- automated quantization with mixed precision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
[51] Y. Wang. Y. Lu, and T. Blankevoort. Differentiable joint pruning and quantization for hardware efficiency. In European Conference on Computer Vision, 2020.
[52] M. Wortsman, A. Farhadi, and M. Rastegari. Discovering neural wirings. In Advances in Neural
Information P Processing Systems, 2019,
[53] J. Wu, Y. Wang. Z. Wu, Z. Wang, A. Veeraraghavan, and Y. Lin. Deep k-means: Re-training and parameter sharing with harder cluster assignments for compressing deep convolutions. In International Conference Machine Learning, 2018.
12

---

[54] X. Xiao, Z. Wang, and S. Rajasekaran. Autoprune: Automatic network pruning by regularizing auxiliary parameters.
H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Ber -Buh E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, 2019.
[55] R. Yu, A. Li, C. Chen, J. Lai, V. L. Morariu, X. Han, M. Gao, C. Lin, and L. S. Davis. NISP: pruning networks using neuron importance score propagation. In Proceedings of the JEEF/CVF Conference Computer Vision and Pattern Recognition; 2018.
[56] O. Zafrir, A. Larey, G. Boudoukh, H. Shen, and M. Wasserblat. Prune once for all: Sparse
pre-trained language models. In Advances in Neural Information Processing Systems, 2021.
[57] Y. Zhang, M. Lin. M. Chen, Z. Xu, F. Chao, Y. Shen, K. Li, Y. Wu, and R. Ji. Optimizing gradient-driven criteria in network sparsity: Gradient all you need. In CoRR, 2022.
[58] X. Zhao. Y. Wang. X. Cai, C. Liu, and L. Zhang. Linear symmetric quantization of neu- ral networks for low-precision integer hardware. In International Conference on Learning Representations, 2019.
[59] A. Zhou, Y. Ma, J. Zhu, J. Liu, Z. Zhang, K. Yuan, W. Sun, and H. Li. Learning nim fine-grained structured sparse neural networks from scratch. In International Conference on Learning Representations, 2021.
[60] H. Zhou, J. Lan, R. Liu, and J. Yosinski. Deconstructing lottery tickets: Zeros, signs, and the supermask. In Advances in Neural Information Processing Systems, 2019.
[61] M. Zhu and S. Gupta. To prune, not to prune: Exploring the efficacy of pruning for model compression. In Workshop, International Conference on Learning Representations, 2018. [62] Z. Zhuang, M. Tan, B. Zhuang, J. Liu, Y. Guo, Q. Wu, J. Huang, and J. Zhu. Discrimination- aware channel pruning for deep neural networks. In Advances in Neural Information Processing Systems, 2019.
13

---

other params
init value: -]
T le 4 .c: 0.015
init. density: 0.5
inil value: -3200
8 alterations
T: Ie-4 E 0.015
F: 1:-e, , 0.015
init density: 0.5
init value: -3200
8 alterations
Ie e 0.015
le 1e-4,,e 0.015
init density: 0.5
init value: -12800
8 alterations
F Te-1,e 0.015
init density: 0.5
init value: -12800
8 alterations
T: le 4, .. 0.015
init value: -90
init value: -110
F 1: -,, 0.015
|3 ::1.0
:3:
8 1.0
B 1:00...
B 1.0
SGD: momentum, weight decay, cosine: learning rale. AdamW: epsilon, multiplicative learning rate. gamma, #wu: the number of warm-up epochs.
T
F
Adam W 1e-8. multiplicative 0.95 SGD 0.875 0.06, cosine le44
SGD0.878 0.065, cosine le-4
Mask optimizer, scheduler
SGD 0.9 0, cosine 0.1 n/a
SGD 0.9 0, cosine 0.1 n/a
SGD 0.9 O. cosine 0.1 SGD 0.875 1e-744, cosine 0.256
same as the main SGD 0.9 0. cosine 0.05 n/a
SGD cosine le-4 n/a
same as the main 11/2
n/a same as the main same as the main
same as the main
same as the main
same as the main
n/a
I//a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
Refer to Scction E and Tables 5 and 6 in [56]
Refer to Section and Table 3 in 44
Refer to Scction and Table 3 in 44
SGD 0.875 22222225777722 17-81--55... cosine 0.256
0.875 3.055..0177778 13e-4, cosine 0.256
SGD 0.875 0.875 2.2517578113 1157813e-5,, cosine 0.256
SGD SGD 0.875 3.051757813e 13-44, cosine 0.256
SGD 0.8875..5517577 13e-5, cosine 0.256
SGD 0.875 111777757881-55, 13e-5, cosine 0.256
SGD 0.875 3.05117777811 13e-4, cosine 0.256
SGD 5.9 multi-step 56,75 by 0.1
SGD0.9 le-4 (0.0 for BN), cosine 0.1
SGD lc-4 (0.0 for BN), cosine 0.1
SGD le-4 (0.0 for BN), cosine 0.1
SGD099 le-4 (0.0 for BN), cosine 0.1
AdamW 1e-8, multiplicative le-4 0.95
SGD 0.875 le-5 cosine 0.256
SGD 0.875 le-5 cosine 0.256
Main optimizer, scheduler
SGD 0.9 le-4, cosine 0.4
SGD 0.9 1c-4, cosine 0.1
SGD 0.9 1.4e-5. cosine 1.8
SGD 0.9 1.4e-5, cosine 1.8
SGD 0.9 1e-4, cosine 0.1
SGD 0.9 1.4e-5. cosine 1.8
SGD 0.9 1.4c-5, cosine 1.8
SGD 0.9 4c-4, cosine 0.1
SGD 0.9 1.4e-,, cosine 1.8
SGD 0.9 4-e4, cosine 0.05
SGD 0.9 8e-6, cosine 0.8
AdamW 1e-8 8, cosine le-4
AdamW 1e-8, cosine lc-4
Refer W
Refer M
Table 8: The hyper parameters Sections 3 andd
SGD
Batch sizz/Hfpocchs Pww.HGPP HGP HOPUH Nooeess
1
2
2
2
1
2
1
2
2
2
1
2
2
2
1
2
2
1
2
2
2
2
2
8
16
16
16
8
16
16
8
16
16
16
8
16
16
8
16
16
8
16
8
16
16
8
16
16
16
16
16
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
85x5
120
100
100
100
100
100
100
200
100
100
100
100
100
100
200
180
100
180
180
180
180
100
180
180
180
140
140
140
140
128
128
128
128
256
256
256
1024
1024
128
128
256
256
256
1024
1024
128
256
256
256
1024
128
256
256
256
1024
32
32
32
32
PDP- PDP-base baste////t///tg
PDP- batee////////
PDP basebaaeesste htr--baa///t//ppppeeeeee /str/ottt
PDP-based TDP-base/str/opt
PDP- base+
PDP PDP-basee base+
STR (85. (8.99)
STR 91.7%)
Method
CS
PDP
OpiG
PDP
GraNet
GMP
STR
OptG
ACDC
GraNct
GMP
STR
OptG
ACDC
GraNet
STR
OptG
ACDC
GraNet
STR
OptG
ACDC
GMP
MOV
POFA
OptG
PDP
MobileNet-vl
MobileNet-v2
Network
ResNer20
ResNet18
ResNer50
GPT2
Bert
14

---

1074
Sparsity
convt
lopert Spper comee
kgyet Loon
lowe..
lopet cood
bapert conv
bypet dotnnnrppppp
spper cood
loyel, corv
kopet
kgeek,
PDP
GraNet
# STR
OptG
ACDC
(a) ResNet18
2004
99%
Sparsity
60%
tepr Roams
lxpet Loons
layel Zoomr
layer byerttomm homm
Inyert Lneer foomi
Mepa Lomi
leped Loars
byyert Loows
lecd teck Soaone
byed Looe
lyyer Lyyet Zome
byyed Soows
looo Loae
lywet lyeet 20o02
lepor
Leyes
layes
lece
layed
lecol
byyel
PDP GraNet STR
OptG
ACDC
(b) ResNet50
1000
Sparsity
THALOO
modiio3
modd1133
RoM120
clantee
PDP
GraNet
STR
OptG
ACDC
(c) MohileNet-vi
Sparsity
60%
Saumut
naseel 1ome
Setum 2etum Sanvo
2
Setues Servdo
betus Sems. Lo
faaore 2
Setunn 7 zamnno
(enures Tooee Lo
Seturs Seturs Renvdo
Satun Kmns. Lo
Scatno Ia Lo
femaee Romm1
foauee Boom
Rotaes Rensi Lo
faaurs 22000
Satar Bomis Lo
natroii Kamin
funne 2oom
datae 10
foomm 2
Saturo IED
deslisi
PDP GraNet STR
OptG
ACDC
(d) MobileNet-v2
Figure 6: Lise sparsity allocation from the experiments in Table
15

---

Inference MAC (e6)
con1
boyert.
leed Nes3hddmaarpoo
Lapont www.tadmmmareea
loyelt cood
loyedt
PDP
GraNet
# STR
OptG
ACDC
(a) ResNet18
30
60
Infererence MAC (e6)
40
20
lypel Indooooopppi
Spen eeed
Jeyer cees)
Jper apel 2003
Joeth Londl
kast bast 2ed
byart byert
bave byer coed
Joed aped Seess
aged
lpped
PDP GraNet
STR
OptG
ACDD
(b) ResNet50
12
Inference MAC (e6)
15
Bhat
mO00111
mOGLAJ
RUGGLE
11.
GraNet
OptG #ACDC
(c) MobileNet-vl
Inference MAC (e6)
10
fonseee D0
Setur SetusLoose Lorel
Renar fonus Soomdo 60
Ienae manckaamb Lomba bil
Fome ia
tusur Somn D0
Ronne Som..n in
Dunnes humess.om Soom 00
Banee Foom11 1d
Ioaaec Ionnec comm 1d
Donme Room i0
Salan A0tum..1..002 10
1lam in
Satanl I coan2
E.com.00 60
(clurs Mauns.2oomni 12amm in
loame bn
Ioaama Cammi4ommen Macem 00
Intures 1 S.cen 0d
hunne 7e ccOmDO bil
foame Tommm bil
foarse M
cholftrl
Rusues Soom
1..0m
funnee
PDP GraNet STR
OptG
ACDC
(d) MobileNet-v2
Figure 7: Layer-wise Inference MAC distribution from the experiments in Table
16

---

Dense
GraNet
STR
OptG
ACDC
PDP
0.0
1.0
1.3
2.0
2.3
3.0
3.3
4.0
4.3
5.0
5.3
6.0
6.3
7.0
7.3
8.0
8.3
9.0
9.3
10.0
10.3
fc
Table 9: The weight histograms in log scale for MobileNet-v I in Table

---

A Training Configurations and Hyper-parameters
Since some techniques in Sections 3 and 4 as shown in Table !
require extra training parameters and pruning scheduling we disclose the training configurations and hypee-parameter we found the best
in Table
B Trade-off in Pruning
Pruning for DNN requires exploring good trade-off between model accuracy and inference latency under given pruning target. Such a challenge can be elaborated with the MobiletNet-vl Dense case in Fig. where the following observations can be made:
The earlier layers have significantly fewer parameters than the later layers while still having comparable inference MACs as shown in (a). For example, the final classifier, which is linear layer, has the lowest inference MAC but the 2nd largest parameters.
When per-parameter inference MAC is computed as in (b) (which is in log-scale), we can easily see that the parameters in the earlier layers get much more involved the inference than those in the later layers. For example, the MAC-per-parameee for the last classifier is only 1.
Then, with given pruning target, one pruning scheme can favor heavily pruning the classifier, as it is easier hit the target without affecting model accuracy much (i.e., each parameter shows up only once in the forward pass), but this would fail to reduce the inference MAC enough. Then, the other scheme may favor aggressively pruning the earlier layers significantly minimize the inference latency at a much greater risk of degrading the model accuracy. Therefore. it is critical to find good balance between accuracy and inference speed. According to our experimental results, PDP can accomplish such a balance using differential pruning w.r.t. the task loss. Such trade-off can be optimized differently depending on whether particular sparsity pattern or structure enforced.
10%
30%
Inferenee MAC:
20%
Parumeter Count
9%
10%
014
100
Tredkl 10
Dok130
duaifi
Normalized Inference MAC
-Normalized Parameter Count
(a) Normalized inference MAC and parameter count for each layer.
000
Inference MAC
109
wade 10
mode 13
we0e 50
aace-33
moce 5.
niCe 60
1O0KK K3
Iacc PA
roce 50
mobe 100
1061113
wece
redde
MAC'parameter
(b) The inference MAC per parameter each layer.
Figure 8: Layer-wise Inference MAC and Parameters from the MobileNet-vl Dense case in Table
18

---

Unstructured schemes make individual and independent pruning decision for each weight to maximize the flexibility and minimize the accuracy degradation. Simple and grad- ual/iterative pruning based on the weight magnitude has been studied extensively [3 15
Random Pruning:
19
61
In these schemes, once a weight is pruned, it does not have the second chance to become unpruned and improve the model quality. To address such challenges, Rigl. [12] proposes to grow sparse network by reallocating the removed weights based on their dense gradients. Applying brain-inspired neurogeneration (i.e., unpruning some weights based gradients) and leveraging pruning plasticity is proposed [4). Altering the phase dense and sparse training to accomplish co-training of sparse and dense models is studied, which results in good model accuracies vision tasks [40]. Unlike other magnitude-drine
pruning, supermask training [60] integrated with gradient- drivvnn sparsity is proposed in [57]. where accumulated gradients are used to generate binary masks and
estimator 5 is used for backward propagation. Based on the lottery hypothesis [3], pruning in
one-shot with heuristics [47 gradient-driven metrics 49 is explored. More advanced techniques based on the second-order sensitivity analysis for language models are proposed in [27].
Structured/Channe Pruning: Unstructured pruning limits inference latency speedup as it suffers from poor memory access performance, and does not fit well on parallel computation [3 35 Recent research extends unstructured pruning by imposing particular sparsity pattern during pruning at the cost of lower model predictive power, but increases the hardware performance during inference. One popular and effective form of structured pruning is channel pruning, where some channels with negligible effects on the model accuracy are discarded [2] [25] 30 Using regularization to prune weights in block is proposed in [29] 26 leverages the 3 in BatchNorm to select the channels to prune (i.e., beta S <) with ReLU assumed, which limits its applicability to wider set of DNNs. N:M pruning enforces that there are N non-zero weights out of every consecutive M weights [59] which enables a compact memory layout and efficient inferences on hardware 24 37 [59.. A non- differentiable method for N:M pruning with complex back-prorogation based on STE is presented in (59)
C Latency reduction with Random Sparsity
While MAC (to measure the latency gain) is rather a theoretical metric, not a measured metric, but it still captures the speedup benefit from pruning to some degree on some modern smartphones where ML accelerators natively support unstructured sparsity [2]. When we ran our pruned model on iPhone12 with the latest OS update, we obtained the following end-to-end latency measurements.
Method
ResNet50
MobileNet-v2
Dense
2.71
0.95
PDP
1.39
0.75
Speedup (%)
94
27
Table 10: A single image classification latency in msscc on iPhone for the pruned models in Table
Due to other system overheads, the end-to-end latency reduction is not as significant MAC saving, yet our result still implies the potential benefits of unstructured sparsity on modern smartphone.
D PDP Algorithm and Training Flow
In order to obtain t in Fig 2 PDP needs a target pruming ratio r. The pruning ratio can be computed by selecting the top weights with larger magnitudes across all the layers and then instantly convert the selections into the per-layer ratios. Another way is to handcraft per-layer ratios, or reuse an existing configuration. Also, PDP is using the softmax operation which makes the softness concentrated over the weights around the t (as shown in Fig. 2). Hence, we gradually increase the target pruning ratio from 0 to r so that all low magnitude weights in the pruning range have a chance use soft-mask and settle down smoothly. For that purpose, we introduce a scaling step to let each weight have opportunities leverage soft-mask at least once, which leads to the training flow in Algorithm []
We select to avoid pruning decisions being dominated by the initial weight values (which can be simply random). Choosing the right number of epochs as depends various factors such as the
19

---

Algorithm Training flow for PDP
1:
procedure TRAIN(E, s, W (Wo: Wi,
2:
for epoch e [0. 8) 3)ddd do for each mini-batch do forward with [Wo, W2....]
3:
4:
5:
backward-pase and update W end for end for
6:
7:
8:
W topk abs(W)s)))))))) W') r. (w)))
[ro, rl. .....-.--- = mWW, n(W,
9:
(W))
(W)) -....
10:
for epoch e in [s, 1, 8 1 2, do
11:
Fo, f.... min(1, (e s)) [ro. ..... for each mini-batch do
12:
13:
for i 1,.., do
14:
(abs W = (ab
R) n(W)))
Wi bottomk (abenn/Kaaaasaa))aaaa.. n(W):)
4 0.5(min((W,) (wa)(Waa-Wm))))) (Wi)
16:
end for
18:
for i
E {0, 30. do
[Z M = sof coftmax( I.W.W... Welement-wise
19:
20:
W; = M, W; /element-wise
21:
forward-pase with W, end for backward-pass and update W end for end for
22:
23:
24:
25:
26:
W; = [M] o Wi, Vi E {0,1...
27:
end procedure
network size, the target sparsity and on. Hence, is user hyper-parameter. In our experiments, the following guidelines worked best (and this is how we selected it).
must be larger than the number of warm-up epochs.
After r epochs, the validation accuracy hits consistently over the half of the accuracy upper-bound (which is 50% for classifications) for epochs.
In lines 2-7, a normal training is performed for the first epochs. Then, in lines and 9, the per-layer target pruning ratio is computed by selecting the bottom r n(W)) weights globally in terms of the magnitude. Then, the remaining epochs, we use PDP to generate soft-masks as the line 15, while gradually increasing the target ratio in lines 10 and 11. The updated weight distribution is captured by updating &; as in the line 16 for all weight matrices. Once the entire training is over, we binarize the last mask for each weight to output the fully pruned weight for inference as in the line 26. Overall, the average runtime complexity of PDP is OW), as we only need to exercise topk algorithm (i.e, sorintg W is not necessary).
While fewer learnable parameters will reduce the model footprint and the communication cost (thus speeding up the training) in a distributed training setup, it does not necessarily lead to smaller memory consumption as the soft mask from PDP needs to generate related activation tensors. To understand the memory consumption better, we made small GPT2 case with block_size = 128 and n layer = 3 (a downscale version of the one in Table 2 And, then GPU memory is measured at three different spots for PDP and OptG, and reported in Table II
Spot0: right after the model/optimizer are created (the parameter and PyTorch overheads)
*
Spot1: right after forward, before backward (the peak memory consumption)
Spot2: right after backward, before weight update (other parameter-related overheads)
From Table II we can see the following:
20

---

Method
SpotO
Spot1
Spot2
OptG
2.93
18.3
3.58
PDP
1.30
16.3
2.93
Saving
1.63
2.0
0.65
Table 11: Memory footprint reduction from PDP.
Not having extra mask parameters helps to reduce the model/optimizer overheads.
The peak memory is dominated by the activations (both data and mask). Yet, about 10 saving the peak memory is observed.
After backward, the gradients for the learnable masks take up a large memory space, which have been all-reduced in the multi-GPU setting.
To isolate the benefit of the proposed soft mask in PDP from the gradual pruning in Algorithm [ we PDP for ResNet50 and MobileNet-vl training in the exactly same configuration in Section 4 without the proposed soft mask, and obtained the following results. The result supports the efficacy of the proposed soft mask in PDP: without it, the ImageNet top1 accuracy drops by 1.6% and 1.4%, respectively.
Method PDP w/o softmask
ResNet50
MobileNet-vi
73.1
66.8
PDP
74.7
68.2
A
1.6
1.4
Table 12: Top-1 accuracy with ImageNetlk witout the softmax in PDP
E
Ablation Study: Hyper-Parameter search
In the current PDP implementation, we use global r to control the level of softness the pruning mask. Therefore, the selection of 7 affects the model predictive power and should be carefully tuned. In order to explore the methodology for the search, we tried various values for MobileNet-v2 training, and the results are plotted Fig. 9 The selection of affects the model predictive power as shown in Fig. / where there appears to be an optimal т. For examples of MobileNet-v2, T le the best value and is used for all the experiments in Section 4
Top-1 Accuracy
66.9%
66.8%
66.7%
66.6%
66.5%
66.4%
1.03-06
1.0E-05
1.0E-04
L.0E-03
1.0E-02
Figure 9: MobileNet-v2 with varying values.
Since, Fig. Jhows a concave curve, one could use a binary search find the T values w.r.t. the top-1 accuracy. Also, it could be possible to cast 7 as a learnable parameter for each layer or apply some scheduling to improve the model accuracy further (as future work), still both approaches need an excellent initial point which can be found using binary search technique.
21

---

F Additional Result for Section 4
Different approaches made different sparsity allocations per the characteristics of the algorithm for a given pruning target, which results in complex trade-off's between model accuracy and inference speed. We report the detailed sparsity and inference MAC break-down for each layer in Fig. 6 and Fig. 7 on ImageNetlk and summarize our observations as follows:
OptG prunes the early convolution layers quite aggressively ResNet18 and ResNet50, which leads to very low inference MACs shown in Fig. (a) and (b), yet at the cost of the worse Top-1 accuracy. For example, the inference MAC of ResNet18 from OptG is more than 2x less than that from ACDC,
Interestingly, STR becomes aggressive in pruning the early convolution layers in MobileNet- vl/v2, while OptG does not expose such behavior to MobileNet-v (unlike it did for ResNer18/50) 8/50). Such characteristics also favor the low inference latency over the model accuracy. Also, STR tends not to prune the last linear layer much as discussed in
Unlike OptG and STR, ACDC does prune the early convolution layers much for the tested networks, but prunes somewhat actively for the late convolution and linear layers, which leads to high model accuracies at the cost of higher inference latencies.
PDP is somewhat between STR and ACDC and modest across all layers in pruning alloca- tion for all the networks, leading to superior accuracy and inference trade-offs. For example, the layers model.13.3 of MobileNet-vi and features. 17..ures..1.... 1w.111..coo.. ww.coov........mm 1.0 of MobileNet-v2 have the most difference among algorithms, and PDP modest pruning these two layers.
OptG has very low inference MACs on the earlier layers ResNet18 and ResNet50 due its aggressive pruning on these as seen in Fig. 6 (a) and (b), which leads to the extremely low inference latencies as shown in Fig. (a) and (b).
GraNet tends to prune the earlier layers less but the later layers more in general which explains why GraNet shows the highest inference MACs in Fig.
Sparsity (%)
Network
Method
80
70
60
50
PDP
69.8
70.8
71.0
71.3
ResNet-18
ACDC
69.4
70.3
70.6
70.8
PDP
69.5
71.0
71.6
71.9
MobileNet-vi
OptG
68.1
69.1
69.6
69.7
ACDC
68.5
69.9
70.9
71.4
Table 13: 1 accuracy with ImageNetlk: PDP outperforms other schemes with various pruning rates.
Validation
Sparsity (%)
Network
Method
dataset
80
70
60
50
matched
83.7
84.0
84.3
84.7
PDP
mismatched
83.4
83.8
84.4
84.5
Bert
matched
80.3
80.7
81.3
81.2
OptG
mismatched
80.1
80.7
80.5
81.0
Table 14: Accuracies with MNLI benchmark: PDP maintains the similar accuracy lead over other schemes.
Table. 9 shows the pruned weight histograms of MobileNet-vi from Table We observe that each a ggriithm affects the weight distribution in a slightly different way.
create mild STR prefers to split the distribution more widely than others. For the example of the layer 5.0, STR clearly separated the positive and negative weights with a wide gap centered at the zero, while others sis not, except PDP created slight dip around the separation.
PDP tends to spread out the sparsified weight distributions more than others. For the example of the fe layer, the weights from PDP range from -0.5 to 2.0, while those from
22

---

others are from -0.5 to 1.5. On the other hand, GraNet tends to keep the weight distributions tight.
We also experimented with varying pruning rates for PDP, OptG and ACDC for MobiletNet-vl and ResNet- 18 with ImageNet1k, k; and Bert with MNLI benchmark under the same configurations as in Section 4 Overall, as shown in Table 13 and Table [4] all tested schemes delivered higher accuracy with lower pruning rate, yet we can observe that PDP keeps its superiority to other schemes over all the tested pruning rates.
Code References G
Dense http://yyooccoogghhhhiniiaaeeeel html
GradNet htgp:////gggggloooooooooCCC/CCCo
OptG http://ggihubbccom//yxxx////CCC
ACDC http://giihbbbcoom//////SSSSAAC
STR https//gittbb.com//al///aa..11
GMP http://gittbb.com//AAlWWlaabbbbb
DNW http://jiihbb.coo//AAAAAVLMMsss
CS tttp://gihbcccooocoocccsoooooonp
LNM hutp://gthh..ccomm/.m.ssssssh/ps
23


Reviews:
[{"review_id": "4i1uFWbv05", "reviewer": "Reviewer_qvwL", "summary": "The paper introduces a new DNN pruning scheme called Parameter-free Differentiable Pruning (PDP), which is an efficient and effective train-time pruning method that offers state-of-the-art qualities in model size, accuracy, and training cost. Unlike existing pruning approaches, PDP generates soft pruning masks for weights in a parameter-free manner, making it easy to apply to various vision and natural language tasks, DNN architectures, and structured pruning constraints. The paper presents experimental results on several benchmark datasets, including MobileNet-v1 and BERT, demonstrating that PDP achieves impressive results in terms of model size, accuracy, and training cost.", "questions": "As shown in weaknesses part.", "limitations": "The paper did not any (Peak) memory consumption result of PDP during differentiable pruning.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper introduced a PDP, a novel differentiable pruning methods that is parameter-free, which uses a dynamic function of weights to generate soft pruning masks for the weights.\n- PDP can be applied tto structured and channel pruning, such as N:M pruning, where top-of-the-line GPUs support such configuration.", "weaknesses": "- The PDP differentiable pruning does not introduce extra parameter, but it still need to generate (soft) mask from weight, which would induce extra activation maps, how is the memory consumption during differentiable pruning compare to other SoTAs?\n- The paper seem provide results on MobileNet-v1/v2 but not on MobileNet-v3, can the author elaborate on why?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q0: The PDP differentiable pruning does not introduce extra parameter, but it still need to generate (soft) mask from weight, which would induce extra activation maps, how is the memory consumption during differentiable pruning compare to other SoTAs?**\n\nThank you for raising an important question regarding memory consumption. While fewer learnable parameters will reduce the model footprint and the communication cost (thus speeding up the training) in a multi-node setup, we agree that still extra activation maps will be induced. To understand the memory consumption better, we made a small GPT2 case with block_size=128 and n_layer=3 (a down-scale version of the one in Table 3). And, then GPU memory (in GB) is measured at three different spots for PDP and OptG (i.e., other SoTAs). \n* **Spot0**: right after model/optimizer are created (the parameter + pytorch overheads)\n* **Spot1**: right after forward, before backward (the peak memory consumption)\n* **Spot2**: right after backward, before weight update (other parameter-related overheads)\n\n|     | Spot0  | Spot1 | Spot2 |\n| --- | --- | --- | --- | \n| OptG | 2.93 |  18.3  | 3.58 |\n| PDP   | 1.30 |  16.3 | 2.93 |\n\nFrom the table, we can see the following:\n* Not having extra mask parameters helps to reduce the model/optimizer overheads. \n* However, the peak memory is dominated by the activations (both data and mask). And as pointed out by the reviewer, even a soft mask still requires activation space. Yet, about 10% saving in the peak memory is observed.\n* After backward, the gradients for the learnable masks take a large memory space, which have been all-reduced in the multi-GPU setting.\n\nWe will add the above discussion in the final draft.\n\n**Q1: The paper seem provide results on MobileNet-v1/v2 but not on MobileNet-v3, can the author elaborate on why?**\n\nWe appreciate your feedback on MobileNet-v3. We didn't experiment with MobileNet-v3, because the prior arts don't report the related results, thus making comparison with other SoTAs on MobileNet-v3 hard. \n\nWhen we applied PDP with an 80% sparsity target for MobileNet-v3 (5.5M parameters), we achieved 71.5% top-1 ImageNet1K accuracy, which is 2.5\\% down from the dense version. We will include this result in the final draft.\n"}, {"review_id": "r9E2BNg9Dh", "reviewer": "Reviewer_R2q9", "summary": "This paper describes a novel pruning algorithm named parameter-free differentiable pruning (PDP). The core idea is to generate soft pruning masks (i.e., mask values are not finalized until the final iteration of fine-tuning) using a parameter-free, differentiable dynamic function of the weights of the network. This approach eliminates the need to train additional mask parameters (and related hyper-parameter tuning) and appears to be easy to integrate into existing training pipelines. PDP is evaluated on networks drawn from vision and NLP, and on both unstructured and structured N:M sparsity patterns. The paper reports both accuracy and MAC improvements and provides a comparison of PDP to other SOTA pruning algorithms.\n", "questions": "* What is the motivation for selecting the per-layer sparsity value (r) after a few epochs of training? How would a user select the right number of epochs for a given network,  sparsity pattern and global sparsity degree?\n* Readability suggestion: consider moving Section 3.2 before 3.1.\n", "limitations": "Limitations have been adequately addressed in the paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is fairly well-written, but could use some reorganization to help with the overall flow.\n* The paper targets the relevant and important problem of structured sparsification of large DNNs to improve inference efficiency.\n* While soft and progressive pruning are well-known in the literature, its parameter-free nature and the specific mask computation make PDP unique.\n* Evaluation methodology is sound, with relevant comparisons to other SOTA approaches, and results being reported on various types of networks (CNNs, attention-based, etc.) across disparate domains.", "weaknesses": "Measuring inference efficiency in MACs can often be misleading. For instance, unstructured pruning can reduce MACs by getting rid of individual ineffectual computations, but this is difficult to realize in modern parallel hardware such as GPUs and TPUs. Reporting at least some data points with actual runtime numbers would be useful.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q0: Measuring inference efficiency in MACs can often be misleading. For instance, unstructured pruning can reduce MACs by getting rid of individual ineffectual computations, but this is difficult to realize in modern parallel hardware such as GPUs and TPUs. Reporting at least some data points with actual runtime numbers would be useful.**\n\nThank you for your feedback on MAC. We agree that MAC is rather a theoretical metric, not a measured metric, but it still captures the speedup benefit from pruning to some degree. GPUs don't support unstructured sparsity, but there exist modern smartphones where ML accelerators natively support unstructured sparsity (unfortunately, the link with details cannot be provided per the NeurIPS rebuttal policy). We used a 2-year old phone with the latest OS update and obtained the following latency measurements in msec.\n|     | ResNet50  | MobileNet_v2 |\n| --- | --- | --- |  \n| Dense | 2.71 |  0.95  |\n| PDP   |  1.39 |  0.75 |\n\nDue to other system overheads, the end-to-end latency reduction is not as significant as MAC saving, yet still shows the potential benefits of unstructured sparsity on a modern device. We hope these data points would address the reviewer's concern .\n\n**Q1: What is the motivation for selecting the per-layer sparsity value (r) after a few epochs of training? How would a user select the right number of epochs for a given network, sparsity pattern and global sparsity degree?**\n\nThank you for the question. $r$ is selected after a few epochs to avoid pruning decisions being dominated by the initial weight values (which can be simply random). Choosing the right number of epochs for a given network depends on various factors the reviewer has already mentioned. Hence, this is a part of hyper-parameter tuning. In our experiments, the following guidelines worked best (and this is how we selected it).\n* passes the warm-up epochs\n* hits consistently over the half of the accuracy upper-bound (which is 50% for classifications) for 5 epochs.\n\n**Q2: Readability suggestion: consider moving Section 3.2 before 3.1.**\n\nThank you for the suggestion. We will accommodate in the updated draft.\n\n"}, {"review_id": "r5b1ER5I6E", "reviewer": "Reviewer_tv9j", "summary": "The paper focuses on DNN pruning and proposes a novel approach using a soft mask during training. The soft mask is designed to encourage weights around the pruning threshold to actively switch their states, aiding in the recovery of pruned weights. This method is simple, efficient, and introduces no additional parameters. Experimental evaluations of the proposed approach have been conducted on ResNets, MobileNets, and BERT. It also explores different pruning settings such as N:M pruning, random pruning and channel pruning.", "questions": "Please see the method and experiment points in the weakness part. If my concerns are addressed, I would like to increase my score.", "limitations": "There is no potential negative societal impact of their work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "* The paper proposes a soft mask during training, which smooths the participation of weights around the pruning threshold in the neural network. This can give more chances for weights around pruning threshold to change their states, especially help pruned weights actively recover themselves.\n* The compared methods are enough in experiments.\n* The paper discusses relevant works and provides a comprehensive analysis.", "weaknesses": "* Method:\n  *  Lack of differentiable analysis: The title and content of the paper stress the differentiable pruning, however, method part  (3.2) does not give differentiable analysis.\n  * Insufficient explanation of the design of \"t\" and calculation of \"m(w)\": Why is “t” in PDP training flow designed in this way? The paper lacks clear explanations about the design of “t” and calculation of “m(w)”.\n  * Lack of analysis and ablation study on the efficacy of the proposed soft mask: I find that the paper adopts progressive pruning as illustrated in the algorithm chart in supplementary material. As progressive pruning is an effective technique for final accuracy, I am doubtful about the real effect of the proposed method. To better understand the effectiveness of the proposed soft mask, we need more analysis and ablation studies.\n* Experiments: \n  * More experiments about Transformer architectures are preferred. The paper provides BERT models experiments on MNLI dataset. Experiments with ViT models or BERT models on the whole GLUE benchmark will be preferred.\n  * In the main body, there are not enough sparsity ratio selections. Table 4 provides only one sparsity ratio for each model. More sparsity ratios, ranging from low to high are necessary to understand the effect of the proposed method.\n* Writing: \n  * Redundancy between Figure 5 and Table 4: It seems that Figure 5 and Table 4 convey the same information. What is the difference between Figure 5 and Table 4 or why illustrate the same information twice? I think each table or figure should contain specific information, especially in the main body.\n  * Improper organization of the paper: The paper first illustrates the benefits of PDP over existing pruning approaches, then introduces the PDP algorithm, which can be confusing. Before readers know the method clearly, it would be difficult to understand its benefits. The better solution would be to change their order or to reconsider the effects of section 3.1, such as serving it as a motivation section.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q0: Insufficient explanation of the design of \"t\" and calculation of \"m(w)\": Why is “t” in PDP training flow designed in this way? The paper lacks clear explanations about the design of “t” and calculation of “m(w)”.**\n\nThank you for the chance to clarify the important notation in our work.\n\n$t$ is the weight value which will have exactly 0.5 chance to be pruned and 0.5 chance to be not-pruned of a layer. And, we capture $t$ from the weight distribution of a layer as in Fig. 3 (a). Consequently, $t$ serves as a conditional point to decide whether a weight will have a soft mask $m(w)$ larger than 0.5. \n\nThe way we compute $m(w)$ is differentiable using softmax, as shown in Fig. 3 (a). More specifically, we compute $m(w)$ as follows:\n\n$m(w) = \\frac{e^{\\frac{w^2}{\\tau}}}{e^{\\frac{w^2}{\\tau}}+e^{\\frac{t^2}{\\tau}}} $\n\nThus, when $w$ happens to be the same as $t$, $m(w)$ is 0.5. If $w$ is relatively larger than $t$, then $m(w)$ is also larger than 0.5 as in Fig 3. (b).\n\n**Q1:The title and content of the paper stress the differentiable pruning, however, method part (3.2) does not give differentiable analysis.**\n\nThank you for the opportunity to explain more details. Without loss of generality, let's assume $\\tau = 1$ (the temperature of softmax). Then, the masked weight $\\hat w$ in PDP is \n\n$\\hat w = m(w) \\cdot w = \\frac{e^{w^2}}{e^{w^2}+e^{t^2}} \\cdot w $\n\nThen, for a given gradient on $\\hat w,  \\frac{\\partial L}{\\partial \\hat w}$,\n\n$\\frac{\\partial L}{\\partial w} = \\frac{\\partial L}{\\partial \\hat w} m(w)  + 2\\frac{\\partial L}{\\partial \\hat w} w^2 m(w)(1-m(w)) $\n\nThe 1st term is a typical gradient in mask-based pruning, and the 2nd term is an additional gradient with a positive factor. Then,\n* if $m(w)  \\approx 0$ or hard-prune, $\\frac{\\partial L}{\\partial w} = 0$ likewise other pruning algorithms.\n* if $m(w)  \\approx 1$ or hard-not-to-prune,  $\\frac{\\partial L}{\\partial w} = \\frac{\\partial L}{\\partial \\hat w}$ likewise other pruning algorithms.\n* when $m(w)  \\approx 0.5$ (i.e., pruning decision is unclear), $m(w)(1-m(w))$ is maximized, boosting the $w$ movement.\n \nHence, PDP will accelerate the SGD updates for the weights near the pruning boundary ($t$) toward a loss-decreasing direction (which means more learning from each iteration), encouraging the weights to settle with the proper pruning decision at the end. Even if the current gradient is not globally beneficial for the task, many \"second\" chances will eventually help recover the damages in an accelerated manner. We will add this differentiable analysis to the final draft.\n\n\n**Q2: Lack of analysis and ablation study on the efficacy of the proposed soft mask**\n\nThank you for the feedback. Gradual pruning is strong and popular; in fact, GMP, STR, ACDC, GradNet, and OptG all use some form of gradual pruning. To analyze the benefit of the proposed soft mask in PDP as ablation study, we re-run PDP for resnet50 and mobilenet_v1 training in the exactly same configuration as in Section 4 **without**  the proposed soft mask.\n|     | ResNet50  | MobileNet_v1 | \n| --- | --- | --- | \n| PDP w/o softmask   | 73.1 |  66.8 |\n| PDP | 74.7 |  68.2 |\n| $\\Delta$| 1.6  | 1.4 |\n\nThe result supports the efficacy of the proposed soft mask in PDP: without it, the ImageNet top1 accuracy drops by 1.6% and 1.4%, respectively. \n\n**Q3: More experiments about Transformer architectures are preferred. The paper provides BERT models experiments on MNLI dataset. Experiments with ViT models or BERT models on the whole GLUE benchmark will be preferred.**\n\nThank you for suggesting helpful experiments. While we couldn't complete the whole GLUE benchmark due to the limited time and resources, we were able to perform additional experiments with the QQP (Quora Question Pairs2) benchmark which is\n* The 2nd largest dataset in GLUE (363,846 samples), following MNLI (392,702)\n* MVP reported only MNLI and QQP (two largest) out of the GLUE benchmarks.\n* MVP and POFA reported the results on QQP, thus making comparison possible.\n\nWe used the same hyper-params as in Section 4 and targeted 90% sparsity, and obtained the following results (including MNLI results from the current submission).\n\n|     | OptG |MVP  | POFA | PDP| \n| --- | --- | -- | --- | --|\n| MNLI-m    | 78.5 |81.2 |  81.5 | 83.1|\n| MNLI-mm   | 78.3 |81.8 |  82.4 | 83.0|\n| QQP-acc   | 89.8 | 90.2 |  90.9 | 90.9 |\n| QQP-f1   | 86.2 |86.8 |  87.7 | 87.7 |\n\nThe result shows that PDP can offer the state-of-the-art 90.9% accuracy for QQP which is already very close to a dense model accuracy of ~91%. POFA showed the same quality results but using a more complex training flow. \n\n**Q4: In the main body, there are not enough sparsity ratio selections. Table 4 provides only one sparsity ratio for each model. More sparsity ratios, ranging from low to high are necessary to understand the effect of the proposed method.**\n\nThank for the suggestion. In Tables 10/11 of Appendix, we have MobileNet-v1, ResNet18, and BERT results on the sparsity targets ranging from 50% to 80%. We will mention these additional results clearly in Section 4.\n\n**Q5: Redundancy between Figure 5 and Table 4**\n\nThank you for the review. Indeed, some redundant information exists between Fig. 5 and Table 4, because they are complementary in the sense that one for high-level abstraction and the other for low-level details. The numbers in the Table couldn't abstract out the complex trade-off among accuracy-latency-cost in an intuitive manner. Hence, we added Fig. 5 with key numbers to visualize the trade-off and help readers on this complex matter. \n\nWe agree that the space in the main body can be better utilized for new information/results (especially ones from this rebuttal), and we accommodate your feedback in the final draft.\n\n**Q6:  The better solution would be to change their order or to reconsider the effects of section 3.1**\n\nThank you for the feedback. We will reorder as suggested and polish the write-up in the updated draft.\n\n"}, {"review_id": "VERSk50Y74", "reviewer": "Reviewer_8xNe", "summary": "This paper deals with the pruning algorithm PDP on DNN, the main innovation is to generate a differentiable mask based on weights using a designed threshold t, and the softmax function. This mask can ensure the gradient propagation during forward and backward process while training, reducing the accuracy loss.", "questions": "Please see the weakness part.", "limitations": "No limitations discussed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper is well written and easy to follow.\n2. The proposed method is novel and provides an efficient way to learn pruning masks without incurring heavy parameter burden.\n3. The performance increase remains constant on various sparse ratios and models.", "weaknesses": "1. The employment of different masks can impact training process performance, yet a higher SAD[1] may result in a significant decline in accuracy. Given that the masks are still changeable in the later stages of PDP training, how can it provide high accuracy? \n\n[1] Learning N: M fine-grained structured sparse neural networks from scratch. In ICLR, 2020.\n\n2. The description of PDP algorithm in section 3.2 may not be completely unambiguous. For the calculation of the conditional point t, the meaning of the paper should be to take the median of $r*n(W)$ and $(1-r)*n(W)$, but figure 3 (a) shows that $t=0.5\\{min(W_h)+max(W_l)\\}$, which seems to be wrong.\n\n3. In Table 6, the performance of LNM seems to be quite different from that reported in the original paper. Please explain this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q0: The employment of different masks can impact training process performance, yet a higher SAD[1] may result in a significant decline in accuracy. Given that the masks are still changeable in the later stages of PDP training, how can it provide high accuracy?**\n\nThank you for the question. Sparse Architecture Divergence (SAD) from [1] defines the changes in the pruning mask during training. The insight behind in SAD is that lower SAD will incur less disruption to the training process, potentially improving the model quality. We first like to point out that our PDP pruning mask is soft and continuous during training, and only get binarized for validation/test/inference. According, in fact, our PDP can make SAD between iterations smaller than the hard/binarized pruning cases. \n\nAs an illustration, for the hard masks, the following scenario leads to total SAD 2 and max SAD 1.\n\n|     | iter0  | iter1 | iter2  | iter3 |\n| --- | --- |  --- | --- | --- |  \n| Mask | 0 |  1  | 1 |  0  |\n| SAD   |  - |  1 | 0 |  1  |\n\nFor PDP case with $t$=0.5, the same scenario could be the following, where total SAD is 0.4 and max SAD 0.2. \n\n|     | iter0  | iter1 | iter2  | iter3 |\n| --- | --- |  --- | --- | --- |  \n| training-mask | 0.4 |  0.6  | 0.6 |  0.4  |\n| SAD   |  - |  0.2 | 0 |  0.2  | \n\nwhich implies PDP slowly tries the same pruning scenario as the above but very gradually using the soft mask.\n\nHence, PDP will offer much better model stability during training in terms of SAD\nthanks to our proposed differentiable soft mask, and let each weight converge into the right pruning choice at the end. When the network is finalized with a binary mask at the end of the training, our PDP can yield a higher-quality model as shown in Table 6 (i.e. comparison against LNM[1] (which proposed SAD).\n\n\n**Q1:  The description of PDP algorithm in section 3.2 may not be completely unambiguous. For the calculation of the conditional point t, the meaning of the paper should be to take the median of  and , but figure 3 (a) shows that , which seems to be wrong.**\n\nAppreciate the opportunity to clarify the definition of the conditional point $t$. $t$ is essentially a weight value that would have the exact 0.5 mask value. Assuming every weight value in a layer is unique, $W_l \\cup W_h$ includes all the weights in the layer. Therefore in this case, $t$ is the median of max($W_l$) and min($W_h$), and also is essentially the mean of the two numbers  as in Fig. 3 (b). Note that $r \\cdot n(W)$ and $(1-r) \\cdot n(W)$ are to find out $W_l$ and $W_h$ using TopK, so practically the same as $|W_h|$ and $|W_l|$ (i.e., the number of weights).\n\nThe example on the right side in Fig. 4 (a) also illustrates the process of finding $t=0.08$ by taking the median (or mid-point) of 0.09 and 0.07.\n\n \n\n**Q2: In Table 6, the performance of LNM seems to be quite different from that reported in the original paper. Please explain this.**\n\nThank you for the opportunity to clarify the LNM results.  We believe it could be due to two reasons:\n* It is because of different dense baseline models as starting points. For example of ResNet50, LNM used a dense model with 77.3% top-1 accuracy but this starting checkpoint is not publicly available. Hence, we used the public dense model (in torchvision) with 76.1% top-1 accuracy. Such difference in the dense model accuracy contributes to the differences in the accuracies from LNM.\n\n* LNM is using a custom color augmentation according to the public repo. Hence, we removed this feature to keep the data augmentation normalized. "}, {"review_id": "ANXlVJ8VvS", "reviewer": "Reviewer_v4x5", "summary": "This paper proposes a parameter-free differentiable pruning, which uses a dynamic function of weights during training to generate soft pruning masks. It can generalize well on random/structured/channel pruning on both vision and NLP tasks. It achieves superior pruning results.", "questions": "see the weakness section.", "limitations": "The main limitation of the paper is that, aside from empirical experimental results, there is no theoretical proof or explanation as to why it is superior to existing non-parameter-free approaches. The solution presented in the paper raises doubts about its effectiveness on larger neural networks, such as ViT and LLMs.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed PDP approach is novel and interesting, and can generalize well on random/structured/channel pruning.\n2. The analysis in Section 3.1 is interesting and provides valuable insights.\n3. The proposed PDP approach can be faster than existing differentiable pruning approaches, with smaller pruning accuracy loss.", "weaknesses": "1. Instead of learning masks, which would introduce extra trainable parameters, PDP lets the weights of the network adjust themselves and generate soft masks. This approach may seem counterintuitive, as most weights are obtained and converged through expensive training processes. It may be more effective to adjust and learn masks rather than adjusting network weights to generate masks.\n\n2. The paper only provides a simple explanation for why PDP is more effective. A theoretical proof may be more convincing.\n\n3. Although the method is effective on the CNN and BERT models evaluated in this paper, its effectiveness on other models, such as larger vision transformers and LLMs, is still unclear.\n\n4. An important related work[1] is missing.  \n\n[1]Xia, Mengzhou, Zexuan Zhong, and Danqi Chen. \"Structured pruning learns compact and accurate models.\" ACL (2022).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q0:  This approach may seem counterintuitive, as most weights are obtained and converged through expensive training processes. This approach may seem counterintuitive, as most weights are obtained and converged through expensive training processes. It may be more effective to adjust and learn masks rather than adjusting network weights to generate masks.**\n\nThank you for the opportunity to reiterate our key contribution. Our paper in nutshell demonstrates that adjusting weights to generate masks is possible and cheaper than the conventional approach (i.e., learn/adjust masks based on the weight values), which is our key contribution. Although it may sound counterintuitive, our comprehensive experimental results support our contributions.\n\n**Q1: The paper only provides a simple explanation for why PDP is more effective. A theoretical proof may be more convincing.**\n\nThank you for your feedback. Here, we present more explanation on PDP.\n\nFor a given gradient on $ \\Delta \\hat w$, the gradient on $w$ with magnitude-based pruning can be computed as follows:\n\n$ \\Delta w_{mag} = \\Delta \\hat w \\cdot m(w) $\n\nIn PDP case, the gradient on $w$ is \n\n$ \\Delta w_{pdp} = \\Delta w_{mag} + 2 \\Delta \\hat w  \\cdot w^2 m(w)(1-m(w))  $ \n\nFrom above, we can observe the followings:\n* $ \\Delta w_{pdp} \\ge  \\Delta w_{mag}$: This implies PDP allows the weight to learn more from each iteration (larger gradient toward loss-decreasing direction), proportionally to its proximity to the $t$.\n* $ \\Delta w_{pdp}$ is max when $m(w) \\approx 0.5$: This means PDP encourages the weight on the pruning boundary (around $t$) to learn more aggressively, such that it can converge to the better spot faster.\n\nWe conjecture these features enable PDP to make a better pruning decision for the weight on the boundary, and provide strong \"second\" chances to recover from the undesirable weight updates.\n\n**Q2: Although the method is effective on the CNN and BERT models evaluated in this paper, its effectiveness on other models, such as larger vision transformers and LLMs, is still unclear.**\n\nThank you for the comment. Due to time and resource constraints, we could not experiment with a larger LLM. Yet, the GPT2 results in Table 3 indicate that PDP can offer a better trade-off between LLM training cost and LLM accuracy. Hope this could address some of your concerns.\n\n**Q3: An important related work[1] is missing.**\n\nThanks for the new reference. We will cite the paper in the updated draft."}]

Author rebuttal:
We like to thank the reviewers and ACs for the help and feedback. The highlight of our rebuttal includes the following.

New experimental results:
- Peak memory consumption is measured with PDP and OptG on a small GPT2 model.
- PDP result with MobileNet-v3 and ImageNet1k is added.
- Latency benefit with unstructured sparsity is measured on a modern smartphone.
- The efficacy of the proposed soft mask is studied with ResNet50 and MobileNet-v1 on ImageNet1k.
- PDP and other techniques have been experimented with the QQR dataset of GLUE benchmark.

New analysis:
- A theoretical aspect of PDP is presented.
- PDP is explained in the context of SAD.


Meta-review: Although the reviewers highlighted several strengths of the paper, such as the interest of the problem tackled, the novelty of the method, the good discussion of the literature, and the soundness of the experiments, they also initially raised some concerns regarding the theoretical analysis of the method, some aspects of the experiments, and the clarity of some parts of the paper. The authors' feedback was well received by the reviewers, who eventually reached a consensus for acceptance. The AC nonetheless recommends the authors to update their manuscript based on their feedback and double-check the writing as mentioned by Reviewer tv9j.
Decision: Accept (poster)