Title: PDP: Parameter-free Differentiable Pruning is All You Need
Venue: NeurIPS 2023
Accepted: yes
Keywords: pruning

cnn

transformers
Abstract: DNN pruning is a popular way to reduce the size of a model, improve the inference
latency, and minimize the power consumption on DNN accelerators. However,
existing approaches might be too complex, expensive or ineffective to apply to
a variety of vision/language tasks, DNN architectures and to honor structured
pruning constraints. In this paper, we propose an efficient yet effective train-time
pruning scheme, Parameter-free Differentiable Pruning (PDP), which offers state-
of-the-art qualities in model size, accuracy, and training cost. PDP uses a dynamic
function of weights during training to generate soft pruning masks for the weights
in a parameter-free manner for a given pruning target. While differentiable, the
simplicity and efficiency of PDP make it universal enough to deliver state-of-the-art
random/structured/channel pruning results on various vision and natural language
tasks. For example, for MobileNet-v1, PDP can achieve 68.2% top-1 ImageNet1k
accuracy at 86.6% sparsity, which is 1.7% higher accuracy than those from the
state-of-the-art algorithms. Also, PDP yields over 83.1% accuracy on Multi-Genre
Natural Language Inference with 90% sparsity for BERT, while the next best from
the existing techniques shows 81.5% accuracy. In addition, PDP can be applied to
structured pruning, such as N:M pruning and channel pruning. For 1:4 structured
pruning of ResNet18, PDP improved the top-1 ImageNet1k accuracy by over 3.6%
over the state-of-the-art. For channel pruning of ResNet50, PDP reduced the top-1
ImageNet1k accuracy by 0.6% from the state-of-the-art.

PDP: Parameter-free Differentiable Pruning is All You Need

Abstract

DNN pruning is a popular way to reduce the size of a model, improve the inference latency, and minimize the power consumption on DNN accelerators. However, existing approaches might be too complex, expensive or ineffective to apply to a variety of vision/language tasks, DNN architectures and to honor structured pruning constraints. In this paper, we propose an efficient yet effective train-time pruning scheme, Parameter-free Differentiable Pruning (PDP), which offers stateof-the-art qualities in model size, accuracy, and training cost. PDP uses a dynamic function of weights during training to generate soft pruning masks for the weights in a parameter-free manner for a given pruning target. While differentiable, the simplicity and efficiency of PDP make it universal enough to deliver state-of-the-art random/structured/channel pruning results on various vision and natural language tasks. For example, for MobileNet-v1, PDP can achieve 68.2% top-1 ImageNet1k accuracy at 86.6% sparsity, which is 1.7% higher accuracy than those from the state-of-the-art algorithms. Also, PDP yields over 83.1% accuracy on Multi-Genre Natural Language Inference with 90% sparsity for BERT, while the next best from the existing techniques shows 81.5% accuracy. In addition, PDP can be applied to structured pruning, such as N:M pruning and channel pruning. For 1:4 structured pruning of ResNet18, PDP improved the top-1 ImageNet1k accuracy by over 3.6% over the state-of-the-art. For channel pruning of ResNet50, PDP reduced the top-1 ImageNet1k accuracy by 0.6% from the state-of-the-art.

1 Introduction

Deep neural networks (DNN) have shown human performance on complex cognitive tasks [46], but their deployment onto mobile/edge devices for enhanced user experience (i.e., reduced latency and improved privacy) is still challenging. Most such on-device DNN systems are battery-powered and heavily resource-constrained, thus requiring high power/compute/storage efficiency [22, 48, 50, 53].

Such efficiency can be accomplished by mixing and matching various techniques, such as designing efficient DNN architectures like MobileNet/MobileViT/ MobileOne [36, 43, 48], distilling a complex DNN into a smaller architecture [41], quantizing/compressing the weights of DNNs [8, 18, 23, 31, 39, 58], and pruning near-zero weights [28, 34, 40, 44, 52, 56, 57, 61]. Also, pruning is known to be highly complementary to quantization/compression [51] when optimizing a DNN model. Training a larger model and then compressing it by pruning has been shown to be more effective in terms of model accuracy than training a smaller model from the beginning [32]. However, pruning comes at the cost of degraded model accuracy, and the trade-off is not straightforward [28].

Hence, a desirable pruning algorithm should achieve high accuracy and accelerate inference for various types of networks without significant training overheads in costs and complexity. In this

a Soft threshold weight re-parameterization for learnable sparsity [28].

b Fire Together Wire Together: A Dynamic Pruning Approach with Self-Supervised Mask Prediction [11]

c Winning the Lottery with Continuous Sparsification [45]

d Sparse training via boosting pruning plasticity with neuro-regeneration [34].

e Optimizing gradient-driven criteria in network sparsity: Gradient is all you need [57].

f AC/DC: alternating compressed/decompressed training of deep neural networks [40].

g Movement Pruning: Adaptive Sparsity by Fine-Tuning [44].

h

Prune Once for All: Sparse Pre-Trained Language Models [56].

Table 1: Comparison of the state-of-the-art pruning schemes and PDP: PDP can explore a good trade-off bewteen accuracy and inference speed without introducing new learnable parameters and with a simple/fast training flow.

work, we propose a simple yet effective pruning technique, Parameter-free Differentiable Pruning or PDP, which uses a dynamic function of weights to generate soft pruning masks for the weights themselves. PDP requires neither additional learning parameters [57] nor complicated training flows [40], yet offers precise control on the target sparsity level [28], while pushing the state-of-thearts in random/structured/channel pruning. The soft masks from PDP make pruning differentiable and let training loss decide whether/how each weight will be pruned [42]. Table 1 compares PDP with the latest state-of-the-art pruning schemes, and our major contributions include:

PDP outperforms the state-of-the-art schemes on a variety of models and tasks. Being differentiable and parameter-free, PDP offers efficient and effective pruning without complex techniques.

PDP offers a universal and holistic approach for efficient random/structured/channel pruning, while delivering a high-quality model optimization for a given pruning target.

With a dynamic function of weights, PDP generates a soft pruning mask which does not need training, and thus does require neither gradient synchronization nor SGD-update.

2 Related Works

Pruning in DNN incurs a complex trade-off between model accuracy and inference speed in terms of MAC (mult-add operations) [28]. A weight can contribute differently to the model accuracy, depending on the number of times it is used for prediction (i.e., a weight in convolution filter for a large input) and the criticality of the layer it belongs to (i.e., a weight in a bottleneck layer). Therefore, even if two models are pruned to the same level, the accuracy and inference speed of each can be vastly different, which makes exploring the best trade-off challenging yet crucial in DNN pruning. A body of work in random and structured pruning has been proposed to optimize such a trade-off. Please see Fig. 8 and Section B in Appendix for additional reviews and details.

Differentiable techniques gained popularity due to the advances in network architecture search methods as well as network compression techniques [6, 8, 33], and is a powerful technique for random/structured pruning as it drives pruning based on task loss. However, differentiable pruning does not necessarily deliver the best quality pruning (i.e., many state-of-the-art results are from non-differentiable methods), because the computational overhead and training complexity can limit its benefit. Existing differentiable pruning methods can be classified into a few approaches.

Learning pruning budget allocation is to determine target sparsity for each layer in a differentiable way, rather than computing pruning masks. Optimizing and training a per-layer pruning threshold with ReLU based on dynamic re-parameterization was proposed to allocate the sparsity across all layers in STR [28], which shows good model accuracy and low inference overhead. DSA [38] finds the layer-wise pruning ratios in a differentiable fashion by computing a channel-wise keep probability drawn from the Bernoulli distribution and keeping the expected sparsity ratio satisfied.

(a) Using a trainable threshold

(b) Using soft masks with

t

(see Section 3.1)

0.18

0.19

0.19

0.20

0.21

0.22

0.25

0.30

wo

W1

W3

=

W4

W5

W6

W7

：

threshold:0.205

-....

0

0

0

0

1

Training

mo

m

m

m3

m4

m5

m6

m

WeightUpdate

0.18

0.19

0.19

0.20

0.19

0.21

0.23

0.31

wo

W1

W3

W4

-

W5

W6

W7

threshold: 0.202

I

- - -

0

0

0

0

0

1

mo

m

m2

m3

m4

m5

m6

m

0

0

0.21

0.23

0.31

wo

W

W

W

W4

W5

W6

W

0.18

0.19

0.19

0.20

I

0.21

0.22

0.25

0.30

1

Wo

W1

W3

W4

W5

W6

W7

t : 0.205

0.20

0.25

0.25

0.49

0.51

0.60

0.85

0.90

Training

mo

m

m

m3

m4

m5

m6

m

WeightUpdate

0.18

0.20

0.22

0.22

0.19

0.21

0.23

0.31

Wo

W1

W3

W4

W5

W6

W

t:0.215

0.19

0.45

0.51

0.51

0.24

0.49

0.58

0.90

mo

m

m

m3

m

m5

m6

m

Test

0

0

0.22

0.22

0

0

.....

0.23

0.31

Wo

W3

W4

W'5

W6

W7

Learning pruning mask as an extra parameter is a popular way for differentiable pruning. In CS [45], a new trainable mask parameter is learned to continuously remove sub-networks based on the lottery hypothesis [13]. In detail, the new mask parameter is first multiplied by a scheduled temperature parameter, and then L0-regularized inside the Sigmoid function to continuously increase the sparsity. OptG [57] proposed learning a mask parameter based on the loss changes due to pruning a particular weight, which is proportional to the gradient and the magnitude of the corresponding masked weight when approximated with STE [5]. AutoPrune [54] also introduces extra learnable parameters (controlled by a regularizer) to generate a differentiable/approximated pruning mask using STE [5]. In [42], new learnable parameters, α are introduced and L 1 -regularized to enforce sparsity. Then, α is compared with a hyper-param t to derive the masks for the model parameters (i.e., prune if α < t ). As such comparison is not differentiable, comparison is approximated into a foothill function [4]. Learning channel pruning mask is proposed in SCP [25] based on the operation-specific observation that a feature map with a large and negative batch mean will get zeroed by ReLU .

Generating pruning masks is proposed in FTWT [11] where new layers are attached to the exiting main architecture. The attached new layers are fed with activations from the main network, then trained to generate masks for the weights in the main network. Thus, although it does not learn masks directly, it still requires new trainable parameters to learn how to generate masks, not to mention the changes to the model network. Using knowledge distillation to generate pruning masks is explored in NISP [55] and DCP [62], where a fully trained teacher network guides the mask generation in a way that the distortion to the layer responses can be minimized. A differentiable Markov process is studied to generate a channel pruning mask probabilistically, where a state represents a channel and the transitions from states account for the probability of keeping one channel unpruned [16].

3 PDP: Parameter-free Differentiable Pruning

Complex pruning schemes do not always yield the best quality results, and their complexity and cost can make them impractical and difficult to use. The proposed Parameter-free Differentiable Pruning (PDP) is a highly effective and efficient scheme that generates soft pruning masks using a dynamic function of weights in a parameter-free and differentiable fashion. Since PDP is differentiable, the task loss can directly guide the pruning decision, offering an effective pruning solution. Simultaneously, being parameter-free, PDP can be fast and less intrusive to the existing training flow. Overall, PDP finds a weight distribution that is best for task loss and pruning. Instead of having extra parameters, PDP indirectly influences the weight distribution for high-quality pruning. For example, if a weight

(a) PDP training flow

(b) Probability plot by

w

∈

R

[0

,

1]

with

t

= 0

.

6

GradientAggregation andWeightUpdate

0

r·n(W)

t

(1-r)·n(W)

8

W1

Wh

Wh = topK(abs(W),(1 - r) · n(W))

Capture Weight

W, =bottomK(abs(W),r· n(W))

Distribution

1.0

t = 0.5{min(Wh) +max(W)}

Probability

0.7

0.5

-m(w)

[t²

0.2

(m)z...

Forward/Backward

[z(w),m(w)]=softmax(

withMaskedWeight

2

0.0

w=m(w)·w

0

0.2

0.4

0.6

0.8

M

w is destined to be pruned for some reason, instead of having a new parameter to denote "to-prune" , PDP lets SGD gradually make w itself smaller relatively against other parameters in the same entity, increasing its chance to be pruned over time. We will first present PDP in Section 3.1, provide in-depth analysis in Section 3.2, discuss the benefits of PDP over existing differentiable pruning approaches in Section 3.3, and then show the extension to structured and channel pruning in Section 3.4.

3.1 PDP Algorithm

To address the drawbacks of existing differentiable pruning algorithms, we propose PDP. A soft mask should ideally represent the chance of a weight w being in one of two symbolic states, "to-prune" (noted as Z ) or "not-to-prune" (noted as M ), without requiring extra parameters or expensive bookkeeping. While the chance of w being in either state is not straightforward to compute, PDP generates a soft mask based on the fact that there exists an equal chance point for both states. Let us consider differentiable functions, z, m : R [0 , ∞ ] ↦→ R [0 , 1] , to compute the chances of being in Z and M , respectively, which must satisfy the following conditions as a soft mask for magnitude-based pruning:

z ( | a | ) < z ( | b | ) for | a | > | b | : a weight with a smaller magnitude has a higher chance to be in Z .

m ( | a | ) > m ( | b | ) for | a | > | b | : a weight with a larger magnitude has a higher chance to be in M .

z ( w ) + m ( w ) = 1 for any w : the total probability is 1.

Then, by the monotonicity and continuity, there exists t ∈ R ≥ 0 such that z ( t ) = m ( t ) = 0 . 5 (the equal chance for Z and M ), which leads to the following boundary conditions on the left. Any function that satisfies these conditions can be used to compute m ( w ) as a soft mask of w for train-time pruning. Let denote that topK ( X,k ) /bottomK ( X,k ) is selecting the largest/smallest k elements from a matrix X , abs ( X ) is an element-wise absolute operation, and n ( X ) returns the element count. In PDP, we uniquely identify t for a given prune ratio r ∈ [0 , 1) for a layer with a weight matrix W as in Fig. 2 (a) based on the pruning context.

The sparsity r for each W can be easily obtained by sorting the weights from the network per magnitude after a few epochs w.r.t the global target sparsity as a one-time task, by a pruning budget allocation algorithm [14, 28], or set manually by a user.

Right after the SGD weight update, t is computed for the weights W in each layer or entity. The role of t is to abstract the current weight distribution of each layer/entity for pruning.

During forward-pass, a soft mask, m ( w ) for the weight w is computed and then the masked weight ˆ w is applied. τ is the temperature parameter (see Section E in Appendix for details).

Computing t and generating ˆ w repeat iteratively to adapt to the updated weight distribution.

Figure 2 (b) shows how the value of t is obtained in PDP and a soft mask is computed. Specifically, t is set to the value that is halfway between the largest pruned weight and the smallest unpruned weight when a hard mask is applied for a given sparsity ratio r . This ensures that each weight has an equal probability of being pruned or kept. As a result, PDP satisfies all the constraints for z and m . More details on PDP training flow are in Section D in Appendix.

PDP uses a dynamic function of t to generate soft pruning masks of W without the need for any extra trainable parameters. Instead, PDP lets the weights of the network adjust themselves such that the information that would otherwise be learned by the extra trainable parameters is instead fused into the weights themselves and their distribution. This is possible because each weight w is not only a coefficient in a layer, but also an indicator of the relative chance of that weight being pruned against the other weights in W . This relative chance is determined by the value of t , as shown in Figure 2 (b). Overall, PDP looks simple but is shown to be quite effective, which opens to broad applicability.

3.2 PDP Gradient Analysis

To better understand how PDP helps pruning decision, we derive the gradient of the masked weight ˆ w . As in Fig. 2 (a), ˆ w is the following:

Then, the gradient of w, ∆ w can be simplified as the next:

where we can make the following observations:

The 1st term is a typical gradient in mask-based pruning, and the 2nd term is an additional gradient with a positive factor (i.e, w 2 τ m ( w ) { 1 -m ( w ) } ≥ 0 ) from PDP.

If m ( w ) ≈ 0 or hard-prune, ∆ w ≈ 0 , which is true for any pruning algorithm.

If m ( w ) ≈ 1 or hard-not-to-prune, ∆ w ≈ ∆ˆ w , which is true for any pruning algorithm.

When m ( w ) ≈ 0 . 5 (i.e., pruning decision is unclear), m w (1 -m w ) is maximized and accordingly ∆ w is too, boosting the w movement.

τ serves as an inverse scaling factor for the boosted gradient.

Hence, PDP will accelerate the SGD updates for the weights near the pruning boundary (i.e., t ) toward a loss-decreasing direction, which can encourage the weights to settle with the proper pruning decision at the end. Even if the current gradient is not globally beneficial for the task, many second chances will eventually help recover the damages in an accelerated manner. We conjecture these features enable PDP to make a better pruning decision for the weight on the boundary.

3.3 PDP vs. Existing Differentiable Pruning Strategies

Learning Pruning Budget Allocation focuses on obtaining a pruning threshold in a differentiable way based on marginal loss or L 1 -regularization [25, 38]. Meanwhile, PDP directly generates a soft pruning mask for each weight also in a differentiable way. Fig. 1 illustrates the differences with an example. Learning a pruning threshold is helpul for global pruning budget allocation, as the threshold gets adjusted per the task loss as in Fig. 1 (a), but has the following drawbacks:

Sparsity level is hard to control, as the pruning threshold is not explicitly related to the sparsity, as masks and the thresholds are not directly co-optimized. In Fig. 1 (a), after one weight update, the threshold is reduced from 0.205 to 0.202, but the sparsity is increased from 50% to 66% as w 4 becomes smaller than the threshold.

Once a weight gets pruned during training, it does not get updated as the gradient is masked out. The bottom weights w { 0 , 1 , 2 , 3 } get no weight update due to the zero masks, and remain pruned.

On the contrary, PDP allows all the weights to be updated through soft masks as in Fig. 1 (b), providing higher flexibility and recovery from undesirable pruning [17]. For example, consider the weight w 2 of a value 0.19 which would have been permanently pruned in (a). PDP discourages a weight from being permanently pruned through the weight update, w 2 still receives a scaled-down gradient. If a near-zero weight continues to get negative gradients over time (although scaled down by the soft mask), it can eventually get unpruned at the end of training. Similarly, if a very large weight gets positive gradients many times enough to be near zero, it will get pruned eventually, even

(a) PDP: epoch 20

(b) PDP: epoch 30

(c) PDP: epoch 40

(d) PDP: epoch 50

(e) PDP: epoch 60

(f) PDP: epoch 70

(g) PDP: epoch 80

(h) PDP: epoch 90

(i) STR: epoch 20

(j) STR: epoch 30

(k) STR: epoch 40

(l) STR: epoch 50,70,80,90

5

■

！

HC

1n.8056

LH

if it was not pruned in the early stage. Hence, such gradual pruning decision over time during training allows PDP to make better pruning decisions w.r.t. the task loss.

Such soft masking allows pruning decisions to be flipped multiple times during the entire training process, and such effects are compared between PDP and STR [28] (from Table 1) in Fig. 3 where the pruning decisions for the first Conv2d (3x3 kernels with 32 output channels) in MobileNet-v1 for the end-to-end ImageNet1k training are captured. The white cell means that the pruning decision has been flipped (from to-prune to not-to-prune or vice-versa), while the black cell means the decision has never changed, during the given epoch. Then, we can observe from Fig. 3 that while PDP in (a)-(h) continues to flip the decisions until the very late training stage, STR in (i)-(l) finalizes the pruning decisions early (i.e., in the first 30 epochs).

Learning/Generating Pruning Masks with Extra Parameters allows the pruning decision to be driven by a task loss through back-propagation rather than the weight value itself (i.e., a hard mask will zero out the gradient of a pruned weight), thus addressing the problems with differentiable pruning budget allocation, but comes with the following issues.

A pruning mask is (or derives from) a learnable parameter. Hence the number of total trainable parameters increases significantly, making the training process slow and complex [11, 44, 45, 57].

Ahard mask is approximated into a soft mask using a differentiable function, without guaranteeing the key properties of a pruning mask, such as the [0,1] value range or monotonicity [42].

Table 2 compares parameter-free PDP with a differentiable mask pruning scheme, CS [45] from Table 1 (see Table 8 in Appendix for detail). The results show that PDP outperforms CS and is 2.8 faster, without adding extra trainable parameters or complicating the training flow. Table 3 also demonstrates being parameter-free can provide the substantial training efficiency gains for very large language models like GPT. When trained and sparsified on 32 GPUs in mixed-precision (to fit OptG into GPU memory) using the recipe in [1], PDP delivers the best perplexity at 75% sparsity with 42% lower cost than OptG [57].

(a) 1:4 pruning (75% sparsity)

(b) Channel pruning (50% sparsity)

Weights

0.18

0.20

0.01

0.10

0.07

0.32

0.09

0.12

Channels

Norms

t=0.35

Masks

PDP

PDP

0.18

0.20

0.01

0.10

0.29

0.41]

19

0.34

Wn = [0.18,0.20,0.10]

Wh = [0.32,0.12,0.09]

0.15

0.07

0.09

0.02

0.19

PDP

.29,0.

0.19

masks

W, = [0.01]

t=0.06

W = [0.07]

t=0.08

0.02

0.52

0.39

0.01

0.65

0.81

II

0.58

0.68

0.1

0.21

0.13

0.88

0.15

0.28

0.22

0.12

0.19

0.26

0.41

WW

0.66

1

3.4 PDP for Structured and Channel Pruning

The simplicity and non-intrusive nature of PDP make it readily applicable to differentiable structured and channel pruning. As an example of structured pruning, we consider N:M pruning, where only N weights are kept out of every M consecutive weights. N:M pruning is attracting high research and industrial attention because top-of-the-line GPUs support 2:4 configuration [24]. To apply PDP to N:M pruning, we apply it to every M consecutive weights of the layer, as if the layer were composed of many sub-layers, each with M weights, which is illustrated in Figure 4 (a). Since N:M dictates the target sparsity, we can easily find the threshold t and generate the soft mask, as shown in Figure 2 (a).

Channel pruning is another type of structured pruning that can be easily applied to PDP with minor modifications. To do this, we first compute the L 2 norm of each channel in the layer, and then use these norm values (in place of the absolute values of the weights in Figure 2(a)) to compute a soft mask for each channel, which is depicted in Figure 4 (b). Using the soft mask to prune all the corresponding weights in the channel will make the channel pruning process differentiable.

4 Experimental Results

We compared our PDP with state-of-the-art random, structured, and channel pruning schemes on various computer vision and natural language models. We used two x86 Linux nodes with eight NVIDIA V100 GPUs on each in a cloud environment. All cases were trained from scratch. More experimental results and the hyper-parameters are in Section F and Table 8 in Appendix.

Random Pruning for Vision Benchmark: We compared the proposed PDP with the latest prior arts, STR [28], GMP [61], DNW [52], GraNet [34], OptG [57], and ACDC [40] on ResNet18, ResNet50, MobileNet-v1, and MobileNet-v2 [20, 22, 43] with the ImageNet1k dataset [9]. Since all of these schemes have been experimented only with ResNet50 and/or MobileNet-v1, we reproduced the pruning results in our controlled environment with the identical data augmentations by running the official implementations from the authors [28, 34, 40, 57] or verified implementations from the prior arts [52, 61] as in Section G in Appendix. Since the primary goal of pruning is to trade-off the model accuracy with the compute reduction as in Section 2, we measured the accuracies and Multiply-Accumulate Operation (MAC) during inference on each experiment with layer fusion (i.e., BatchNorm folding), and mainly focused on the high-sparsification cases. Note that the MAC is purely theoretical and reported to understand the trade-off among accuracy, size, and compute across

(a) ResNet18 (sparsity: 85.5%)

(b) ResNet50 (sparsity: 89.8%)

(c) MobileNet-v1 (sparsity: 86.6%)

(d) MobileNet-v2 (sparsity: 80.2%)

550

880

520

GraNet

ACDC

GraNet

490

6

e

800

ACDC

C

A

460

430

PDP-base+

C

720

M

400

。

M

A

640

ferernce

370

340

STR

PDP-base

rence

560

PDP-base 0

PDP-base+

310

fere

480

280

250

OptG

PDP-optg"

400

OptG

STR

PDP-str

PDP-optg

220

320

66%

67%

68%

69%

70%

72%

73%

74%

75%

Top-1 Accuracy

Top-1 Accuracy

150

110

140

130

GraNet

ACDC

6

100

GraNet

ACDC-

e

C

A

120

110

PDP-optg

MA

C

90

PDP-base

M

100

80

OptG

ference

90

80

OptG-

Inference

70

PDP-optg

70

PDP-base

60

60

STR

50

50

STR

40

40

61%

63%

65%

67%

69%

56%

60%

64%

68%

Top-1 Accuracy

Top-1 Accuracy

various algorithms, but it still captures the latency benefit on some hardware platforms (see C in Appendix for details). In our experiments with ImageNet1k, all layers, including both the first and last layers, are pruned without any restriction. Also, we estimated the algorithmic overhead and training efficiency by the total monetary cost for GPUs on commercial cloud spot instances [7].

We applied not only the proposed hyper-parameters from the authors but also a set of further finetuned hyper-parameters for competing methods (see Table 8 in Appendix for details). Also, since each algorithm used a different number of epochs and showed results at different sparsity levels, a) we ran STR first to set the target sparsity levels for all the networks for fair comparisons, because all other schemes can control the sparsity level precisely, b) we trained ResNet18/50 for 100 epochs and MobileNet-v1/v2 for 180 epochs following [28, 34, 40, 57] except STR (which diverged with more epochs for MobileNet-v1/v2). For PDP , we fixed the target sparsity for each layer based on the global weight magnitude at the epoch 16 and started pruning at the rate of 1.5% of the target sparsity per epoch for all the experiments which correspond to s = 16 and ϵ = 0 . 015 in Algorithm 1 in Appendix. For detailed experiment configurations, please refer to Table 8. Every experiment began with a randomly initialized model (i.e., no pre-trained model). For PDP , we had the following variants to show the value of PDP with the same training overhead or per-layer pruning budgets.

PDP-base globally computes the target sparsity by abs ( W ) at epoch 16 across all the layers.

PDP-base+ is the same as PDP-base yet with more epochs to match the GPU cost of ACDC .

PDP-str/optg uses the per-layer sparsity from STR/OptG as input to normalize the MAC.

Our experimental results are highlighted in Fig. 5, where the size of circles indicates the relative training overhead due to pruning. Note that we used only one single node with 8 GPUs due to the limitation in the official implementations for GraNet and OptG , thus both have the advantage of not having the inter-node communication cost. Also, each approach imposes a different level of training-time overhead, mainly due to the various complexities of training flow and pruning itself as captured in Fig. 5. Overall results can be summarized as follows:

PDP showed the best the model accuracy: PDP-base on ResNet18 delivered 69% Top-1 accuracy which is superior to other schemes but at higher MAC than only STR and OptG .

$ the GPU cost ($) is based on a commercial cloud spot instance pricing.

* used only one with 8 GPUs due to the limitations in the public code.

♢ when PDP applied for MobileNet-v3 (5.5M parameters) with an 80% sparsity target, we achieved 71.5% top-1 ImageNet1K accuracy, which is 2.5% down from the dense version.

Table 4: PDP compared with other unstructured pruning algorithms on ImageNet1K shows the best trade-off among accuracy, inference MAC, and training overheads. More results are available in Section F in Appendix.

PDP offered the better model accuracy for a given pruning target: With the custom sparsification target for each layer, PDP-str/optg demonstrated the 2-3% higher Top-1 accuracy at the same MAC, demonstrating the effectiveness of the proposed method.

When we use the similar GPU budget for additional epochs with ACDC which is noted as PDP-base+ , our method further improved the Top-1 accuracy from 69% to 69.5% for ResNet18 and from 74.7% to 75.3% for ResNet50 with slight fewer MACs.

Random Pruning for NLP Benchmark: We compared PDP with the state-of-the-art pruning results from MVP [44] and POFA [56] (quoted from the respective papers) in addition to OptG and STR (reproduced in our environment) on a BERT model [10] for the two largest NLP tasks of the GLUE benchmark, MNLI (Multi-Genre Natural Language Inference) with 392,702 samples and QQP (Question-answering Natural Language Inference) with 363,836 samples. Following the setups in MVP and POFA , we used the same batch size 32 per GPU (i.e., global mini-batch size of 512), excluded the embedding and the last linear layer from pruning, and trained the bert-base-uncased from HuggingFace from scratch with self-distillation. The teacher for the distillation is from the best checkpoint in the first 40 epochs. We use the weights of 0.95 on the distillation loss and 0.05 on the task loss for PDP , and 0.75 on the distillation loss and 0.25 on the task loss for STR . We could not use distillation for OptG , as it caused the out-of-memory error due to the extra-parameter overheads from both pruning and distillation. Our experimental results in Table 5 can be summarized as follows:

STR underperforms on all the best cases even though it could not achieve the target sparsity.

OptG shows better model accuracy only than GMP and worse than others in our setup.

PDP outperformed all other methods for both MNLI and QQP.

Structured/Channel Pruning for Vision Benchmarks: We compared PDP-driven N:M pruning and channel pruning on the ImageNet1k dataset [9]. For N:M pruning, we reproduced the LNM [59] results using the public code base but without the color augmentation to keep the experimental environment normalized. For PDP , we simply reused the hyper-parameters and configurations as in Table 8 in our Appendix, and the top-1 accuracies by various N:M configs with ImageNet1k on ResNet18/50 are presented in Table 6. We can observe that PDP outperforms LNM on all the test

Since STR cannot control the sparsity precisely, we report the metrics with our closest achieved sparsity levels, 85.9% for the 90% case and 91.8% for the 94% case.

* For LNM , we used the the public dense model from TorchVision for channel pruning, as the original baseline from [59] (which has higher accuracy) is not available publicly. For the example of ResNet50 on ImageNet1k, LNM used a dense model with 77.3% top-1 accuracy while the public dense model in TorchVision) yields 76.1% top-1 accuracy.

cases, even with 4x larger batch size in 20 fewer epochs. LNM training cost is also much higher than PDP because of its costly weight regularization and complex back-propagation scheme.

For channel pruning, we compared PDP with SCP [26], NISP [55], and DCP [62]. Note that SCP uses the β in BatchNorm to select the channels to prune (i.e., beta ≤ ϵ ), hence applicable to limited types of networks only. We again reused the hyper-parameters and configurations as in Table 8 in our Appendix for PDP , and the top-1 accuracy with ImageNet1k on ResNet50 is reported in Table 7. We can see that PDP can be used for channel pruning and show superior performance out of the box.

5 Conclusion

We showed that a simple and universal pruning method PDP can yield the state-of-the-art random/structured/channel pruning quality on popular computer vision and natural language models. Our method requires no additional learning parameters, yet keeps the training flow simple and straightforward, making it a practical method for real-world scenarios. We plan to extend our differentiable pruning into quantization, making both jointly differentiable and optimizable by the task-loss.

References

https://github.com/karpathy/nanoGPT .

https://apple.github.io/coremltools/docs-guides/source/pruning-overview. html .

S. Anwar, K. Hwang, and W. Sung. Structured pruning of deep convolutional neural networks. ACM Journal on Emerging Technologies in Computing Systems , 2017.

M. Belbahri, E. Sari, S. Darabi, and V. P. Nia. Foothill: A quasiconvex regularization for edge computing of deep neural networks. In F. Karray, A. Campilho, and A. C. H. Yu, editors, International Conference on Image Analysis and Recognition , 2019.

Y. Bengio, N. Léonard, and A. C. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR , 2013.

J. Chang, x. zhang, Y. Guo, G. MENG, S. XIANG, and C. Pan. Data: Differentiable architecture approximation. In Advances in Neural Information Processing Systems , 2019.

B. Chen, T. Dao, K. Liang, J. Yang, Z. Song, A. Rudra, and C. Ré. Pixelated butterfly: Simple and efficient sparse training for neural network models. In International Conference on Learning Representations , 2022.

M. Cho, K. Alizadeh-Vahid, S. Adya, and M. Rastegari. DKM: differentiable k-means clustering layer for neural network compression. In International Conference on Learning Representations , 2022.

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2009.

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics , 2019.

S. Elkerdawy, M. Elhoushi, H. Zhang, and N. Ray. Fire together wire together: A dynamic pruning approach with self-supervised mask prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022.

U. Evci, T. Gale, J. Menick, P. S. Castro, and E. Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning , 2020.

J. Frankle and M. Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations , 2019.

E. Frantar and D. Alistarh. SPDY: Accurate pruning with speedup guarantees. In International Conference on Machine Learning , 2022.

T. Gale, E. Elsen, and S. Hooker. The state of sparsity in deep neural networks. CoRR , 2019.

S. Guo, Y. Wang, Q. Li, and J. Yan. Dmcp: Differentiable markov channel pruning for neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , June 2020.

Y. Guo, A. Yao, and Y. Chen. Dynamic network surgery for efficient dnns. In Advances in Neural Information Processing Systems , 2016.

S. Han, H. Mao, and W. J. Dally. Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. In International Conference on Learning Representations , 2016.

S. Han, J. Pool, J. Tran, and W. J. Dally. Learning both weights and connections for efficient neural network. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems , 2015.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2016.

Y. He, X. Zhang, and J. Sun. Channel pruning for accelerating very deep neural networks. In The IEEE International Conference on Computer Vision , Oct 2017.

A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 , 2017.

B. H. J. Lee, D. Kim. Network quantization with element-wise gradient scaling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2021.

C. Y. Jeff Pool. Channel permutations for n:m sparsity. In Advances in Neural Information Processing Systems , 2021.

M. Kang and B. Han. Operation-aware soft channel pruning using differentiable masks. In International Conference on Machine Learning , 2020.

M. Kang and B. Han. Operation-aware soft channel pruning using differentiable masks. In International Conference on Machine Learning , 2020.

E. Kurtic, D. Campos, T. Nguyen, E. Frantar, M. Kurtz, B. Fineran, M. Goin, and D. Alistarh. The optimal bert surgeon: Scalable and accurate second-order pruning for large language models. arXiv preprint arXiv:2203.07259 , 2022.

A. Kusupati, V. Ramanujan, R. Somani, M. Wortsman, P. Jain, S. Kakade, and A. Farhadi. Soft threshold weight reparameterization for learnable sparsity. In International Conference on Machine Learning , July 2020.

F. Lagunas, E. Charlaix, V. Sanh, and A. M. Rush. Block pruning for faster transformers. In The Conference on Empirical Methods in Natural Language Processing , 2021.

H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf. Pruning filters for efficient convnets. In International Conference on Learning Representations , 2017.

Y. Li, X. Dong, and W. Wang. Additive powers-of-two quantization: An efficient non-uniform discretization for neural networks. In International Conference on Learning Representations , 2019.

Z. Li, E. Wallace, S. Shen, K. Lin, K. Keutzer, D. Klein, and J. E. Gonzalez. Train large, then compress: Rethinking model size for efficient training and inference of transformers. In International Conference on Machine Learning , 2020.

H. Liu, K. Simonyan, and Y. Yang. DARTS: differentiable architecture search. In International Conference on Learning Representations , 2019.

S. Liu, T. Chen, X. Chen, Z. Atashgahi, L. Yin, H. Kou, L. Shen, M. Pechenizkiy, Z. Wang, and D. C. Mocanu. Sparse training via boosting pruning plasticity with neuroregeneration. In Advances in Neural Information Processing Systems , 2021.

Z. Liu, P. N. Whatmough, Y. Zhu, and M. Mattina. S2ta: Exploiting structured sparsity for energy-efficient mobile cnn acceleration. In 2022 IEEE International Symposium on HighPerformance Computer Architecture (HPCA) , 2022.

S. Mehta and M. Rastegari. Mobilevit: Light-weight, general-purpose, and mobile-friendly vision transformer. In International Conference on Learning Representations , 2022.

A. K. Mishra, J. A. Latorre, J. Pool, D. Stosic, D. Stosic, G. Venkatesh, C. Yu, and P. Micikevicius. Accelerating sparse deep neural networks. CoRR , 2021.

X. Ning, T. Zhao, W. Li, P. Lei, Y. Wang, and H. Yang. DSA: more efficient budgeted pruning via differentiable sparsity allocation. In European Conference on Computer Vision , 2020.

E. Park and S. Yoo. Profit: A novel training method for sub-4-bit mobilenet models. In European Conference on Computer Vision , 2020.

A. Peste, E. Iofinova, A. Vladu, and D. Alistarh. AC/DC: alternating compressed/decompressed training of deep neural networks. In Advances in Neural Information Processing Systems , 2021.

A. Polino, R. Pascanu, and D. Alistarh. Model compression via distillation and quantization. In International Conference on Learning Representations , 2018.

R. K. Ramakrishnan, E. Sari, and V. P. Nia. Differentiable mask for pruning convolutional and recurrent networks. In International Conference on Computer and Robot Vision , 2020.

M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , June 2018.

V. Sanh, T. Wolf, and A. Rush. Movement pruning: Adaptive sparsity by fine-tuning. In Advances in Neural Information Processing Systems , 2020.

P. Savarese, H. Silva, and M. Maire. Winning the lottery with continuous sparsification. In Advances in Neural Information Processing Systems , 2020.

D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science , 362(6419):1140-1144, 2018.

H. Tanaka, D. Kunin, D. L. K. Yamins, and S. Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. In Advances in Neural Information Processing Systems , 2020.

P. K. A. Vasu, J. Gabriel, J. Zhu, O. Tuzel, and A. Ranjan. An improved one millisecond mobile backbone. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2023.

C. Wang, G. Zhang, and R. B. Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations , 2020.

K. Wang, Z. Liu, Y. Lin, J. Lin, and S. Han. Haq: Hardware-aware automated quantization with mixed precision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2019.

Y. Wang, Y. Lu, and T. Blankevoort. Differentiable joint pruning and quantization for hardware efficiency. In European Conference on Computer Vision , 2020.

M. Wortsman, A. Farhadi, and M. Rastegari. Discovering neural wirings. In Advances in Neural Information Processing Systems , 2019.

J. Wu, Y. Wang, Z. Wu, Z. Wang, A. Veeraraghavan, and Y. Lin. Deep k-means: Re-training and parameter sharing with harder cluster assignments for compressing deep convolutions. In International Conference on Machine Learning , 2018.

X. Xiao, Z. Wang, and S. Rajasekaran. Autoprune: Automatic network pruning by regularizing auxiliary parameters. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , 2019.

R. Yu, A. Li, C. Chen, J. Lai, V. I. Morariu, X. Han, M. Gao, C. Lin, and L. S. Davis. NISP: pruning networks using neuron importance score propagation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2018.

O. Zafrir, A. Larey, G. Boudoukh, H. Shen, and M. Wasserblat. Prune once for all: Sparse pre-trained language models. In Advances in Neural Information Processing Systems , 2021.

Y. Zhang, M. Lin, M. Chen, Z. Xu, F. Chao, Y. Shen, K. Li, Y. Wu, and R. Ji. Optimizing gradient-driven criteria in network sparsity: Gradient is all you need. In CoRR , 2022.

X. Zhao, Y. Wang, X. Cai, C. Liu, and L. Zhang. Linear symmetric quantization of neural networks for low-precision integer hardware. In International Conference on Learning Representations , 2019.

A. Zhou, Y. Ma, J. Zhu, J. Liu, Z. Zhang, K. Yuan, W. Sun, and H. Li. Learning n:m fine-grained structured sparse neural networks from scratch. In International Conference on Learning Representations , 2021.

H. Zhou, J. Lan, R. Liu, and J. Yosinski. Deconstructing lottery tickets: Zeros, signs, and the supermask. In Advances in Neural Information Processing Systems , 2019.

M. Zhu and S. Gupta. To prune, or not to prune: Exploring the efficacy of pruning for model compression. In Workshop, International Conference on Learning Representations , 2018.

Z. Zhuang, M. Tan, B. Zhuang, J. Liu, Y. Guo, Q. Wu, J. Huang, and J. Zhu. Discriminationaware channel pruning for deep neural networks. In Advances in Neural Information Processing Systems , 2019.

The hyper-parameters in Sections 3 and 4.

Table 8:

(a) ResNet18

(b) ResNet50

(c) MobileNet-v1

100%

80%

Sparsity

60%

40%

20%

0%

conv1

layer1.0.conv1

layer1.0.conv2

layer1.1.conv1

layer1.1.conv2

layer2.0.conv1

layer2.0.conv2

layer2.0.downsample.0

layer2.1.conv1

layer2.1.conv2

layer3.0.conv1

layer3.0.conv2

layer3.0.downsample.0

layer3.1.conv1

layer3.1.conv2

layer4.0.conv1

layer4.0.conv2

layer4.0.downsample.0

layer4.1.conv1

layer4.1.conv2

■PDP

■GraNet STR OptG

■ACDC

100%

80%

山

Sparsity

60%

40%

20%

0%

conv1

layer1.0.conv1

layer1.0.conv2

layer1.0.conv3

layer1.0.downsample.0

layer1.1.conv1

layer1.1.conv2

layer1.1.conv3

layer1.2.conv1

layer1.2.conv2

layer1.2.conv3

layer2.0.conv1

layer2.0.conv2

layer2.0.conv3

layer2.0.downsample.0

layer2.1.conv1

layer2.1.conv2

layer2.1.conv3

layer2.2.conv1

layer2.2.conv2

layer2.3.conv1

layer2.3.conv2

layer2.3.conv3

layer3.0.conv1

layer3.0.conv2

layer3.0.conv3

layer3.0.downsample.0

layer3.1.conv1

layer3.1.conv2

layer3.1.conv3

layer3.2.conv1

layer3.2.conv3

layer3.3.conv1

layer3.3.conv2

layer3.3.conv3

layer3.4.conv1

layer3.4.conv2

layer3.4.conv3

layer3.5.conv1

layer3.5.conv2

layer3.5.conv3

layer4.0.conv1

layer4.0.conv2

layer4.0.conv3

layer4.0.downsample.0

layer4.1.conv1

layer4.1.conv2

layer4.1.conv3

layer4.2.conv1

layer4.2.conv2

layer4.2.conv3

■

PDP

GraNet

STR

OptG

■ACDC

100%

80%

Sparsity

60%

40%

20%

0%

model.0.0

model.1.0

model.1.3

model.2.0

model.2.3

model.3.0

model.3.3

model.4.0

model.4.3

model.5.0

model.6.0

model.6.3

model.7.0

model.7.3

model.8.0

model.8.3

model.9.0

model.9.3

model.10.0

model.10.3

model.11.3

model.12.0

model.12.3

model.13.0

model.13.3

classifier

■PDPGraNet STROptGACDC

100%

80%

Sparsity

%09

40%

20%

0%

features.0.0

features.1.conv.0.0

features.1.conv.1

features.2.conv.0.0

features.2.conv.1.0

features.2.conv.2

features.3.conv.0.0

features.3.conv.1.0

features.3.conv.2

features.4.conv.0.0

features.4.conv.1.0

features.4.conv.2

features.5.conv.0.0

features.5.conv.1.0

features.5.conv.2

features.6.conv.0.0

features.6.conv.1.0

features.6.conv.2

features.7.conv.0.0

features.7.conv.1.0

features.7.conv.2

features.8.conv.0.0

features.8.conv.1.0

features.8.conv.2

features.9.conv.0.0

features.9.conv.1.0

features.9.conv.2

features.10.conv.0.0

features.10.conv.1.0

features.10.conv.2

features.11.conv.0.0

features.11.conv.1.0

features.11.conv.2

features.12.conv.0.0

features.12.conv.1.0

features.12.conv.2

features.13.conv.0.0

features.13.conv.1.0

features.13.conv.2

features.14.conv.0.0

features.14.conv.1.0

features.14.conv.2

features.15.conv.0.0

features.15.conv.1.0

features.15.conv.2

features.16.conv.0.0

features.16.conv.1.0

features.16.conv.2

features.17.conv.0.0

features.17.conv.1.0

features.17.conv.2

features.18.0

classifier.1

■PDP

GraNetSTROptG

■ACDC

(d) MobileNet-v2

80

60

Inference MAC (e6)

40

20

0

conv1

layer1.0.conv1

layer1.0.conv2

layer1.1.conv1

layer1.1.conv2

layer2.0.conv1

layer2.0.conv2

layer2.0.downsample.0

layer2.1.conv1

layer2.1.conv2

layer3.0.conv1

layer3.0.conv2

layer3.0.downsample.0

layer3.1.conv1

layer3.1.conv2

layer4.0.conv1

layer4.0.conv2

layer4.0.downsample.0

layer4.1.conv1

layer4.1.conv2

PDP

GraNet STR OptG

■ACDC

(a) ResNet18

80

60

Infererence MAC (e6)

40

20

0

Ilil,

d

山

ml

conv1

layer1.0.conv1

layer1.0.conv2

layer1.0.conv3

layer1.0.downsample.0

layer1.1.conv1

layer1.1.conv2

layer1.1.conv3

layer1.2.conv1

layer1.2.conv2

layer1.2.conv3

layer2.0.conv1

layer2.0.conv2

layer2.0.conv3

layer2.0.downsample.0

layer2.1.conv1

layer2.1.conv2

layer2.1.conv3

layer2.2.conv1

layer2.2.conv2

layer2.3.conv1

layer2.3.conv2

layer2.3.conv3

layer3.0.conv1

layer3.0.conv2

layer3.0.conv3

layer3.0.downsample.0

layer3.1.conv1

layer3.1.conv2

layer3.1.conv3

layer3.2.conv1

layer3.2.conv3

layer3.3.conv1

layer3.3.conv2

layer3.3.conv3

layer3.4.conv1

layer3.4.conv2

layer3.4.conv3

layer3.5.conv1

layer3.5.conv2

layer3.5.conv3

layer4.0.conv1

layer4.0.conv2

layer4.0.conv3

layer4.0.downsample.0

layer4.1.conv1

layer4.1.conv2

layer4.1.conv3

layer4.2.conv1

layer4.2.conv2

layer4.2.conv3

■PDP

GraNet

■STROptG

■ACDC

25

20

InferenceMAC (e6)

15

10

5

0

model.0.0

model.1.0

model.1.3

model.2.0

model.2.3

model.3.0

model.3.3

model.4.0

model.4.3

model.5.0

model.6.0

model.6.3

model.7.0

model.7.3

model.8.0

model.8.3

model.9.0

model.9.3

model.10.0

model.10.3

model.11.3

model.12.0

model.12.3

model.13.0

model.13.3

classifier

PDPGraNet

tSTROptG

■ACDC

15

Inference MAC (e6)

10

5

0

features.0.0

features.1.conv.0.0

features.1.conv.1

features.2.conv.0.0

features.2.conv.1.0

features.2.conv.2

features.3.conv.0.0

features.3.conv.1.0

features.3.conv.2

features.4.conv.0.0

features.4.conv.1.0

features.4.conv.2

features.5.conv.0.0

ilil

features.5.conv.1.0

features.5.conv.2

features.6.conv.0.0

lill

features.6.conv.1.0

features.6.conv.2

features.7.conv.0.0

features.7.conv.1.0

Ill.

features.7.conv.2

features.8.conv.0.0

features.8.conv.1.0

I11

features.8.conv.2

features.9.conv.0.0

features.9.conv.1.0

.l hh..hh

features.9.conv.2

features.10.conv.0.0

features.10.conv.1.0

features.10.conv.2

features.11.conv.0.0

features.11.conv.1.0

features.11.conv.2

features.12.conv.0.0

features.12.conv.1.0

features.12.conv.2

features.13.conv.0.0

features.13.conv.1.0

features.13.conv.2

features.14.conv.0.0

features.14.conv.1.0

features.14.conv.2

features.15.conv.0.0

features.15.conv.1.0

features.15.conv.2

features.16.conv.0.0

features.16.conv.1.0

features.16.conv.2

features.17.conv.0.0

features.17.conv.1.0

features.17.conv.2

features.18.0

classifier.1

■PDP

GraNet

■STROptG

■ACDC

(d) MobileNet-v2

Dense

GraNet

STR

OptG

ACDC

PDP

0.0

1.0

1.3

2.0

2.3

3.0

3.3

4.0

4.3

5.0

5.3

6.0

6.3

7.0

7.3

8.0

8.3

9.0

9.3

10.0

10.3

fc

17

A Training Configurations and Hyper-parameters

Since some techniques in Sections 3 and 4 require extra training parameters and pruning scheduling as shown in Table 1, we disclose the training configurations and hyper-parameters we found the best in Table 8.

B Trade-off in Pruning

Pruning for DNN requires exploring a good trade-off between model accuracy and inference latency under a given pruning target. Such a challenge can be elaborated with the MobiletNet-v1 Dense case in Fig. 8 where the following observations can be made:

The earlier layers have significantly fewer parameters than the later layers while still having comparable inference MACs as shown in (a). For example, the final classifier, which is a linear layer, has the lowest inference MAC but the 2nd largest parameters.

When per-parameter inference MAC is computed as in (b) (which is in log-scale), we can easily see that the parameters in the earlier layers get much more involved in the inference than those in the later layers. For example, the MAC-per-parameter for the last classifier is only 1.

Then, with a given pruning target, one pruning scheme can favor heavily pruning the classifier, as it is easier to hit the target without affecting model accuracy much (i.e., each parameter shows up only once in the forward pass), but this would fail to reduce the inference MAC enough. Then, the other scheme may favor aggressively pruning the earlier layers to significantly minimize the inference latency at a much greater risk of degrading the model accuracy. Therefore, it is critical to find a good balance between accuracy and inference speed. According to our experimental results, PDP can accomplish such a balance using differential pruning w.r.t. the task loss. Such trade-off can be optimized differently depending on whether a particular sparsity pattern or structure is enforced.

(a) Normalized inference MAC and parameter count for each layer.

10%

30%

InferenceMAC

Parameter Count

20%

5%

10%

0%

0%

model.0.0

model.1.0

model.1.3

model.2.0

model.2.3

model.3.0

modcl.3.3

model.4.0

model.4.3

model.5.0

model.6.0

model.6.3

model.7.0

model.7.3

model.8.0

model.8.3

model.9.0

model.9.3

model.10.0

model.10.3

model.11.3

model.12.0

model.12.3

model.13.0

model.13.3

classificr

NormalizedInferenceMAC

—NormalizedParameter Count

10,000

Inference MAC

100

modcl.0.0

model.1.0

model.1.3

model.2.0

model.2.3

model.3.0

model.3.3

modcl.4.0

modcl.4.3

model.5.0

model.6.0

model.6.3

model.7.0

model.7.3

model.8.0

model.8.3

model.9.0

model.9.3

model.10.0

model.10.3

model.12.0

model.12.3

model.13.0

model.13.3

classifier

MAC/parameter

(b) The inference MAC per parameter for each layer.

Random Pruning : Unstructured schemes make individual and independent pruning decision for each weight to maximize the flexibility and minimize the accuracy degradation. Simple and gradual/iterative pruning based on the weight magnitude has been studied extensively [13, 15, 19, 61]. In these schemes, once a weight is pruned, it does not have the second chance to become unpruned and improve the model quality. To address such challenges, RigL [12] proposes to grow a sparse network by reallocating the removed weights based on their dense gradients. Applying brain-inspired neurogeneration (i.e., unpruning some weights based on gradients) and leveraging pruning plasticity is proposed [34]. Altering the phase of dense and sparse training to accomplish co-training of sparse and dense models is studied, which results in good model accuracies on vision tasks [40]. Unlike other magnitude-driven pruning, supermask training [60] integrated with gradient-driven sparsity is proposed in [57], where accumulated gradients are used to generate binary masks and straight-through estimator [5] is used for backward propagation. Based on the lottery hypothesis [13], pruning in one-shot with heuristics [47] or gradient-driven metrics [49] is explored. More advanced techniques based on the second-order sensitivity analysis for language models are proposed in [27].

Structured/Channel Pruning : Unstructured pruning limits inference latency speedup as it suffers from poor memory access performance, and does not fit well on parallel computation [3, 35]. Recent research extends unstructured pruning by imposing a particular sparsity pattern during pruning at the cost of lower model predictive power, but increases the hardware performance during inference. One popular and effective form of structured pruning is channel pruning, where some channels with negligible effects on the model accuracy are discarded [21, 25, 30]. Using regularization to prune weights in a block is proposed in [29]. [26] leverages the β in BatchNorm to select the channels to prune (i.e., beta ≤ ϵ ) with ReLU assumed, which limits its applicability to wider set of DNNs. N:M pruning enforces that there are N non-zero weights out of every consecutive M weights [59] which enables a compact memory layout and efficient inferences on hardware [24, 37, 59]. A nondifferentiable method for N:M pruning with complex back-prorogation based on STE is presented in [59].

C Latency reduction with Random Sparsity

While MAC (to measure the latency gain) is rather a theoretical metric, not a measured metric, but it still captures the speedup benefit from pruning to some degree on some modern smartphones where ML accelerators natively support unstructured sparsity [2]. When we ran our pruned model on iPhone12 with the latest OS update, we obtained the following end-to-end latency measurements.

Due to other system overheads, the end-to-end latency reduction is not as significant as MAC saving, yet our result still implies the potential benefits of unstructured sparsity on a modern smartphone.

D PDP Algorithm and Training Flow

In order to obtain t in Fig 2, PDP needs a target pruning ratio r . The pruning ratio can be computed by selecting the top weights with larger magnitudes across all the layers and then instantly convert the selections into the per-layer ratios. Another way is to handcraft per-layer ratios, or reuse an existing configuration. Also, PDP is using the softmax operation which makes the softness concentrated over the weights around the t (as shown in Fig. 2). Hence, we gradually increase the target pruning ratio from 0 to r so that all low magnitude weights in the pruning range have a chance to use a soft-mask and settle down smoothly. For that purpose, we introduce a scaling step ϵ to let each weight have opportunities to leverage a soft-mask at least once, which leads to the training flow in Algorithm 1.

We select r to avoid pruning decisions being dominated by the initial weight values (which can be simply random). Choosing the right number of epochs as r depends on various factors such as the

Algorithm 1 Training flow for PDP

network size, the target sparsity and so on. Hence, r is a user hyper-parameter. In our experiments, the following guidelines worked best (and this is how we selected it).

r must be larger than the number of warm-up epochs.

After r epochs, the validation accuracy hits consistently over the half of the accuracy upper-bound (which is 50% for classifications) for 5 epochs.

In lines 2-7, a normal training is performed for the first s epochs. Then, in lines 8 and 9, the per-layer target pruning ratio is computed by selecting the bottom r · n ( W ) weights globally in terms of the magnitude. Then, in the remaining epochs, we use PDP to generate soft-masks as in the line 15, while gradually increasing the target ratio as in lines 10 and 11. The updated weight distribution is captured by updating t i as in the line 16 for all weight matrices. Once the entire training is over, we binarize the last mask for each weight to output the fully pruned weight for inference as in the line 26. Overall, the average runtime complexity of PDP is O ( W ) , as we only need to exercise topK algorithm (i.e, sorintg W is not necessary).

While fewer learnable parameters will reduce the model footprint and the communication cost (thus speeding up the training) in a distributed training setup, it does not necessarily lead to smaller memory consumption as the soft mask from PDP needs to generate related activation tensors. To understand the memory consumption better, we made a small GPT2 case with block _ size = 128 and n _ layer = 3 (a downscale version of the one in Table 2). And, then GPU memory is measured at three different spots for PDP and OptG, and reported in Table 11.

Spot0 : right after the model/optimizer are created (the parameter and PyTorch overheads)

Spot1 : right after forward, before backward (the peak memory consumption)

Spot2 : right after backward, before weight update (other parameter-related overheads)

From Table 11, we can see the following:

Not having extra mask parameters helps to reduce the model/optimizer overheads.

The peak memory is dominated by the activations (both data and mask). Yet, about 10 saving in the peak memory is observed.

After backward, the gradients for the learnable masks take up a large memory space, which have been all-reduced in the multi-GPU setting.

To isolate the benefit of the proposed soft mask in PDP from the gradual pruning in Algorithm 1, we ran PDP for ResNet50 and MobileNet-v1 training in the exactly same configuration as in Section 4 without the proposed soft mask, and obtained the following results. The result supports the efficacy of the proposed soft mask in PDP: without it, the ImageNet top1 accuracy drops by 1.6% and 1.4%, respectively.

E Ablation Study: Hyper-Parameter τ search

In the current PDP implementation, we use a global τ to control the level of softness in the pruning mask. Therefore, the selection of τ affects the model predictive power and should be carefully tuned. In order to explore the methodology for the τ search, we tried various values for MobileNet-v2 training, and the results are plotted in Fig. 9. The selection of τ affects the model predictive power as shown in Fig. 9 where there appears to be an optimal τ . For examples of MobileNet-v2, τ = 1 e -4 is the best value and is used for all the experiments in Section 4.

66.9%

66.8%

66.7%

66.6%

Top-1

66.5%

66.4%

1.0E-06

1.0E-05

1.0E-04

1.0E-03

1.0E-02

1

Since, Fig. 9 shows a concave curve, one could use a binary search to find the best τ values w.r.t. the top-1 accuracy. Also, it could be possible to cast τ as a learnable parameter for each layer or apply some scheduling to improve the model accuracy further (as future work), but still both approaches need an excellent initial point which can be found using a binary search technique.

F Additional Result for Section 4.

Different approaches made different sparsity allocations per the characteristics of the algorithm for a given pruning target, which results in complex trade-offs between model accuracy and inference speed. We report the detailed sparsity and inference MAC break-down for each layer in Fig. 6 and Fig. 7 on ImageNet1k and summarize our observations as follows:

OptG prunes the early convolution layers quite aggressively in ResNet18 and ResNet50, which leads to very low inference MACs as shown in Fig. 5 (a) and (b), yet at the cost of the worse Top-1 accuracy. For example, the inference MAC of ResNet18 from OptG is more than 2x less than that from ACDC ,

Interestingly, STR becomes aggressive in pruning the early convolution layers in MobileNetv1/v2, while OptG does not expose such behavior to MobileNet-v1/v2 (unlike it did for ResNet18/50). Such characteristics also favor the low inference latency over the model accuracy. Also, STR tends not to prune the last linear layer much as discussed in [28].

Unlike OptG and STR , ACDC does not prune the early convolution layers much for the tested networks, but prunes somewhat actively for the late convolution and linear layers, which leads to high model accuracies at the cost of higher inference latencies.

PDP is somewhat between STR and ACDC and modest across all layers in pruning allocation for all the networks, leading to superior accuracy and inference trade-offs. For example, the layers model.13.3 of MobileNet-v1 and features.17.conv.1.0 of MobileNet-v2 have the most difference among algorithms, and PDP is modest in pruning these two layers.

OptG has very low inference MACs on the earlier layers of ResNet18 and ResNet50 due to its aggressive pruning on these as seen in Fig. 6 (a) and (b), which leads to the extremely low inference latencies as shown in Fig. 5 (a) and (b).

GraNet tends to prune the earlier layers less but the later layers more in general which explains why GraNet shows the highest inference MACs in Fig. 5.

Table. 9 shows the pruned weight histograms of MobileNet-v1 from Table 5. We can observe that each algorithm affects the weight distribution in a slightly different way.

STR prefers to split the distribution more widely than others. For the example of the layer 5.0, STR clearly separated the positive and negative weights with a wide gap centered at the zero, while others sis not, except PDP created a slight dip around the zero to create mild separation.

PDP tends to spread out the sparsified weight distributions more than others. For the example of the fc layer, the weights from PDP range from -0.5 to 2.0, while those from

others are from -0.5 to 1.5. On the other hand, GraNet tends to keep the weight distributions tight.

We also experimented with varying pruning rates for PDP, OptG and ACDC for MobiletNet-v1 and ResNet-18 with ImageNet1k, and Bert with MNLI benchmark under the same configurations as in Section 4. Overall, as shown in Table 13 and Table 14, all tested schemes delivered higher accuracy with lower pruning rate, yet we can observe that PDP keeps its superiority to other schemes over all the tested pruning rates.

G Code References

Dense https://pytorch.org/vision/stable/index.html

GradNet https://github.com/VITA-Group/GraNet

OptG https://github.com/zyxxmu/OptG

ACDC https://github.com/IST-DASLab/ACDC

STR https://github.com/RAIVNLab/STR

GMP https://github.com/RAIVNLab/STR

DNW https://github.com/RAIVNLab/STR

CS https://github.com/lolemacs/continuous-sparsification

LNM https://github.com/NM-sparsity/NM-sparsity

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a parameter-free differentiable pruning, which uses a dynamic function of weights during training to generate soft pruning masks. It can generalize well on random/structured/channel pruning on both vision and NLP tasks. It achieves superior pruning results.\n\nSTRENGTHS:\n1. The proposed PDP approach is novel and interesting, and can generalize well on random/structured/channel pruning.\n2. The analysis in Section 3.1 is interesting and provides valuable insights.\n3. The proposed PDP approach can be faster than existing differentiable pruning approaches, with smaller pruning accuracy loss.\n\nWEAKNESSES:\n1. Instead of learning masks, which would introduce extra trainable parameters, PDP lets the weights of the network adjust themselves and generate soft masks. This approach may seem counterintuitive, as most weights are obtained and converged through expensive training processes. It may be more effective to adjust and learn masks rather than adjusting network weights to generate masks.\n\n2. The paper only provides a simple explanation for why PDP is more effective. A theoretical proof may be more convincing.\n\n3. Although the method is effective on the CNN and BERT models evaluated in this paper, its effectiveness on other models, such as larger vision transformers and LLMs, is still unclear.\n\n4. An important related work[1] is missing.  \n\n[1]Xia, Mengzhou, Zexuan Zhong, and Danqi Chen. \"Structured pruning learns compact and accurate models.\" ACL (2022).\n\nQUESTIONS:\nsee the weakness section.\n\nLIMITATIONS:\nThe main limitation of the paper is that, aside from empirical experimental results, there is no theoretical proof or explanation as to why it is superior to existing non-parameter-free approaches. The solution presented in the paper raises doubts about its effectiveness on larger neural networks, such as ViT and LLMs.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

comments: SUMMARY:
This paper deals with the pruning algorithm PDP on DNN, the main innovation is to generate a differentiable mask based on weights using a designed threshold t, and the softmax function. This mask can ensure the gradient propagation during forward and backward process while training, reducing the accuracy loss.

STRENGTHS:
1. This paper is well written and easy to follow.
2. The proposed method is novel and provides an efficient way to learn pruning masks without incurring heavy parameter burden.
3. The performance increase remains constant on various sparse ratios and models.

WEAKNESSES:
1. The employment of different masks can impact training process performance, yet a higher SAD[1] may result in a significant decline in accuracy. Given that the masks are still changeable in the later stages of PDP training, how can it provide high accuracy? 

[1] Learning N: M fine-grained structured sparse neural networks from scratch. In ICLR, 2020.

2. The description of PDP algorithm in section 3.2 may not be completely unambiguous. For the calculation of the conditional point t, the meaning of the paper should be to take the median of $r*n(W)$ and $(1-r)*n(W)$, but figure 3 (a) shows that $t=0.5\{min(W_h)+max(W_l)\}$, which seems to be wrong.

3. In Table 6, the performance of LNM seems to be quite different from that reported in the original paper. Please explain this.

QUESTIONS:
Please see the weakness part.

LIMITATIONS:
No limitations discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on DNN pruning and proposes a novel approach using a soft mask during training. The soft mask is designed to encourage weights around the pruning threshold to actively switch their states, aiding in the recovery of pruned weights. This method is simple, efficient, and introduces no additional parameters. Experimental evaluations of the proposed approach have been conducted on ResNets, MobileNets, and BERT. It also explores different pruning settings such as N:M pruning, random pruning and channel pruning.\n\nSTRENGTHS:\n* The paper proposes a soft mask during training, which smooths the participation of weights around the pruning threshold in the neural network. This can give more chances for weights around pruning threshold to change their states, especially help pruned weights actively recover themselves.\n* The compared methods are enough in experiments.\n* The paper discusses relevant works and provides a comprehensive analysis.\n\nWEAKNESSES:\n* Method:\n  *  Lack of differentiable analysis: The title and content of the paper stress the differentiable pruning, however, method part  (3.2) does not give differentiable analysis.\n  * Insufficient explanation of the design of \"t\" and calculation of \"m(w)\": Why is “t” in PDP training flow designed in this way? The paper lacks clear explanations about the design of “t” and calculation of “m(w)”.\n  * Lack of analysis and ablation study on the efficacy of the proposed soft mask: I find that the paper adopts progressive pruning as illustrated in the algorithm chart in supplementary material. As progressive pruning is an effective technique for final accuracy, I am doubtful about the real effect of the proposed method. To better understand the effectiveness of the proposed soft mask, we need more analysis and ablation studies.\n* Experiments: \n  * More experiments about Transformer architectures are preferred. The paper provides BERT models experiments on MNLI dataset. Experiments with ViT models or BERT models on the whole GLUE benchmark will be preferred.\n  * In the main body, there are not enough sparsity ratio selections. Table 4 provides only one sparsity ratio for each model. More sparsity ratios, ranging from low to high are necessary to understand the effect of the proposed method.\n* Writing: \n  * Redundancy between Figure 5 and Table 4: It seems that Figure 5 and Table 4 convey the same information. What is the difference between Figure 5 and Table 4 or why illustrate the same information twice? I think each table or figure should contain specific information, especially in the main body.\n  * Improper organization of the paper: The paper first illustrates the benefits of PDP over existing pruning approaches, then introduces the PDP algorithm, which can be confusing. Before readers know the method clearly, it would be difficult to understand its benefits. The better solution would be to change their order or to reconsider the effects of section 3.1, such as serving it as a motivation section.\n\nQUESTIONS:\nPlease see the method and experiment points in the weakness part. If my concerns are addressed, I would like to increase my score.\n\nLIMITATIONS:\nThere is no potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper describes a novel pruning algorithm named parameter-free differentiable pruning (PDP). The core idea is to generate soft pruning masks (i.e., mask values are not finalized until the final iteration of fine-tuning) using a parameter-free, differentiable dynamic function of the weights of the network. This approach eliminates the need to train additional mask parameters (and related hyper-parameter tuning) and appears to be easy to integrate into existing training pipelines. PDP is evaluated on networks drawn from vision and NLP, and on both unstructured and structured N:M sparsity patterns. The paper reports both accuracy and MAC improvements and provides a comparison of PDP to other SOTA pruning algorithms.\n\nSTRENGTHS:\n* The paper is fairly well-written, but could use some reorganization to help with the overall flow.\n* The paper targets the relevant and important problem of structured sparsification of large DNNs to improve inference efficiency.\n* While soft and progressive pruning are well-known in the literature, its parameter-free nature and the specific mask computation make PDP unique.\n* Evaluation methodology is sound, with relevant comparisons to other SOTA approaches, and results being reported on various types of networks (CNNs, attention-based, etc.) across disparate domains.\n\nWEAKNESSES:\nMeasuring inference efficiency in MACs can often be misleading. For instance, unstructured pruning can reduce MACs by getting rid of individual ineffectual computations, but this is difficult to realize in modern parallel hardware such as GPUs and TPUs. Reporting at least some data points with actual runtime numbers would be useful.\n\nQUESTIONS:\n* What is the motivation for selecting the per-layer sparsity value (r) after a few epochs of training? How would a user select the right number of epochs for a given network,  sparsity pattern and global sparsity degree?\n* Readability suggestion: consider moving Section 3.2 before 3.1.\n\nLIMITATIONS:\nLimitations have been adequately addressed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
The paper introduces a new DNN pruning scheme called Parameter-free Differentiable Pruning (PDP), which is an efficient and effective train-time pruning method that offers state-of-the-art qualities in model size, accuracy, and training cost. Unlike existing pruning approaches, PDP generates soft pruning masks for weights in a parameter-free manner, making it easy to apply to various vision and natural language tasks, DNN architectures, and structured pruning constraints. The paper presents experimental results on several benchmark datasets, including MobileNet-v1 and BERT, demonstrating that PDP achieves impressive results in terms of model size, accuracy, and training cost.

STRENGTHS:
- The paper introduced a PDP, a novel differentiable pruning methods that is parameter-free, which uses a dynamic function of weights to generate soft pruning masks for the weights.
- PDP can be applied tto structured and channel pruning, such as N:M pruning, where top-of-the-line GPUs support such configuration.

WEAKNESSES:
- The PDP differentiable pruning does not introduce extra parameter, but it still need to generate (soft) mask from weight, which would induce extra activation maps, how is the memory consumption during differentiable pruning compare to other SoTAs?
- The paper seem provide results on MobileNet-v1/v2 but not on MobileNet-v3, can the author elaborate on why?

QUESTIONS:
As shown in weaknesses part.

LIMITATIONS:
The paper did not any (Peak) memory consumption result of PDP during differentiable pruning.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-10