SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Channel pruning is another type of structured pruning that can be easily applied to PDP with minor modifications. To do this, we first compute the  $\mathrm{L}_2$  norm of each channel in the layer, and then use these norm values (in place of the absolute values of the weights in Figure 2(a)) to compute a soft mask for each channel, which is depicted in Figure 4 (b). Using the soft mask to prune all the corresponding weights in the channel will make the channel pruning process differentiable.
We compared our PDP with state-of-the-art random, structured, and channel pruning schemes on various computer vision and natural language models. We used two x86 Linux nodes with eight NVIDIA V100 GPUs on each in a cloud environment. All cases were trained from scratch. More experimental results and the hyper-parameters are in Section F and Table 8 in Appendix.
Random Pruning for Vision Benchmark: We compared the proposed PDP with the latest prior arts, STR [28], GMP [61], DNW [52], GraNet [34], OptG [57], and ACDC [40] on ResNet18, ResNet50, MobileNet-v1, and MobileNet-v2 [20, 22, 43] with the ImageNet1k dataset [9]. Since all of these schemes have been experimented only with ResNet50 and/or MobileNet-v1, we reproduced the pruning results in our controlled environment with the identical data augmentations by running the official implementations from the authors [28, 34, 40, 57] or verified implementations from the prior arts [52, 61] as in Section G in Appendix. Since the primary goal of pruning is to trade-off the model accuracy with the compute reduction as in Section 2, we measured the accuracies and Multiply-Accumulate Operation (MAC) during inference on each experiment with layer fusion (i.e., BatchNorm folding), and mainly focused on the high-sparsification cases. Note that the MAC is purely theoretical and reported to understand the trade-off among accuracy, size, and compute across
(a) ResNet18 (sparsity:  $85.5\%$ )
(b) ResNet50 (sparsity:  $89.8\%$ )
(c) MobileNet-v1 (sparsity:  $86.6\%$ )
(d) MobileNet-v2 (sparsity:  $80.2\%$
Figure 5: PDP-powered pruning (in white box markers) delivers the Pareto superiority to the other schemes (i.e., the top-bottom corner is the best trade-off) for ResNet18, ResNet50, and MobileNet-v1/v2 on ImageNet1k. The size of markers indicates the relative training overheads. The detailed numbers are in Table 4.
various algorithms, but it still captures the latency benefit on some hardware platforms (see C in Appendix for details). In our experiments with ImageNet1k, all layers, including both the first and last layers, are pruned without any restriction. Also, we estimated the algorithmic overhead and training efficiency by the total monetary cost for GPUs on commercial cloud spot instances [7].
We applied not only the proposed hyper-parameters from the authors but also a set of further finetuned hyper-parameters for competing methods (see Table 8 in Appendix for details). Also, since each algorithm used a different number of epochs and showed results at different sparsity levels, a) we ran STR first to set the target sparsity levels for all the networks for fair comparisons, because all other schemes can control the sparsity level precisely, b) we trained ResNet18/50 for 100 epochs and MobileNet-v1/v2 for 180 epochs following [28, 34, 40, 57] except STR (which diverged with more epochs for MobileNet-v1/v2). For PDP, we fixed the target sparsity for each layer based on the global weight magnitude at the epoch 16 and started pruning at the rate of  $1.5\%$  of the target sparsity per epoch for all the experiments which correspond to  $s = 16$  and  $\epsilon = 0.015$  in Algorithm 1 in Appendix. For detailed experiment configurations, please refer to Table 8. Every experiment began with a randomly initialized model (i.e., no pre-trained model). For PDP, we had the following variants to show the value of PDP with the same training overhead or per-layer pruning budgets.
- PDP-base globally computes the target sparsity by  $abs(W)$  at epoch 16 across all the layers.
- PDP-base+ is the same as PDP-base yet with more epochs to match the GPU cost of ACDC.
- PDP-str/optg uses the per-layer sparsity from STR/OptG as input to normalize the MAC.
Our experimental results are highlighted in Fig. 5, where the size of circles indicates the relative training overhead due to pruning. Note that we used only one single node with 8 GPUs due to the limitation in the official implementations for GraNet and OptG, thus both have the advantage of not having the inter-node communication cost. Also, each approach imposes a different level of training-time overhead, mainly due to the various complexities of training flow and pruning itself as captured in Fig. 5. Overall results can be summarized as follows:
- PDP showed the best the model accuracy: PDP-base on ResNet18 delivered  $69\%$  Top-1 accuracy which is superior to other schemes but at higher MAC than only STR and OptG.
Network Sparsity	Method	Top-1 (%)	GPU$ cost($)	MAC (×e6)	Network Sparsity	Method	Top-1 (%)	GPU$ cost ($)	MAC (×e6)
ResNet18 85.5%	Dense	69.8	167	1814.1	Dense	76.1	248	4089.2
GMP	65.2	217	263.5	GMP	73.6	483	419.0
DNW	64.4	206	263.5	DNW	70.7	466	419.0
GraNet*	66.0	198	539.6	GraNet*	72.5	321	868.0
STR	66.7	171	334.6	ResNet50	STR	72.8	417	373.7
OptG*	65.5	277	223.7	89.8%	OptG*	72.1	591	333.0
ACDC	68.7	356	502.8	ACDC	74.7	635	735.6
PDP-base	69.0	169	408.6	PDP-base	74.7	325	502.8
PDP-base+	69.5	336	405.1	PDP-base+	75.3	610	483.0
PDP-str	68.6	169	334.7	PDP-str	74.0	325	373.7
PDP-optg	68.5	174	223.7	PDP-optg	74.2	332	332.9
MobileNet-v1 86.6%	Dense	70.9	277	568.7	Dense	71.9	297	300.8
GraNet*	61.4	367	145.7	GraNet*	56.3	439	103.4
STR	61.7	176	47.2	MobileNetv-2	STR	60.0	285	40.6
OptG*	66.3	340	87.4	80.2%	OptG*	65.4	545	76.8
ACDC	66.5	641	124.5	ACDC	64.1	812	93.9
PDP-base	68.2	281	88.3	PDP-base	66.8	354	95.3
PDP-str	65.3	307	47.2	PDP-str	60.7	307	40.6
PDP-optg	68.2	297	87.3	PDP-optg	66.5	343	76.6
$ the GPU cost ($) is based on a commercial cloud spot instance pricing.
* used only one with 8 GPUs due to the limitations in the public code.
$\diamond$  when PDP applied for MobileNet-v3 (5.5M parameters) with an  $80\%$  sparsity target, we achieved  $71.5\%$  top-1 ImageNet1K accuracy, which is  $2.5\%$  down from the dense version.
Table 4: PDP compared with other unstructured pruning algorithms on ImageNet1K shows the best trade-off among accuracy, inference MAC, and training overheads. More results are available in Section F in Appendix.
- PDP offered the better model accuracy for a given pruning target: With the custom sparsification target for each layer, PDP-str/optg demonstrated the  $2 - 3\%$  higher Top-1 accuracy at the same MAC, demonstrating the effectiveness of the proposed method.
- When we use the similar GPU budget for additional epochs with ACDC which is noted as PDP-base+, our method further improved the Top-1 accuracy from  $69\%$  to  $69.5\%$  for ResNet18 and from  $74.7\%$  to  $75.3\%$  for ResNet50 with slight fewer MACs.
Random Pruning for NLP Benchmark: We compared PDP with the state-of-the-art pruning results from MVP [44] and POFA [56] (quoted from the respective papers) in addition to OptG and STR (reproduced in our environment) on a BERT model [10] for the two largest NLP tasks of the GLUE benchmark, MNLI (Multi-Genre Natural Language Inference) with 392,702 samples and QQP (Question-answering Natural Language Inference) with 363,836 samples. Following the setups in MVP and POFA, we used the same batch size 32 per GPU (i.e., global mini-batch size of 512), excluded the embedding and the last linear layer from pruning, and trained the bert-base-uncased from HuggingFace from scratch with self-distillation. The teacher for the distillation is from the best checkpoint in the first 40 epochs. We use the weights of 0.95 on the distillation loss and 0.05 on the task loss for PDP, and 0.75 on the distillation loss and 0.25 on the task loss for STR. We could not use distillation for OptG, as it caused the out-of-memory error due to the extra-parameter overheads from both pruning and distillation. Our experimental results in Table 5 can be summarized as follows:
- STR underperforms on all the best cases even though it could not achieve the target sparsity.
- OptG shows better model accuracy only than GMP and worse than others in our setup.
- PDP outperformed all other methods for both MNLI and QQP.
Structured/Channel Pruning for Vision Benchmarks: We compared PDP-driven N:M pruning and channel pruning on the ImageNet1k dataset [9]. For N:M pruning, we reproduced the LNM [59] results using the public code base but without the color augmentation to keep the experimental environment normalized. For PDP, we simply reused the hyper-parameters and configurations as in Table 8 in our Appendix, and the top-1 accuracies by various N:M config with ImageNet1k on ResNet18/50 are presented in Table 6. We can observe that PDP outperforms LNM on all the test
Benchmark	Metric	Methods
Dense	STR*	OptG	GMP	MVP	POFA	PDP
MNLI matched	accuracy (%)	90	84.5	75.8	78.5	n/a	81.2	81.5	83.1
94	74.4	76.9	74.8	80.7	n/a	82.0
MNLI mismatched	90	84.9	76.3	78.3	n/a	81.8	82.4	83.0
94	74.1	76.5	75.6	81.2	n/a	82.4
QQP	accuracy (%)	90	91.2	n/a	89.8	n/a	90.2	90.9	91.0
f1	90	88.1	n/a	86.2	n/a	86.8	87.7	88.0
* Since STR cannot control the sparsity precisely, we report the metrics with our closest achieved sparsity levels, 85.9% for the 90% case and 91.8% for the 94% case.
Table 5: PDP delivers the best accuracies on the BERT-base model with the MNLI (Multi-Genre Natural Language Inference) and QQP (Quora Question Pairs2) tasks of the GLUE benchmark.
Network	Method	Batch size	#epochs	N:M	avg GPU cost ($)
2:4	4:8	1:4	2:8
ResNet18	LNM	256	120	69.6	70.2	65.1	68.4	395
PDP	1024	100	70.2	70.1	68.7	69.1	275
ResNet50	LNM	256	120	74.6	75.1	74.1	75.0	812
PDP	1024	100	75.9	75.8	75.0	75.3	380
* For LNM, we used the the public dense model from TorchVision for channel pruning, as the original baseline from [59] (which has higher accuracy) is not available publicly. For the example of ResNet50 on ImageNet1k, LNM used a dense model with 77.3% top-1 accuracy while the public dense model in TorchVision) yields 76.1% top-1 accuracy.
Table 6: Structured Pruning: PDP can be directly to do N:M pruning due to its generality. PDP delivers the superior results than the latest N:M pruning in [59] at  $46\%$  less training cost.
Method	Batch size	#epochs	Top-1 (%)	MAC drop (%)
NISP	?	90	75.3	44.0
DCP	256	60	75.0	55.0
SCP	256	100	75.3	54.3
PDP	1024	100	75.9	54.9
* SCP, DCP, and NISP reported only ResNet50 results with MAC drop instead of sparsity. Hence, for PDP, we report the nearest MAC drop we obtained (54.9%) at 57% channel sparsity. Table 7: Channel Pruning: the generality of PDP helps deliver the state-of-the-art results without modifications.
cases, even with 4x larger batch size in 20 fewer epochs. LNM training cost is also much higher than PDP because of its costly weight regularization and complex back-propagation scheme.
For channel pruning, we compared PDP with SCP [26], NISP [55], and DCP [62]. Note that SCP uses the  $\beta$  in BatchNorm to select the channels to prune (i.e.,  $beta \leq \epsilon$ ), hence applicable to limited types of networks only. We again reused the hyper-parameters and configurations as in Table 8 in our Appendix for PDP, and the top-1 accuracy with ImageNet1k on ResNet50 is reported in Table 7. We can see that PDP can be used for channel pruning and show superior performance out of the box.
We showed that a simple and universal pruning method PDP can yield the state-of-the-art random/structured/channel pruning quality on popular computer vision and natural language models. Our method requires no additional learning parameters, yet keeps the training flow simple and straightforward, making it a practical method for real-world scenarios. We plan to extend our differentiable pruning into quantization, making both jointly differentiable and estimizable by the task-loss.
[1] https://github.com/karpathy/nanoGPT.
[2] https://apple.github.io/coremltools/docs-guides/source/pruning-overview.html.
[3] S. Anwar, K. Hwang, and W. Sung. Structured pruning of deep convolutional neural networks. ACM Journal on Emerging Technologies in Computing Systems, 2017.
[54] X. Xiao, Z. Wang, and S. Rajasekaran. Autoprune: Automatic network pruning by regularizing auxiliary parameters. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, 2019.
[55] R. Yu, A. Li, C. Chen, J. Lai, V. I. Morariu, X. Han, M. Gao, C. Lin, and L. S. Davis. NISP: pruning networks using neuron importance score propagation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018.
[56] O. Zafrir, A. Larey, G. Boudoukh, H. Shen, and M. Wasserblat. Prune once for all: Sparse pre-trained language models. In Advances in Neural Information Processing Systems, 2021.
[57] Y. Zhang, M. Lin, M. Chen, Z. Xu, F. Chao, Y. Shen, K. Li, Y. Wu, and R. Ji. Optimizing gradient-driven criteria in network sparsity: Gradient is all you need. In CoRR, 2022.
[58] X. Zhao, Y. Wang, X. Cai, C. Liu, and L. Zhang. Linear symmetric quantization of neural networks for low-precision integer hardware. In International Conference on Learning Representations, 2019.
[59] A. Zhou, Y. Ma, J. Zhu, J. Liu, Z. Zhang, K. Yuan, W. Sun, and H. Li. Learning n:m fine-grained structured sparse neural networks from scratch. In International Conference on Learning Representations, 2021.
[60] H. Zhou, J. Lan, R. Liu, and J. Yosinski. Deconstructing lottery tickets: Zeros, signs, and the supermask. In Advances in Neural Information Processing Systems, 2019.
[61] M. Zhu and S. Gupta. To prune, or not to prune: Exploring the efficacy of pruning for model compression. In Workshop, International Conference on Learning Representations, 2018.
[62] Z. Zhuang, M. Tan, B. Zhuang, J. Liu, Y. Guo, Q. Wu, J. Huang, and J. Zhu. Discrimination-aware channel pruning for deep neural networks. In Advances in Neural Information Processing Systems, 2019.
Network	Method	Batch size	#epochs	#wu	# GPUs	#Nodes	Main optimizer, scheduler	Mask optimizer, scheduler	other params
ResNet20	CS	128	85x5	0	1	1	SGD 0.9 0.1, multi-step 56,75 by 0.1	same as the main	init value: -1
PDP	128	120	5	1	1	SGD 0.9 1e-4, cosine 0.4	n/a	τ: 1e-4, ε: 0.015
GPT2	OptG	Refer to [1]
PDP	Refer to [1]
ResNet18	GraNet	128	100	5	8	1	SGD 0.9 1e-4 (0.0 for BN), cosine 0.1	n/a	init density: 0.5
GMP	128	100	5	16	2	SGD 0.875 1e-5, cosine 0.256	same as the main
STR	256	100	5	16	2	SGD 0.875 2.251757813e-5, cosine 0.256	same as the main	init value: -3200
OptG	256	100	5	16	2	SGD 0.9 1e-4, cosine 0.1	SGD 0.9 0, cosine 0.1	β: 1.0
ACDC	256	100	5	8	1	SGD 0.875 3.051757813e-4, cosine 0.256	n/a	8 alterations
PDP-base/str/optg	1024	100	5	16	2	SGD 0.9 1.4e-5, cosine 1.8	n/a	τ: 1e-4, ε: 0.015
PDP-base+	1024	200	5	16	2	SGD 0.9 1.4e-5, cosine 1.8	n/a	τ: 1e-4, ε: 0.015
ResNet50	GraNet	128	100	5	8	1	SGD 0.9 1e-4 (0.0 for BN), cosine 0.1	n/a	init density: 0.5
GMP	128	100	5	16	2	SGD 0.875 1e-5, cosine 0.256	same as themain
STR	256	100	5	16	2	SGD 0.875 2.251757813e-5, cosine 0.256	same as themain	init value: -3200
OptG	256	100	5	16	2	SGD 0.9 1e-4, cosine 0.1	SGD 0.9 0, cosine 0.1	β: 1.0
ACDC	256	100	5	8	1	SGD 0.8753.051757813e-4, cosine 0.256	n/a	8 alterations
PDP-base/str/optg	1024	100	5	16	2	SGD 0.9 1.4e-5, cosine 1.8	n/a	τ: 1e-4, ε: 0.015
pDP-base+	1024	200	5	16	2	SGD 0.9 1.4e-5, cosine 1.8	n/a	τ: 1e-4, ε: 0.015
MobileNet-v1	GraNet	128	180	5	8	1	SGD 0.9 1e-4 (0.0 for BN), cosine 0.1	n/a	init density: 0.5
STR	256	100	5	16	2	SGD 0.875 3.751757813e-5, cosine 0.256	same as themain	init value: -12800
OptG	256	180	5	16	2	SGD 0.9 4e-4, cosine 0.1	SGD 0.9 0, cosine 0.1	β: 1.0
ACDC	256	180	5	8	1	SGD 0.875 3.051757813e-4, cosine 0.256	n/a	8 alterations
PDP-base/str/optg	1024	180	5	16	2	SGD 0.9 1.4e-5, cosine 1.8	n/a	τ: 1e-4, ε: 0.015
MobileNet-v2	GraNet	128	180	5	8	1	SGD 0.9 1e-4 (0.0 for BN), cosine 0.1	n/a	init density: 0.5
STR	256	100	5	16	2	SGD 0.875 3.751757813e-5, cosine	same as themain	init value: -12800
OptG	256	180	5	16	2	SGD 0.9 4-e4, cosine 0.05	SGD 0.9 0, cosine 0.05	β: 1.0
ACDC	256	180	5	8	1	SGD 0.875 3.051757813e-4, cosine 0.256	n/a	8 alterations
PDP-base/str/optg	1024	180	5	16	2	SGD 0.98e-6, cosine 0.8	n/a	τ: 1e-4, ε: 0.015
Bert	GMP	Refer to Section 6 and Table 3 in [44]
MOV	Refer to Section 6 and Table 3 in [44]
POFA	Refer to Section E and Tables 5 and 6 in [56]
STR (85.9%)	32	140	5	16	2	AdamW 1e-8, multiplicative 1e-4 0.95	SGD 0.875 0.06, cosine 1e-4	init value: -90
STR (91.7%)	32	140	5	16	2	AdamW 1e-8, multiplicative 1e-4 0.95	SGD 0.875 0.065, cosine 1e-4	init value: -110
OptG	32	140	5	16	2	AdamW 1e-8, cosine 1e-4	SGD 0.9 0, cosine 1e-4	β: 1.0
PDP	32	140	5	16	2	AdamW 1e-8, cosine 1e-4	n/a	τ: 1e-4, ε: 0.015
SGD: momentum, weight decay, cosine: learning_rate, AdamW: epsilon, multiplicative: learning rate, gamma, #wu: the number of warm-up epochs. Table 8: The hyper-parameters in Sections 3 and 4.
(b) ResNet50
(c) MobileNet-v1
(d) MobileNet-v2
Figure 6: Layer-wise sparsity allocation from the experiments in Table 5.
(d) MobileNet-v2
Figure 7: Layer-wise Inference MAC distribution from the experiments in Table 5.
f#	10.3	10.0	9.3	9.0	8.3	8.0	7.3	7.0	6.3	6.0	5.3	5.0	4.3	4.0	3.3	3.0	2.3	2.0	1.3	1.0	0.0
Table 9: The weight histograms in log scale for MobileNet-v1 in Table 5.
A Training Configurations and Hyper-parameters
Since some techniques in Sections 3 and 4 require extra training parameters and pruning scheduling as shown in Table 1, we disclose the training configurations and hyper-parameters we found the best in Table 8.
B Trade-off in Pruning
Pruning for DNN requires exploring a good trade-off between model accuracy and inference latency under a given pruning target. Such a challenge can be elaborated with the MobiletNet-v1 Dense case in Fig. 8 where the following observations can be made:
- The earlier layers have significantly fewer parameters than the later layers while still having comparable inference MACs as shown in (a). For example, the final classifier, which is a linear layer, has the lowest inference MAC but the 2nd largest parameters.
- When per-parameter inference MAC is computed as in (b) (which is in log-scale), we can easily see that the parameters in the earlier layers get much more involved in the inference than those in the later layers. For example, the MAC-per-parameter for the last classifier is only 1.
Then, with a given pruning target, one pruning scheme can favor heavily pruning the classifier, as it is easier to hit the target without affecting model accuracy much (i.e., each parameter shows up only once in the forward pass), but this would fail to reduce the inference MAC enough. Then, the other scheme may favor aggressively pruning the earlier layers to significantly minimize the inference latency at a much greater risk of degrading the model accuracy. Therefore, it is critical to find a good balance between accuracy and inference speed. According to our experimental results, PDP can accomplish such a balance using differential pruning w.r.t. the task loss. Such trade-off can be optimized differently depending on whether a particular sparsity pattern or structure is enforced.
(a) Normalized inference MAC and parameter count for each layer.
(b) The inference MAC per parameter for each layer.
Figure 8: Layer-wise Inference MAC and Parameters from the MobileNet-v1 Dense case in Table 5.
Random Pruning: Unstructured schemes make individual and independent pruning decision for each weight to maximize the flexibility and minimize the accuracy degradation. Simple and gradual/iterative pruning based on the weight magnitude has been studied extensively [13, 15, 19, 61]. In these schemes, once a weight is pruned, it does not have the second chance to become unpruned and improve the model quality. To address such challenges, RigL [12] proposes to grow a sparse network by reallocating the removed weights based on their dense gradients. Applying brain-inspired neurogeneration (i.e., unpruning some weights based on gradients) and leveraging pruning plasticity is proposed [34]. Altering the phase of dense and sparse training to accomplish co-training of sparse and dense models is studied, which results in good model accuracies on vision tasks [40]. Unlike other magnitude-driven pruning, supermask training [60] integrated with gradient-driven sparsity is proposed in [57], where accumulated gradients are used to generate binary masks and straight-through estimator [5] is used for backward propagation. Based on the lottery hypothesis [13], pruning in one-shot with heuristics [47] or gradient-driven metrics [49] is explored. More advanced techniques based on the second-order sensitivity analysis for language models are proposed in [27].
Structured/Channel Pruning: Unstructured pruning limits inference latency speedup as it suffers from poor memory access performance, and does not fit well on parallel computation [3, 35]. Recent research extends unstructured pruning by imposing a particular sparsity pattern during pruning at the cost of lower model predictive power, but increases the hardware performance during inference. One popular and effective form of structured pruning is channel pruning, where some channels with negligible effects on the model accuracy are discarded [21, 25, 30]. Using regularization to prune weights in a block is proposed in [29]. [26] leverages the  $\beta$  in BatchNorm to select the channels to prune (i.e.,  $\beta \alpha \leq \epsilon$ ) with ReLU assumed, which limits its applicability to wider set of DNNs. N:M pruning enforces that there are N non-zero weights out of every consecutive M weights [59] which enables a compact memory layout and efficient inferences on hardware [24, 37, 59]. A non-differentiable method for N:M pruning with complex back-prorogation based on STE is presented in [59].
C Latency reduction with Random Sparsity
While MAC (to measure the latency gain) is rather a theoretical metric, not a measured metric, but it still captures the speedup benefit from pruning to some degree on some modern smartphones where ML accelerators natively support unstructured sparsity [2]. When we ran our pruned model on iPhone12 with the latest OS update, we obtained the following end-to-end latency measurements.
Method	ResNet50	MobileNet-v2
Dense	2.71	0.95
PDP	1.39	0.75
Speedup (%)	94	27
Table 10: A single image classification latency in msec on iPhone12 for the pruned models in Table 4.
Due to other system overheads, the end-to-end latency reduction is not as significant as MAC saving, yet our result still implies the potential benefits of unstructured sparsity on a modern smartphone.
D PDP Algorithm and Training Flow
In order to obtain  $t$  in Fig 2, PDP needs a target pruning ratio  $r$ . The pruning ratio can be computed by selecting the top weights with larger magnitudes across all the layers and then instantly convert the selections into the per-layer ratios. Another way is to handcraft per-layer ratios, or reuse an existing configuration. Also, PDP is using the softmax operation which makes the softness concentrated over the weights around the  $t$  (as shown in Fig. 2). Hence, we gradually increase the target pruning ratio from 0 to  $r$  so that all low magnitude weights in the pruning range have a chance to use a soft-mask and settle down smoothly. For that purpose, we introduce a scaling step  $\epsilon$  to let each weight have opportunities to leverage a soft-mask at least once, which leads to the training flow in Algorithm 1.
We select  $r$  to avoid pruning decisions being dominated by the initial weight values (which can be simply random). Choosing the right number of epochs as  $r$  depends on various factors such as the