SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our RPG method is applied on ResNet-50 under high sparsities:  $80\%$ ,  $90\%$ ,  $95\%$ , and  $98\%$ . We compare RPG with recent baselines. Among the baselines, STR [29] automatically learns pruning sparsity; WoodFisher [48], GraNet [34] and ProbMask [60] are methods based on gradual pruning; AC/DC [40] and ProbMask [60] are baselines targeted at high sparsities; PowerPropagation [45] is an improvement of Top-KAST [26] that relies on a pre-set layerwise sparsity distribution. For fair comparison, all results are 100-epoch baselines; we used standard ImageNet config, detailed in the Appendix. The results are presented in Table 2. The advantage of adversarial rank-based pruning is manifested at high sparsities.
Results analysis. Our method could achieve outstanding performance for sparsities  $90\%$ ,  $95\%$ , and  $98\%$ . At lower sparsities (e.g.  $80\%$ ,  $90\%$ ), WoodFisher [48] takes the lead among the baselines. Our RPG method is slightly lower than WoodFisher [48] by  $0.07\%$  in ImageNet accuracy at  $80\%$  sparsity. At higher sparsities, our method outcompetes other baselines. Other competitive baselines at high sparsities include PowerPropagation [45] and AC/DC [40]. However, the gap between our RPG method and these baselines widened at high sparsities. Specifically, our method outperforms current top baseline by  $1.13\%$  of ImageNet Top-1 accuracy at  $98\%$  sparsity.
Algorithm	Sparsity	Accuracy
ResNet-50 [22]	0	76.80
STR [29]	0.8	76.19
WoodFisher [48]	0.8	76.73
GraNet [34]	0.8	76.00
AC/DC [40]	0.8	76.30
PowerPropagation [45]	0.8	76.24
RPG (Ours)	0.8	76.66
STR [29]	0.9	74.31
WoodFisher [48]	0.9	75.26
GraNet [34]	0.9	74.50
AC/DC [40]	0.9	75.03
ProbMask [59]	0.9	74.68
PowerPropagation [45]	0.9	75.23
RPG (Ours)	0.9	75.80
STR [29]	0.95	70.40
WoodFisher [48]	0.95	72.16
AC/DC [40]	0.95	73.14
ProbMask [59]	0.95	71.50
PowerPropagation [45]	0.95	73.25
RPG (Ours)	0.95	74.05
STR [29]	0.98	62.84
WoodFisher [48]	0.98	65.55
AC/DC [40]	0.98	68.44
ProbMask [59]	0.98	66.83
PowerPropagation [45]	0.98	68.00
RPG (Ours)	0.98	69.57
Table 2. Sparsified ResNet-50 on ImageNet. All results are official reports from the original works. Best and second best results are bolded and underlined.
Erdos-Renyi-Kernel (ERK) [14] is a layerwise sparsity distribution that is commonly used for performance boosting in weight pruning methods that require a pre-set sparsity distribution. However, ERK-based sparse models are computationally costly. Differently, RPG automatically maintains a more balanced sparsity throughout the whole network under the same total sparsity constraint. Though our sparse model slightly lags behind the current ERK variant of SOTA [45] under lower spurities in certain accuracy, it is much cost-effective. Quantitatively, for  $80\%$  sparse ResNet-50, the reported ERK-based State-of-the-Art ImageNet accuracy is merely  $0.10\%$  higher than our RPG method (reaching  $76.76\%$  for [45]), but costing an extra  $58\%$  of FLOPs. The advantage of our RPG method over ERK-based methods is clearly illustrated in Figure 3, where we compare RPG with the ERK variant of TOP-KAST [26] and the State-of-the-Art PowerPropagation [45].
DeepSparse [28] is a recent sparse acceleration framework on CPU that makes unstructured-sparse network acceleration possible in applications. We time sparse ResNet-50 on DeepSparse for single-image inference. Results in Table 3 show that highly-sparse ResNet-50 could achieve around
Sp.	Acc.	Runtime
Dense	76.80	40.25ms
0.8	76.66	39.26ms
0.9	75.80	27.98ms
0.95	74.05	22.20ms
0.98	69.57	20.89ms
Table 3. Sparse acceleration of sparse ResNet-50 on DeepSparse. Unstructured pruning could bring  $2 \times$  acceleration effects on CPU at high sparsities.
Algorithm	Sp.	BoxAP	MaskAP
Mask R-CNN	0	38.6	35.2
RigL [14]	0.5	36.4	32.8
AC/DC [40]	0.5	37.9	34.6
RPG (Ours)	0.5	37.7	34.4
RigL [14]	0.7	32.3	29.1
AC/DC [40]	0.7	36.6	33.5
RPG (Ours)	0.7	37.6	34.4
RigL [14]	0.8	26.0	23.7
AC/DC [40]	0.8	34.9	32.1
RPG (Ours)	0.8	37.1	33.8
Table 4. Mask R-CNN pruning on COCO val2017. "Sp." stands for model sparsity. Best results are bolded.
Algorithm	Sp. Accuracy
DeiT-S [52]	0	79.85
SViT-E [6]	0.5	79.72
AC/DC [40]	0.5	80.15
RPG (Ours)	0.5	80.15
SViT-E [6]	0.6	79.41
AC/DC [40]	0.6	79.69
RPG (Ours)	0.6	79.89
AC/DC [40]	0.8	76.24
RPG (Ours)	0.8	77.42
Table 5. Sparse DeiT-S on ImageNet. "Sp." stands for model sparsity. The best results are bolded.
$2 \times$  acceleration on CPU. This observation reveals that highly unstructured-sparse networks have promising applicative prospects on edge devices that could not afford power and cost-intensive GPUs, e.g. micro robots, wearable devices, et cetera. These devices feature limited memory and power, but high inference speed demands. In this sense, our RPG unstructured pruning method is of great application value.
3.3 Downstream Vision Tasks
We also test our weight pruning method on downstream vision tasks. Mask R-CNN [21] is a widely used benchmark for conventional downstream tasks, namely, object detection and instance segmentation. We try to apply our weight pruning method to Mask R-CNN and compare its detection and segmentation performance against other pruning baselines. As for the choice of baselines, we found that limited weight pruning works conducted experiments on downstream vision tasks. We choose the following baselines for comparison: RigL [14] is a commonly used sparse-to-sparse baseline. AC/DC [40] is good at high-sparsity pruning on ImageNet classification. All methods are applied on Mask R-CNN ResNet-50 FPN variants to measure the mAP for bounding boxes and segmentation masks.
For all Mask R-CNN experiments, we follow the official training of COCO  $1 \times [21]$ : pruning and finetuning lasts for 90K iterations in total. The pruning results evaluated on COCO val2017
are illustrated in Table 4. Similar to the trend in classification experiments, our RPG method gains an advantage at high sparsities compared with AC/DC [40]. As sparsity increases from  $70\%$  to  $80\%$ , the gap between AC/DC and RPG widens from 1.0 to nearly 2.0 for both detection and segmentation mAPs. This finding shows that RPG is a weight pruning method that could be generalized to various vision tasks: it always works well at high sparsities without the need for significant modifications.
Figure 3. ImageNet accuracy versus FLOPs on sparse ResNet-50. Our method achieves better Accuracy-FLOPs trade-off compared with competitive pruning baselines, especially at high sparsities.
3.4 Vision Transformers
Recent works on vision model architectures focus on transformers [13, 52]. Transformer architecture models are proven particularly effective on large-scale image recognition tasks and are well applied
to various downstream tasks [4, 58, 5], but they are still struggling for industrial applications due to large model size and computation cost. To address these problems, works like SViT-E [6] attempted to apply unstructured pruning on vision transformers.
Though our method is not specifically designed for models with the attention mechanism, we explore the effect of our weight pruning method on DeiT-S [52] and compare it with high-sparsity weight pruning baseline [40] and the transformer pruning baseline [6] in Table 5. For fair comparison, all pruning experiments follow the setting of SViT-E [6]: the DeiT-S model is pruned for 600 epochs on ImageNet [9]. All other settings are identical to the official training setting of [52], including batchsize, learning rate, etc.
3.5 Ablations
In this section, we inspect the effect of rank loss. The rank-based pruning objective involves an affine parameter  $\lambda$  that controls the amount of rank loss with respect to the original task loss. When  $\lambda = 0$ , rank loss is turned off. Investigating the relations of rank versus  $\lambda$  and accuracy versus  $\lambda$  on a ResNet-32 of  $99.5\%$  sparsity as shown in Figure 4, we found rank loss could significantly increase the average rank throughout all layers of the sparse network. A substantial increase of accuracy is also observed. But as  $\lambda$  further increases, the average rank will be saturated. Reversely, as  $\lambda$  further increases, the classification accuracy will decrease. This could be attributed to the property of affine combination in Equation (2.7). When  $\lambda$  is large, the pruning objective will pay too much attention to maintain weight ranks and neglect the goal of performing the task well. Hence, it is necessary to tune  $\lambda$  and find the most optimal one.
Type	Time	FLOPs
SVD	16.5min	5.07e15
RPG90%	1003min	1.34e18
Figure 4. Average weight matrix rank of ResNet-32 [22] versus affine hyperparameter  $\lambda$ . Accuracies on CIFAR-10 are marked.
Table 6. SVD overhead compared with the overall pruning & finetuning cost of RPG on  $90\%$  sparse ResNet-50.
Baseline	Sparsity	Train FLOPs
ResNet-50	(Dense)	3.14e18
AC/DC	0.9	0.58×
PowerProp.	0.9	0.49×
RPG(Ours)	0.9	0.43×
Table 7. Training FLOPs comparison with pruning baselines on sparse ResNet-50.
3.6 Overhead Analysis
As introduced in Section 2.4, RPG involves costly SVD calculations. However, we conduct experiments and illustrate that SVD accounts for very minimal cost overhead during pruning in terms of both time and FLOPs. As shown in Table 6, the overall time and FLOPs for SVD calculations only accounts for  $< 2\%$  of the whole RPG pruning cost. We also compare the FLOPs overhead of RPG with other pruning methods. Observing from Table 7, our method is the most cost-effective compared with baselines. Above all, the extra overhead brought by rank loss calculations is not a concern.
This paper proposes the Rank-based Pruning (RPG) method. We investigate weight pruning from a matrix rank perspective, and yield the observation that higher ranks of sparse weight tensors could yield better performance. Based on this heuristic, the adversarial rank loss is designed as the optimization objective that guides the mask update process during pruning. In this manner, our method prunes weight tensors in a rank-favorable fashion. Our RPG method is experimented on various settings and outperforms various baselines.
Unstructured pruning has limited acceleration effect on GPU devices. New GPU architectures or GPU sparse acceleration supports are needed to exert the speed potential of our unstructured pruning method on GPUs.
This work is supported by National Key R&D Program of China under Grant No.2022ZD0160304. We gratefully acknowledge the support of MindSpore [38], CANN (Compute Architecture for Neural Networks) and Ascend AI Processor used for this research. We also gratefully thank Yuxin Zhang and Xiaolong Ma for their generous help.
[57] Xiangyu Zhang, Jianhua Zou, Xiang Ming, Kaiming He, and Jian Sun. Efficient and accurate approximations of nonlinear convolutional networks. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pages 1984-1992. IEEE Computer Society, 2015.
[58] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip H. S. Torr, and Li Zhang. Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 6881-6890. Computer Vision Foundation / IEEE, 2021.
[59] Xiao Zhou, Weizhong Zhang, Hang Xu, and Tong Zhang. Effective sparsification of neural networks with global sparsity constraint. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 3599-3608. Computer Vision Foundation / IEEE, 2021.
[60] Michael Zhu and Suyog Gupta. To prune, or not to prune: Exploring the efficacy of pruning for model compression. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Workshop Track Proceedings. OpenReview.net, 2018.
A Proofs of Theorems
Theorem 3 (The best low-rank approximation). Suppose  $W$  is decomposed via SVD and yield  $W = \sum_{i=1}^{r} \sigma_i u_i v_i^T$  where singular values  $\{\sigma_i\}$  are sorted in descending order. Given integer  $k < r$ , the best  $k$ -rank approximation of  $W$ , namely the  $k$ -rank matrix that has the smallest  $l_2$  distance to  $W$  is
This theorem is also the Eckart-Young-Mirsky Theorem for Frobenius norm.
Though the proof is easily accessible (e.g. from Wikipedia 2), we will provide a sketch of proof here for reference.
Proof. Denote  $W_{k} = \sum_{i=1}^{k} \sigma_{i} u_{i} v_{i}^{T}$  be the best  $k$ -rank approximation of  $W$ ,  $\sigma_{i}(W)$  be the  $i^{th}$  largest singular value of  $W$ . Then the low-rank approximation error could be reduced as follows:
$$
\left\| W - \widetilde {W} \right\| _ {F} ^ {2} = \left\| \sum_ {i = k + 1} ^ {r} \sigma_ {i} u _ {i} v _ {i} ^ {T} \right\| _ {F} ^ {2} = \sum_ {i = k + 1} ^ {r} \sigma_ {i} ^ {2}. \tag {A.1}
$$
Given  $W = W' + W''$ , according to the triangle inequality of spectral norm,
$$
\sigma_ {1} (W) = \sigma_ {1} \left(W ^ {\prime}\right) + \sigma_ {1} \left(W ^ {\prime \prime}\right). \tag {A.2}
$$
Then for two arbitrary ranks  $i,j\geq 1$  , we have:
$$
\begin{array}{l} \sigma_ {i} \left(W ^ {\prime}\right) + \sigma_ {j} \left(W ^ {\prime \prime}\right) = \sigma_ {1} \left(W ^ {\prime} - W _ {i - 1} ^ {\prime}\right) + \sigma_ {1} \left(W ^ {\prime \prime} - W _ {j - 1} ^ {\prime \prime}\right) \\ \geq \sigma_ {1} \left(W - W _ {i - 1} ^ {\prime} - W _ {j - 1} ^ {\prime \prime}\right) \tag {A.3} \\ \geq \sigma_ {1} \left(W - W _ {i + j - 2}\right) \\ = \sigma_ {i + j - 1} (W). \\ \end{array}
$$
Assume there is another  $k$ -rank approximation  $X$ , Then according to the above formula, for arbitrary  $i \geq 1$ ,
$$
\sigma_ {i} (W - X) = \sigma_ {i} (W - X) + \sigma_ {k + 1} (X) \geq \sigma_ {k + i} (W) \tag {A.4}
$$
$$
\left\| W - X \right\| _ {F} ^ {2} \geq \sum_ {i = 1} ^ {n} \sigma_ {i} (W - X) ^ {2} \geq \sum_ {i = k + 1} ^ {n} \sigma_ {i} ^ {2}, \tag {A.5}
$$
which means  $\widetilde{W}$  is the best  $k$ -rank approximation.
Theorem 2 states the effectiveness of rank loss. Before the theorem, recall notations that we previously defined:  $\overline{W} := \frac{W}{\|W\|}$  is the  $l_2$  normalized weight matrix  $W$ ;  $U, \Sigma, V$  are matrices reached from the SVD of  $\overline{W}$ , where  $U = \{u_1, u_2, \ldots\}$  and  $V = \{v_1, v_2, \ldots\}$  are orthonormal bases;  $\Sigma$  is a diagonal matrix where singular values  $\{\sigma_1, \sigma_2, \ldots\}$  are sorted in descending order on the diagonal; operator  $\mathrm{Trun}(U\Sigma V^T) = \sum_{i=1}^{k} \sigma_i u_i v_i^T$  stands for  $k$ -rank truncated SVD, or the  $k$ -rank best approximation of  $\overline{W}$  according to Theorem 1.
$^{2}$ https://en.wikipedia.org/wiki/Low-rank_approximation
Theorem 4 (Effectiveness of the adversarial rank loss). Given the adversarial rank loss
$$
\mathcal {L} _ {\text {r a n k}} = - \left\| \overline {{W}} - \operatorname {T r u n} \left(U \Sigma V ^ {T}\right) \right\| _ {F} ^ {2}. \tag {A.6}
$$
If we optimize  $W$  in rank loss via gradient descent, the rank of  $W$  will increase.
Proof. In gradient descent, the update from weight  $W$  to  $W'$  based on rank loss  $\mathcal{L}_{\text{rank}}$  could be described as:
$$
W ^ {\prime} = W - \gamma \frac {\partial \mathcal {L} _ {\text {r a n k}}}{\partial W}. \tag {A.7}
$$
We first simplify rank loss. Since  $U = \{u_{1}, u_{2}, \ldots\}$  and  $V = \{v_{1}, v_{2}, \ldots\}$  are orthonormal bases, we could easily rewrite rank loss  $\mathcal{L}_{rank}$  as the squared sum of small singular values:
$$
\begin{array}{l} \mathcal {L} _ {r a n k} = - \| \frac {W}{\| W \| _ {F}} - \sum_ {i = 1} ^ {k} \sigma_ {i} u _ {i} v _ {i} ^ {T} \| _ {F} ^ {2} \\ = - \left\| U \Sigma V ^ {T} - U \Sigma_ {[ 1: k ]} V ^ {T} \right\| _ {F} ^ {2} \tag {A.8} \\ = - \| \Sigma - \Sigma_ {[ 1: k ]} \| _ {F} ^ {2} = - \sum_ {i = k + 1} ^ {r} \sigma_ {i} ^ {2}. \\ \end{array}
$$
The form Equation (A.8) allows us to apply chain rule to calculate the gradient of the normalized weight matrix  $\frac{\partial\mathcal{L}_{rank}}{\partial\overline{W}}$ :
$$
\frac {\partial \mathcal {L} _ {\text {r a n k}}}{\partial \bar {W}} = \sum_ {i = k + 1} ^ {r} \frac {\partial \mathcal {L} _ {\text {r a n k}}}{\partial \sigma_ {i}} \frac {\partial \sigma_ {i}}{\partial \bar {W}} = - \sum_ {i = k + 1} ^ {r} 2 \sigma_ {i} u _ {i} v _ {i} ^ {T}. \tag {A.9}
$$
Again, the chain rule is applied for the derivative of the weight matrix. For clarity, we show the gradient expression for a single weight parameter  $W[m,n]$  (the weight value at position  $(m,n)$  in the reshaped weight matrix  $W$ ):
$$
\begin{array}{l} \frac {\partial \mathcal {L} _ {r a n k}}{\partial W [ m , n ]} = t r \left(\frac {\partial \mathcal {L} _ {r a n k}}{\partial \overline {{W}} ^ {T}} \frac {\partial \overline {{W}}}{\partial W [ m , n ]}\right) \\ = - \frac {\left(\sum_ {i = k + 1} ^ {r} 2 \sigma_ {i} u _ {i} v _ {i} ^ {T}\right) [ m , n ]}{\| W \| _ {F}} \tag {A.10} \\ + \frac {W [ m , n ] \sum \left(W \odot \left(\sum_ {i = k + 1} ^ {r} 2 \sigma_ {i} u _ {i} v _ {i} ^ {T}\right)\right)}{\| W \| _ {F} ^ {3}}. \\ \end{array}
$$
Based on the gradient, one step of optimization under learning rate  $\alpha$  could be expressed in a neat matrix multiplication format, decomposed by orthonormal bases  $U = \{u_{1}, u_{2}, \ldots\}$  and  $V = \{v_{1}, v_{2}, \ldots\}$ .
Lemma 1 (Weight optimization based on rank loss). Denote  $\sum \left(W\odot \left(\sum_{i = k + 1}^{r}2\sigma_{i}u_{i}v_{i}^{T}\right)\right)\coloneqq c,$  which is a scalar constant within each step, one step of weight  $W$  optimization under learning rate
$\gamma > 0$  and rank loss (as defined in Equation (2.6)) could be expressed as:
$$
\begin{array}{l} W ^ {\prime} = W - \gamma \frac {\partial \mathcal {L} _ {r a n k}}{\partial W} \\ = W - \gamma \left(- \frac {\sum_ {i = k + 1} ^ {r} 2 \sigma_ {i} u _ {i} v _ {i} ^ {T}}{\| W \| _ {F}} + \frac {c W}{\| W \| _ {F} ^ {3}}\right) \\ = U \Sigma V ^ {T} + \frac {2 \gamma}{\| W \| _ {F}} U \Sigma_ {[ k + 1: r ]} V ^ {T} - \frac {c \gamma}{\| W \| _ {F} ^ {3}} U \Sigma V ^ {T} \tag {A.11} \\ = U \left(\left(1 - \frac {c \gamma}{\| W \| _ {F} ^ {3}}\right) \Sigma + \frac {2 \gamma}{\| W \| _ {F}} \Sigma_ {[ k + 1: r ]}\right) V ^ {T}. \\ \end{array}
$$
From the formula of the optimized weight  $W'$ , we can reach the following conclusions on the optimized weight  $W'$ : firstly,  $W'$  promises the same set of orthonormal bases  $U, V$  after SVD; secondly, comparing small singular values against others, all singular values are penalized by the same amount  $\frac{c\gamma}{\|W\|_F^3}$ ; but the small singular values (ranking from  $k + 1$  to  $r$ ) are awarded with increments  $\frac{2\gamma}{\|W\|_F}$ . Regardless of swapped singular values due to magnitude change (because of small learning rate  $\gamma$ ), small singular values will make up more proportion in all the singular values after one step of update. Recall the definition for  $\delta$ -rank, given fixed sum of squared singular values after  $l_2$  normalization, the rank of  $W$  will increase.
B.1 Details on Gradient-Grow.
We explain in details about the procedure of gradient grow evolved from RigL [14]. At each gradient grow step, first we calculate the Rank-based Pruning (RPG) objective  $\mathcal{L}$ . Then we back-propagate  $\mathcal{L}$  for gradients of network weights. Finally, gradients of pruned network weights are sorted in magnitudes; weights with large gradients are re-activated.
The number of weights to be re-activated are determined by the number of remaining weights. The whole pruning framework is detailed in Algorithm 1. Grow fraction  $\alpha$  is a function of training iterations that gradually decays for stability of training. Cosine Annealing is used for  $\alpha$ , following [14].
B.2 Selection of Approximation Rank
Factor  $k$  of the truncation operator  $\mathrm{Trun}$  controls the rank of the low-rank approximation in this adversarial process. However, controlling  $k$  for each layer is hardly practical, because layers are large in quantity and vary in shapes. We leverage the concept of  $\delta$ -rank in Definition 1, and tend to control the approximation error (also the negative rank loss  $-\mathcal{L}_{\text{rank}}$  for a layer) rather than control  $k$ . We set a constant  $\tilde{\delta}$  between  $(0,1)$  as the target approximation error between the normalized weight matrix  $\overline{W}$  and its low rank counterpart. Then we find the best  $k$  that has the closest low-rank approximation error as  $\tilde{\delta}$  for each layer. Mathematically, this selection process could be written as:
$$
\underset {k} {\arg \min } | - \mathcal {L} _ {\text {r a n k}} - \tilde {\delta} |. \tag {B.1}
$$
C.1 CIFAR Experiments
In the CIFAR experiments, we train VGG-19 and ResNet-32 for 300 epoch, which is identical to Zhou et al. [59]. We use the SGD optimizer with momentum 0.9, batchsize 128, learning rate 0.1, and weight decay 0.005.
Algorithm 1: Rank-based Pruning (RPG)
Input :A dense model  $W$  with  $n$  layers; Target density (a function of iteration)  $d$ ; Grow fraction (a function of iteration)  $\alpha$ ; Mask update interval  $\Delta T$ ; Total training iteration  $T = T_{prune} + T_{finetune}$   
Output: A Sparse Model  $W \odot M$   
1 // Initialize a dense mask;  
2  $M \gets 1$ ;  
3 // Stage 1: prune and grow;  
4 for  $t \gets 1$  to  $T_{prune}$  do  
5 Forward propagation for minibatch loss  $\mathcal{L}_{task}$ ;  
6 if  $t\% \Delta T = 0$  then  
7 // Update mask  $M$ ;  
8 Calculate and sum layerwise rank loss;  
9 Keep top  $d_t$  proportion of weights in  $W$  globally to get layerwise density  $d_t^i$ ;  
10 for  $i \gets 1$  to  $n$  do  
11 Prune to density  $(1 - \alpha_t)d_t^i$  based on  $|W|$ ;  
12 Grow to density  $d_t^i$  again based on  $|\nabla \mathcal{L}|$ ;  
13 end for  
14 end if  
15 else  
16 Train sparse net with task loss;  
17 end if  
18 end for  
19 // Stage 2: keep masks fixed and finetune;  
20 for  $t \gets T_{prune} + 1$  to  $T$  do  
21 Keep mask  $M$  static, and train weight  $W$  till converge;  
22 end for  
23 return Weights of the Pruned Model  $W \odot M$
C.2 ResNet-50 on ImageNet
We mainly follow Goyal et al. [18] for ImageNet experiments. ImageNet experiments are run on 8 NVIDIA Tesla V100s. Our RPG method is applied on ResNet-50 for 100 epoch, and compared with 100-epoch baselines at various sparsities. We use the SGD optimizer with momentum 0.9, 1024 batchsize, a learning rate of 0.4, 0.0001 weight decay, 0.1 label smoothing, 5 epochs of learning rate warmup. We choose 1024 batchsize instead of the commonly-used 4096 batchsize [14, 45] due to GPU memory constraints. We did not use augmentation tricks like mixup or cutmix. Standard pretrained model from torchvision is used in ImageNet setting for fair comparison with top baselines.  $\alpha$  is kept at 0.3;  $\Delta T$  is set to 100;  $T_{prune}$  is set to 90 epochs.
C.3 Downstream Vision Tasks
In Mask R-CNN experiments, all methods are implemented by us on the identical settings for fair comparison. We follow the training setting of the original Mask R-CNN [21] on Detectron-2 [54]. The three methods are applied for 96000 iterations. Notably, certain components in Mask R-CNN ResNet-50 FPN are loaded from pretrained models and left untrained. These components are omitted from pruning and sparsity calculations for RPG and baseline experiments.
C.4 Vision Transformers
In vision transformers, we mainly follow the official setting of DeiT [52], but we extend training epochs from 300 to 600 for fair comparison with SViT-E [6]. All other training settings are identical to the official training setting of DeiT [52].
C.5 Replication of AC/DC