Towards Higher Ranks via Adversarial Weight Pruning
Authors: Yuchuan Tian
Hanting Chen
Tianyu Guo
Chao Xu
Yunhe Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: Weight Pruning
Matrix Rank

Abstract: Convolutional Neural Networks (CNNs) are hard to deploy on edge devices due to its high computation and storage complexities. As a common practice for model compression, network pruning consists of two major categories: unstructured and structured pruning, where unstructured pruning constantly performs better. However, unstructured pruning presents a structured pattern at high pruning rates, which limits its performance. To this end, we propose a Rank-based PruninG (RPG) method to maintain the ranks of sparse weights in an adversarial manner. In each step, we minimize the low-rank approximation error for the weight matrices using singular value decomposition, and maximize their distance by pushing the weight matrices away from its low rank approximation. This rank-based optimization objective guides sparse weights towards a high-rank topology. The proposed method is conducted in a gradual pruning fashion to stabilize the change of rank during training. Experimental results on various datasets and different tasks demonstrate the effectiveness of our algorithm in high sparsity. The proposed RPG outperforms the state-of-the-art performance by 1.13\% top-1 accuracy on ImageNet in ResNet-50 with 98\% sparsity. The codes are available at https://github.com/huawei-noah/Efficient-Computing/tree/master/Pruning/RPG and https://gitee.com/mindspore/models/tree/master/research/cv/RPG.

Towards Higher Ranks via Adversarial Weight Pruning
Yuchuan Tian', Hanting Chen', Tianyu Guo', Chao Xu', Yunhe Wang2* National Key Lab of General AI, School of Intelligence Science and Technology, Peking University.
Huawei Noah's Ark Lab.
t anyc@stu. pku. edu. cn, {chenhanting, tianyu. socyyuugu.yya. guo.guuue....... wagg) xuchaoêcis, pku. edu.cn cn
Abstract
Convolutional Neural Networks (CNNs) are hard deploy on edge devices due to its high computation and storage complexities. As a common practice for model compression, network pruning consists of two major categories: unstrue- tured and structured pruning, where unstructured pruning constantly performs better. However, unstructured pruning presents a structured pattern high prun- ing rates, which limits its performance. To this end, we propose a Rank-based PruninG (RPG) method to maintain the ranks of sparse weights in an adversar- ial manner. In each step, we minimize the low-rank approximation error for the weight matrices using singular value decomposition, and maximize their distance by pushing the weight matrices away from its low rank approximation. This rank-based optimization objective guides sparse weights towards high-rank topol- ogy. The proposed method is conducted in a gradual pruning fashion to stabilize the change of rank during training. Experimental results on various datasets and different tasks demonstrate the effectiveness of our algorithm in high sparsity. The proposed RPG outperforms the state-of-the-e performance by 1.13% top-1 accuracy ImageNet in ResNet-50 with 98% sparsity. The codes are avail- able at https: //gttbb.. www.comm/aaaa wttt:///ffiiiiieeeoooooooiiie//a master Cesster/Pruningg RPG and http::///gcccceccccc//a/cpp/r//p master//esearcc/ccr
1 Introduction
As Convolutional Neural Networks (CNNs) are adapted to various tasks better performance, their sizes also explode accordingly. From shallow CNNs like LeNet 30 larger CNNs like AlexNet [27], to deeper modern CNNs like ResNets 22 and DenseNets 25 CNNs are growing larger for more complex tasks and representations, including large-scale image classification and downstream tasks like object detection 422, segmentation (21, etc. The evolution of CNN gives rise various real- world applications, such as autonomous driving [2]. camera image processing 3), optical character recognition [46), and facial recognition [50]. However,
difficult to deploy large CNN models on mobile devices since they require heavy storage and computation. For example, deploying 5e 22 model costs 8.2G FLOPs for processing a single image with 224 x 224 size, which unaffordable for edge devices with limited computing power such as cellphones and drones.
In order to compress heavy deep models, various methodologies have been proposed, including Weight quantization [99 (6), knowledge distillation [24 44), and network pruning. Network pruning prunes the redundant weights in convolutional neural networks shrink models. Weight (or unstructured) pruning 220 and filter (or structured) pruning 32 are two main pathways to prune CNNs. Weight
"Corresponding Author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

pruning sparsifies dense kernel weight tensors in convolutional layers in an unstructured manner including iterative pruning 20], gradual pruning 66 17. and iterative rewinding [5]] 16 43 Some other works [1
53 48
propose gradient or hessian based weight saliencies that proved effective in prunes filters in convolutional layers as whole,
certain scenarios. Filter pruning
[37 23 35
reducing the redundant width of network layers.
Although structured pruning algorithms can be well supported by existing hardwares and bring large runtime acceleration benefits, their per- formance is much lower than that of unstruc- tured pruning. For example, SOTA unstructured pruning methods could achieve 80% sparsity on ResNet-50 with little performance drop [48 while structured pruning could only reach less than 50% [49]. since filter pruning subset of weight pruning by further imposing struc- tural constraints. However, under circumstances of high sparsities, we observe that unstructured pruning partially degrade to structured pruning. When weights with large proportion of ros, it is highly likely that a structured pattern appears, where whole channel filter is al- most completely pruned. Therefore, existing weight pruning methods usually meet dramatic performance decay at high sparsities.
40
Avg.
Oneshot
Rigl.
AC/D
RPG(Ours)
0.95
0.96
0.97
0.98
0.99
1.00
Sparsity
Average weight matrix rank of ResNet- pruning baselines versus sparsity. Our rank- based method is effective in maintaining weight ranks at high sparsities compared with baselines.
Figure 32 (22
Inspired by the comparison of the two pruning categories, we propose to reduce structural pat- terns in weight pruning. Structured pruning is factually a reduction of weight rank in deep Con- metric vnets. Thus, rank could be adopted unstruc- for evaluating the "structuredness" tured sparse weights: sparse weight is consid- ered highly structured it possesses low rank. To keep unstructured pruning from being too structured, we hope to maintain weight ranks under high sparsities in pruning. Based the goal of rank improvement, we propose an ad- versarial Rank-based PruninG (RPG) approach for unstructured pruning. First, we find a low- rank approximation of the weight by minimiz- ing the approximation error. The best low-rank approximation is found via singular value de- composition. Second, to enhance weight ranks, we maximize the distance between the weight and its low-rank counterpart to increase weight rank. This adversarial rank-based optimization objective guides sparse weights towards high-
Min- max optimization
X
Sparse weights
Low-rank approximation
Figure 2. An illustrative diagram of our Rank-based Pruning (RPG) method.
rank topology. The proposed method is conducted in a gradual pruning fashion to stabilize the change of rank during training. The advantage of the proposed RPG method is evaluated through extensive experiments on image classification and downstream tasks, and Figure demonstrates that method gains matrix rank advantage compared to basclines.
Maintaining Rank via Adversarial Pruning
2.1 Problem Formulation
In conventional terms of supervised neural network learning, given a target loss function L, neural network weight W, and input output pairs X {a:) Y {y hist... weight training of

---

neural network W is formulated as:
arg min C(Y, W W
(2.1)
Weight pruning limits the total number of non-zero weights in weight W; or mathematically, weight pruning imposes lo-norm constraint neural network learning. Given sparsity budget c, the constraint is described as:
[ S c,
(2.2)
common practice to reparameterize weight W with the Hadamard elementwise product of weight tensor W and a binary mask M. The binary mask M has the same shape as W, and each element M represents whether its corresponding parameter in W is pruned. After reparametrrizaiionn, the weight pruning problem is then formulated as:
arg min L(Y, (W @ M)X) s.t. |M c. WBM
(2.3)
In Equation 2.3 @ is the Hadamard elementwise product of matrices.
At high sparsities in unstructured pruning, the rank of sparse networks could decrease substantially. In the following sections, we will demonstrate the problem and propose a solution to maintain sparse weight ranks.
2.2 Analyzing Weight Pruning in High Sparsity
Unstructured and structured pruning are two major pruning methodologies. In unstructured pruning practices, weight tensors of CNNs pruned in fine-grained manner: each and every solitary weight parameters could be turned off (i.e. set to zero) within the network, but the whole weight tensor structure left unchanged. In contrast, structured pruning focuses on the pruning of filters: filters are cut-off as the smallest prunable unit in the pruning process. Comparing the two pruning paradigms under the same sparsity budget, Zhu and Gupta [60] illustrate that unstructured pruning performs much better than structured pruning under the same pruning budget.
This phenomenon could be explained from the perspective of matrix ranks. In fact, structured pruning a direct rank reduce imposed on weight matrices, which means filter pruning is basically weight pruning with low rank. The rank of matrix represents the upper bound of the amount of information contained in the matrix. A powerful network should be rich in information, and we hope features of the sparse network could have high ranks. Feature ranks is closely related to ranks of sparse weight matrices because of the formula below that describes the relationship of ranks in matrix multiplication:
Rank (Y) Rank (W S min (Rank (W), Rank (X))
(2.4)
According to Equation 2.4 when filter pruning is applied on weight W that directly impacts its rank, the rank of the output feature will also degrade, causing dramatic loss information richness. On the other hand, unstructured pruning is free from the structural constraint of filter pruning, and thus maintain more amount of information.
However, under circumstances of high sparsities, we observe that unstructured pruning partially degrades structured pruning. When weights are filled with large proportion of zeros, it is very probably that some filters or channels are almost entirely turned-off: "quasi-structured" sparse weight pattern then formed. A baseline evaluation of matrix ranks in Figure [ illlustrates this concern. Therefore, existing weight pruning methods usually meet dramatic performance decay at high sparsities. Inspired by the properties of the two categories of pruning, we propose reduce the structured pattern in unstructured pruning, and therefore to maintain weight ranks under high sparsities.
2.3 Low Rank Approximation through SVD
Now that weight ranks are important in weight pruning, we need a practical way compute ranks in the context of deep neural networks. Previous deep learning works on ranks apply matrix rank theories to CNN low-rank tensor decomposition. In these works, low-rank approximations proposed to fit
3

---

weight tensors. Denil et al. [10] decomposites W E Rmxx into the multiplication of U E R and V E _= where r is much smaller than and n. UV provides low-rank approximation (rank bounded by r) of weight W. Denton et al. [ uses the sum of k rank-one approximations to provide the k-rank approximation of a feature tensor. Zhang et al. [57] multiplies a low-rank matrix M to weight matrix W, and solves the low-rank M with Singular Value Decomposition (SVD). In modern works [1 71, low-rank approximations are widely studied as well.
Since the weight values are always discrete, as an alternative solution and inspired by low-rank approximation works, we converge an approximated rank rather than compute precise rank solution. Hence, we define the approximated rank as following:
Definition (ő-rank of a matrix). Given a matrix W and a small error tolarance 6 > O, the 6-rank of W is defined the smallest positive integer ic such that there exist a k-rank matrix, whose l2 distance to W is smaller than
In previous works, ranks evaluated via singular values computed from Singular Value Decom- position (SVD). Zhang et al. [57] uses the sum of the top-k PCA eigenvalues to approximate ranks of layer responses; Lin et al. 33 defines rank the number of non nogleggliii singular values and does SVD analysis on feature maps; Shu et al. 47 performs SVD on attention maps and augment model performance by keeping fatter tail in the singular value distribution. These discoveries all acknowledge that singular values from SVD estimates ranks of matrices. We also leverage SVD) to compute 5-rank defined in Definition First, we illustrate that SVD could generate the best low-rank approximation:
Theorem (The best low-rank approximation). Suppose W is decomposed via SVD and yield W = E duun', where singular values }a,} are sorted in descending order. Given integer k < the best k-rank approximation of W, namely the k-rank matrix that has the smallest l2 distance to is
Σ ao.eo
The proof of Theorem I will be shown in Appendix. Since SVD could yield the best low-rank approximation, we could use this property to solve 6-rank defined in Definition Given weight matrix W. we search for the smallest & such that the l2 approximation error of best k-rank approximation W as formulated in Theorem is below the error tolerance 8. In this way, are able to solve the rank of W.
2.4 Adversarial Optimization for Rank Maintenance
Equipped with the method for matrix rank computation, we hope to formulate target loss function according to this heuristic such that optimization the loss could maintain weight ranks.
In contrast to low-rank approximations, high-rank matrices should be hard for low-rank matrices to approximate. Assume S is the set of all low-rank matrices, W should keep its distance away from this set S to increase its rank. But this is a hard problem. for we have to figure out all low-rank matrices. To further simplify the problem, we find the best low-rank approximation rather than all low-rank approximations. W should estrange itself from the best low-rank approximation whose distance is the farthest from W. This simplification is valid and will be proved later.
Using this heuristic as motivation, we design an adversarial mechanism that increase the difficulty for W to be approximated by low-rank matrices, and consequently to advocate higher matrix ranks of W while pruning. At first, the best low-rank approximation W of a small rank l is generated via Singular Value Decomposition, for the purpose of minimizing its distance to weight W; next, W is optimized increase the distance from W. The procedures could be understood as an adversarial combat between W and W: as the low-rank W tries to W. W is optimized to keep itself far away from W. Mathematically, the combat could be expressed as a min-max problem.
But unluckily, the problem may suffer the risk of not getting converged. When W is fixed, the best W is taken when W oc. To resolve this issue during optimization, we constrain W within euclidean norm ball. In other words, we plug JW]F instead of W into the max-min problem. The
W
reasons we use le normalization are: 1. W is bounded rather than growing to infinity; 2. the rank of

---

W could increase if we la normalize W when optimizing the min-mu problem, which will be shown in the mathematical proof in the appendix; 3. l2 normalization on weight is equivalent to imposing l2 normalization on its singular values, providing fair standard for rank comparisons based on the definition of rank in Definition given fixed error tolerance.
Before the introduction of this min-max problem, we introduce several notations: || || is the Frobenius norm (2-norm) of matrices; / is the identity matrix; W := W
[WT is the l2 normalized weight matrix W; U, Σ. V are matrices reached from the SVD of W, where U = fui, u2....) and V = {Uz, ... ...) are orthonormal bases; Σ is diagonal matrix where singular values for, o2. are sorted in descending order on the diagonal; operator Trun (UEV?) = Σ a,nes? stands for k-rank truncated SVD, or the k-rank best approximation of W.
Then formally, we express the min-max problem as follows:
I Trun (UEVT) 1/2.
min max W U,E.V
(2.5)
W
s.t. UT-=I, VV==I, W
[W]
The optimization target defined as the adversarial rank loss:
WW Trun (UEVT) 1
Frank
(2.6)
In deep learning, gradient descent the most widely applied method for optimization problems, and we also adopt gradient descent for our experiments. Hence in this context, we propose the following theorem, stating that our adversarial rank loss could guide weight W towards higher rank:
Theorem (Effectiveness of the adversarial rank loss). Given the adversarial rank loss as defined in Equation (.6). If we optimize W in rank loss via gradient descent, the rank of W will increase.
The theorem could be mathematically proved, and the detailed proof will be provided in the appendix.
With the proposed adversarial rank loss, our optimization objective consists of two goals: 1. we hope to reduce the loss for a certain task (e.g. classification,, detection, etc.) for good sparse network performance; 2. we hope to reduce rank loss for higher weight ranks. We formulate the Rank-based Pruning objective by doing affine combination of the two goals. Given affine hyperparmeter X, the loss for certain task Ltask, the adversarial rank loss Lranks the Rank-based Pruning (RPG) objective defined as:
C == Leask + ALrank-
(2.7)
2.5
The Gradual Pruning Framework
Previous works have proposed various pruning framework, including One-shot Pruning 20 Sparse- to-sparse Pruning g 1 39 12 [4], and Iterative Magnitude Pruning for Lottery Tickets [15] 43
Compared with these frameworks, Gradual Pruning (GP) 60 could reach better performance with modest training budget. We adopt Gradual Pruning as the pruning framework, which is usual practice in many works [8 59
GP prunes small portion of weights once every AT training steps, trying to maintain sparse network performance via iterative "pruning and training" procedures.
we hope the factor of rank could be considered in the choice of weights via the proposed rank loss. Loss gradients are widely-applied weight saliency criteria, because gradient magnitudes reflect the potential importance of pruned weights: if turned-off weight possesses large gradients with respect to the objective loss function, it expected for significant contributions to loss reduction [14. We use periodic gradient-bart weight grow similar to previous pruning works [14] 34, 6 i.c. the weights are periodicly grown at each binary mask update step. But differently, the rank-based pruning objective (defined as Equation (2.7) is used for gradients computation with respect to each model weight in our case. In this way, the rank factor is considered during the selection of active weights: there is tendency that RPG chooses an active set of weights that features high-rank.
However, it is hard to associate rank loss with Gradual Pruming:
5

---

Models
VGG19
ResNet32
Sparsity
99%
99.5%
99.9%
99%
99.5%
99.9%
Dense
93.84
94.78
PBW
19
90.89
10.00
10.00
77.03
73.03
38.64
MLPrune [55]
91.44
88.18
65.38
76.88
67.66
36.09
ProbMask [59]
93,38
92.65
89.79
91.79
89.34
76.87
AC/DC 40
93.35
80.38
78.91
91.97
88.91
85.07
RPG (Ours)
93.62
93.13
90.49
91.61
91.14
89.36
Table 1. Sparsified VGG-19 and ResNet-32 on CIFAR-10. Baseline results obtained from 59
An embedded benefit of periodic gradient-based weight grow lies in computation cost considerations. Singular Value Decomposition (SVD) that is essential for rank computation is costly for large weight tensors. Calculating rank loss for each optimization step is hardly affordable. The adoption of periodic weight updating. however, amortizes the cost of rank loss computations. We also provide an SVD overhead analysis in 3.6
In summary, Our Rank-based Pruning (RPG) method is formulated as follows: once every AT training steps, the prune-and-grow procedures that updates binary mask M is performed. Firstly, we plan the number of parameters to prune and to grow, such that after mask updating, the whole network will reach the target sparsity at the current iteration. Target sparsity will increase gradually as training goes on, which is identical to GP. Secondly, we globally sort all parameters based on magnitude and perform the pruning operation. Thirdly, we grow the parameters based on gradient. For other training steps, mask M is left unchanged; the active weight values are updated.
Specifically, HRank [33 also leverages matrix rank evaluations in pruning. Our idea is significantly different from HRank (33] in the following aspects: 1. HRank performs filter pruning while our work focuses weight pruning: 2. HRank evaluates ranks of feature maps, but we evaluate ranks of weight tensors; 3. HRank uses feature rank as filter saliency; our work uses weight rank to guide the update of a sparse network topology.
Experiments
Our Rank-hased PruninG (RPG) method is evaluated several behchmarks and proved outstanding among recent unstructured pruning baselines. This section presents the experiment results to empiri- cally prove the effectiveness our RPG method, especially on high sparsities. First, we will show the results of RPG on two image classification datasets: the comparatively small-scaled CIFAR- 10, and the large-scaled ImageNet. Then, we will present the results of RPG on downstream vision tasks. Finally, an ablation study will be given.
3.1 CIFAR Experiments
Experiment settings. We first compare our RPG pruning method with other methods on CIFAR-10 classification. CIFAR-10 is one of the most widely used benchmark for image classification. It consists of 60000 32 x 32 images: 50000 for training, and 10000 for validation. We hope to try our RPG method first this relatively small dataset and look for heuristic patterns.
Among the pruning baselines, we choose ProbMask [59 and AC/DC [40] comparison because these two methods intended for high-sparsity pruning. Additionally, ProbMask is recent baseline that provides both CIFAR and ImageNet classification results, enabling us for further investigation on larger-scale datasets. Other baselines including PBW [19] and MLPrune [55] are earlier conventional pruning baselines for references. For fair comparison, our RPG method is applied to modern CNN structures, Lee VGG-19 19 and ResNet-32, and prune for 300 epochs, according to the setting of ProbMask [59]. The results are shown in Table
Results analysis. At relatively low sparsities, the gap between recent baselines are small. Prob- Mask [59], AC/DC 40), and RPG all give satisfactory results at 99% compared with early pruning works, But sparsity further increases, the three methods undergo significant performance decay

---

either network. At 99.5% and 99.9%. our RPG method shows great advantage over the other two baselines. This discovery inspires us further investigate the high-sparsity potential of RPG on the large-scale ImageNet dataset.
3.2 ResNet-50 on ImageNet
Experiment settings. Sparse ResNet-50 networks evaluated on the ImageNet dataset are the most commonly-used and recognized weight pruning benchmarks. ImageNet ISLVRC2012 [9] is large scale image classification dataset. It contains 128 IK
Algorithm
Sparsity Accuracy
ResNet-50
76.80
STR 29
0.8
76.19
images the training set and 50K images in the val-
WoodFisher 48
0.8
76.73
idation set. All the images are shaped 224 x 224
GraNet 34
0.8
76.00
and distributed in 1000 classes. ResNet-50 22 is a
AC/DC 40
0.8
76.30
medium-sia size canonical CNN with 25.5M parameters and 8.2G FLOPs, designed for ImageNet classifica- tion.
PowerPropagation
45
0.8
76.24
RPG (Ours)
0.8
76.66
STR 29
0.9
74.31
Our RPG method is applied ResNet-50 under high sparsities: 80%, 90%, 95%, and 98%. We com-
WoodFisher
48
0.9
75.26
GraNet 34
0.9
74,50
pare RPG with recent baselines. Among the base-
AC/DC 40
0.9
75.03
lines, STR [29] automatically learns pruning sparsity:
ProbMask [59
0.9
74.68
WoodFisher 48]. GraNet 3 and ProbMask [60]
PowerPropagation RPG (Ours)
45
0.9
75.23
are methods based on gradual pruning; AC/DC (40]
0.9
75.80
and ProbMask [60] are baselines targeted high sparsities; PowerPropagation [45] is an improvement of KAST [26] that relies on a pre-set layerwise sparsity distribution. For fair comparison, all results are 100-epoch baselines; we used standard ImageNet configs, detailed in the Appendix. The results are presented in Table 2 The advantage of adversarial rank-based pruning manifested at high sparsities.
STR 29
0.95
70.40
WoodFisher 48
0.95
72.16
AC/DC 40
0.95
73.14
ProbMask (59
0.95
71.50
PowerPropagation RPG (Ours)
45
0.95
73.25
0.95
74.05
STR 29
0.98
62.84
Results analysis. Our method could achieve out- standing performance for sparsities 90%, 95%, and 98%. At lower sparsities (e.g. 80%, 90%), Wood- Fisher [48] takes the lead among the baselines. Our RPG method is slightly lower than WoodFisher [48] by 0.07% in ImageNet accuracy at 80% sparsity. At higher sparsities, our method outcompetes other base- lines. Other competitive baselines at high sparsi- ties include PowerPropagation [45) and AC/DC [40]. However, the gap between our RPG method and these baselines widened at high sparsities. Specificlly, our method outperforms current top baseline by 1.13% of ImageNet 1 accuracy at 98% sparsity.
WoodFisher 48
0.98
65.55
AC/DC 40
0.98
68.44
ProbMask 559
0.98
66.83
PowerPropagation 45
0.98
68.00
RPG (Ours)
0.98
69.57
Table Sparsified ResNet-50 on ImageNet. results are official reports from original works. Best and second hest results are bolded and underlined.
Firdo-Reeni--Kann (ERK) [4] is a layerwise sparsity distribution that is commonly used for performance boosting in weight pruning methods that require a pre-set sparsity distribution. However, ERK-based sparse models are computationally costly. Differently, RPG automatically maintains a more balanced sparsity throughout the whole network under the same total sparsity constraint. Though sparse model slightly lags behind the current ERK variant of SOTA (45) under lower sparsitics in certain accuracy, it much cost-effective. Quantatitively, for 80% sparse ResNet-50, the reported ERK-based State-of-ther Art ImageNet accuracy is merely 0.10% higher than our RPG method (reaching 76.77 76% for [45), but costing an extra 58% of FLOPs. The advantage of our RPG method over ERK- based methods clearly illustrated in Figure 3] where compare RPG with the ERK variant of TOP-KAST 266 and the State-of-ther tfftthe-Art PowerPropagaatin
[45
is recent sparse acceration framework on CPU that makes unstructured-sparr network accerlation possible in applications. We time sparse ResNet-50 on DeepSparse for single- image inference. Results in Table 3 shows that highly-sparse ResNet-50 could achieve around
DeepSparse 28
7

---

Algorithm
Sp.
BoxAP MaskAP
Algorithm Sp. Accuracy
Sp.
Acc.
Runtime
Mask R-CNN
38.6
35.2
DeïT-S 52
79.85
Dense
76.80
40.25ms
RigL [4
0.5
36.4
32.8
SViT-E 6
0.5
79.72
AC/DC 40
0.5
37.9
34.6
AC/DC 40
0.5
80.15
0.8
76,66
39.26ms
RPG (Ours)
0.5
37.7
34.4
RPG (Ours)
0.5
80.15
0.9
75,80
27.98ms
0.95
74.05
22.20mm 20ms
RigL [4]
0.7
32.3
29.1
SViT-E 6
0.6
79.41
0.98
69.57 20.89ms
AC/DC 40
0.7
36.6
33.5
AC/DC 40
0.6
79.69
RPG (Ours)
0.7
37.6
34.4
RPG (Ours) 0.6
79.89
Table Sparse accelera- tion of sparse ResNet-50 on DeepSparse. Unstructured pruming could bring 2 ac- celeration effects CPU high sparsities.
RigL [4]
0.8
26.0
23.7
AC/DC 40
0.8
76.24
AC/DC 40
0.8
34.9
32.1
RPG (Ours) 0.8
77.42
RPG (Ours)
0.8
37.1
33.8
Table 5. Sparse DeïT-S on ImageNet. "Sp." stands for model sparsity. The best sults are bolded.
Table 4. Mask R-CNN pruning COCO val2017. "Sp." stands for model sparsity. Best results are bolded.
2x accerlation CPU. This observation reveals that highly unstructured-sparse networks have promising applicative prospects on edge devices that could not afford power and cost-intensive GPUs, e.g. micro robots, wearable devices, cetera. These devices feature limited memory and power, but high inference speed demands. In this sense, our RPG unstructured pruning method is of great application value.
3.3 Downstream Vision Tasks
We also test our weight pruning method on downstream vision tasks. Mask R-CNN [21 is widely used benchmark for conventional downstream tasks, namely, object detection and instance segmentation. We try to apply our weight pruning method to Mask R-CNN and compare its detection and segmentation perfor-
76
Test Accuracy
74
mance against other pruning baselines. As for the choice of baselines, we found that limited weight pruning works conducted experiments on downstream vision tasks. We choose the following baselines for comparison: RigL (14) is commonly used sparse-to-spars, baseline. AC/DC [40] is good at high-sparsity pruning on ImageNet classification. All methods are ap- plied on Mask R-CNN ResNet-50 FPN variants to measure the mAP for bounding boxes and segmentation masks.
RPG(Ours)
72
WoodFisher
AC/DC
70
PowerProp-ERK
TopKAST-ERK
0.0
0.1
0.2
0.4
FLOPs (x8.2e9)
Figure 3. ImageNet accuracy versus FLOPs on sparse ResNet-50. Our method achieves better Accuracy- FL.OPs trade-off compared with competitive pruning baselines, especially at high sparsities.
For all Mask R-CNN experiments, we follow official training of COCO 1x [21]: pruning and finetuning lasts for 90K iterations in total. The pruning results evaluated on COCO val2017
illustrated in Table 4 Similar to the trend in classification experiments, our RPG method gains an advantage at high sparsities compared with AC/DC [40]. As sparsity increases from 70% to 80%, the gap between AC/DC and RPG widens from 1.0 to nearly 2.0 for both detection and segmentation mAPs. This finding shows that RPG a weight pruning method that could be generalized to various vision tasks: it always works well at high sparsities without the need for significant modifications.
3.4 Vision Transformers
Recent works on vision model architectures focus on transformers 13 models proven particularly effective on large-scale image recognition tasks and are well applied
Transformer architecture

---

to various downstream tasks 58 51, but they are still struggling for industrial applications due to large model size and computation cost. To address these problems, works like SViT-E [6] attempted to apply unstructured pruning on vision transformers.
Though our method is not specifically designed for models with the attention mechanism, explore the effect of our weight pruning method on DeiT-S [52] and compare it with high-sparsity weight pruning baseline [40] and the transformer pruning baseline [6] in T For fair comparison, all pruning experiments follow the setting of SViT-E [6]: the DeiT-S model is pruned for 600 epochs on
ImageNet [9]. All other settings are identical the official training setting of [52], including batchsize. learning rate, etc.
3.5 Ablations
In this section, we inspect the effect of rank loss. The rank-based pruning objective involves an affine parameter that controls the amount of rank loss with respect to the original task loss. When À O, rank loss turned off. Investigating the relations of rank versus À and accuracy versus on ResNet-32 99.5% sparsity as shown Figure D we found rank loss could significantly increase the average rank throughout all layers of the sparse net- work. A substantial increase of accuracy is also observed. But as further increases, the average rank will be satu- rated. Reversely, À further increases, the classification accuracy will decrease. This could be attributed to the property of affine combination in Equation (2.7p. When À large, the pruming objective will pay too much attention maintain weight ranks and neglect the goal of perform- ing the task well. Hence, it necessary to tune À and find the most optimal one.
90.74
63.0
91:11
90:87
62.5
Avg.
90.63
62.0
50.46
Figure 4. Average weight matrix rank of ResNet-32 22 versus affine hyperparameter
Accuracies on CIFAR-10 are marked.
Type
Time
FLOPs
Baseline
Sparsity
Train FLOPs
SVD
16.5min
5.07e15
ResNet-50
(Dense)
3.14c18
RPG90%
1003min
1.34c18
AC/DC
0.9
0.58x
Table SVD overhead compared overall pruning & finetuning cost of RPG on 90% sparse ResNet-50.
PowerProp.
0.49
RPG((urr)
0.9
0.43
Table 7. Training FLOPs comparison with pruning baselines on sparse ResNet-50.
3.6 Overhead Analysis
As introduced in Section 2.4 RPG involves costly SVD calculations. However, we conduct exper- iments and illustrate that SVD accounts for very minimal overhead during pruning in terms of both time and FL.OPs. As shown Table [b] the overall time and FLOPs for SVD calculations only accounts for 2% of the whole RPG pruning cost. We also compare the FLOPs overhead of RPG with other pruning methods. Observing from Table 7 our method is the most cost-effeccive compared with baselines. Above all, the extra overhead brought by rank loss calculations is not concern.
4 Conclusion
This paper proposes the Rank based Pruning (RPG) method. We investigate weight pruning from matrix rank perspective, and yield the observation that higher ranks of sparse weight tensors could yield better performance. Based on this heuristic, the adversarial rank loss is designed as the optimization objective that guides the mask update process during pruning. In this manner, our method prunes weight tensors rank-favorable fashion. Our RPG method is experimented on various settings and outperforms various baselines.

---

5 Limitations
Unstructured pruning has limited acceleration effect on GPU devices. New GPU architectures or GPU sparse acceleration supports are needed to exert the speed potential of our unstructured pruning method on GPUs.
Acknowledgement
This work is supported by National Key R&D Program of China under Grant No. 00.2222220060000 We gratefully acknowledge the support of MindSpore [88, CANN (Compute Architecture for Neural Networks) and Ascend AI Processor used for this research. We also gratefully thank Yuxin Zhang and Xiaolong Ma for their generous help.
References
[1] Guillaume Bellec, David Kappel. Wolfgang Maass, and Robert Legenstein. Deep rewiring: Training very sparse deep networks. In 6th International Conference Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
[2] Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firnrr, Beat Flepp. Prasoon Goyal, Lawrence D. Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao, and Karol Zicba. End to end learning for self-driving cars. CoRR, abs/1664.07316 2016.
[3] Bolun Cai, Xiangmin Xu, Kui Jia, Chunmei Qing, and Dacheng Tao. Dehazenet: An end-to-end system for single image haze removal. IEEE Trans. Image Process., 25(11):5187-55 2016.
[4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision ECCV 2020 16th European Conference, Glasgow. UK, August 23-28, 2020, Proceedings, Part I, volume 12346 of Lecture Notes Computer Science, pages 213-229. Springer, 2020.
[5] Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. In IEEE Conference on Computer Vision and Pottern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 12299-12310. Computer Vision Foundation / IEEE, 2021.
[6] Tianlong Chen, Yu Cheng, Zhe Gan, Lu Yuan, Lei Zhang, and Zhangyang Wang. Chasing sparsity in vision transformers: An end dod-to-end exploration. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pages 19974-19988, 2021.
[7] Bo-Shiuan Chu and Che-Rung Lee. Low-rank tensor decomposition for compression of convolutional neural networks using funnel regularization. CoRR, abs/21 12.03690, 2021.
[8] Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H. Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science. arXiv e-prints, page arXiv:1707.04 047800 July 2017.
[9] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pages 248 -255. IEEE Computer Society, 2009.
[10] Misha Denil, Babak Shakibi, Laurent Dinh, Mare Aurelio Ranzato, and Nando de Freitas. Predicting parameters in deep learning. In Christopher J. C. Burges, Léon Bottou, Zoubin Ghahramani, and Kilian Q Weinberger, editors. Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States, pages 2148-2156, 2013.
[11] Emily L. Denton, Wojeiech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. In Zoubin Ghahramani, Max Welling, Corinna Cortes, Neil D. Lawrence, and Kilian Q Weinberger, editors, Advances in Neural Information Processing Systems 27: Annual Conference on Newral Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pages 269-1227, 2014.
10

---

[12] Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. CoRR, abss/907.04484,, 2019.
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations. ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[14] Utku Evci, Trevor Gale, Jacob Mcnick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 2943-2952. PMLR, 2020.
[15] Jonathan Frankle and Michael Carbin, The lottery ticket hypothesis: Finding sparse, trainahle neural networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[16] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. Linear mode connec- tivity and the lottery ticket hypothesis. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 3259-333-200 3269. PMLR, 2020.
[17] Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. CoRR. abs/1902.09574, 2019.
[18] Priya Goyal, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch. Yangging Jia, and Kaiming He. Accurate, large minibatch SGD: training imagenet hour. CoRR,
2017.
[19] Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
[20] Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connections for efficient neural networks. CoRR, abs/1 506.02226, 2015.
[21] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask R-CNN. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 2980-2988. IEEE Computer Society, 2017.
[22] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference Computer Vision and Pattern Recognition. CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016. pages 770-778. IEEE Computer Society, 2016.
[23] Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 1398-148 IEEE Computer Society, 2017.
[24] Geoffrey E. Hinton, Oriol Vinyals, and Jeffiey Dean. Distilling the knowledge in a neural network. CoRR, abs/1503.02531, 2015. 2015.
[25] Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 2261-2269. IEEE Computer Society, 2017.
[26] Siddhant M. Jayakumar, Razvan Pascanu, Jack W. Rae, Simon Osindero, and Frich Fllsen Top-kast: Top-k always sparse training. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tiee Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020. virtual, 2020.
[27] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. Peter L. Bartlett, Fernando C. Pereira, Christopher C. Burges, Léon Bottou, and Kilian Q Weinberger, editors, Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012, Proceedings of meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States, pages 1106 2012.
11

---

[28] Mark Kurtz, Justin Kopinsky, Rati Gelashvili, Alexander Matveev, John Carr. Michael Goin, William Leiserson, Sage Moore, Bill Nell, Nir Shavit, and Dan Alistarh. Inducing and exploiting activation sparsity for fast inference on deep neural networks. Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 5533 5543, Virtual, 13-18 Jul 2020. PMLR.
[29] Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham M. Kakade, and Ali Farhadi. Soft threshold weight reparameterization for learnable sparsity. In Proceedings of the 37th Internationaa Conference on Machine Learning, ICMI. 2020, 13-18 July 2020, Viruall Event, volume 119 of Proceedings of Machine Learning Rexearch, pages 5544 5555. PMLR, 2020.
[30] Yann LeCun. Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proc. JEEE, 2:278 2324, 1998.
[31] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip H. S. Torr. Snip: single-shot network pruning based on connection sensitivity. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[32] Hao Li, Asim Kadav, Igor Durdanovie, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In 5th International Conference on Learning Reprexentations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedingy. OpenReview.net, 2017.
[33] Mingbao Lin, Rongrong Ji, Yan Wang, Yichen Zhang, Baochang Zhang, Yonghong Tian, and Ling Shao. Hrank: Filter pruning using high-rank feature map. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 1526-1535. Computer Vision Foundation IEEE, 2020.
[34] Shiwei Liu. Tianlong Chen, Xiaohan Chen, Zahra Atashgahi, Lu Yin, Huanyu Kou, Shen, Mykola Pechenizkiy, Zhangyang Wang, and Decebal Constantin Mocanu. Sparse training via boosting pruning plasticity with neuroregeneration. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6 14, 2021, virtual, pages 9908 9922, 2021.
[35] Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 2755-2763. IEEE Computer Society, 2017.
[36] Zhuang Liu, Mingji Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA USA, May 6-9, 2019, OpenRevicw.net, 2019.
[37] Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 5068 5076. IEEE Computer Society, 2017.
[38] MindSpore. Mindspore https: / wwt.:./indsoor....mm cn/
[39] Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterizattion. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 4646-4655. PMLR, 2019.
[40] Alexandra Peste. Eugenia Iofinova. Adrian Vladu, and Dan Alistarh. AC/DC: alternating com- pressed//ecooppressed training of deep neural networks. In Mare' Aurelio Ranzato, Alina Beygelzimer. Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6- 14, 2021, virtual, pages 8557-8570, 2021.
[41] Anh Huy Phan, Konstantin Sobolev, Konstantin Sozykin, Dmitry Ermilov, Julia Gusak, Petr Tichavsky. Valeriy Glukhov, Ivan V. Oseledets, and Andrzej Cichocki. Stable low-rank tensor decomposition for compression of convolutional neural network. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision ECCV 2020 16th European Conference, Glasgow, UK. August 23-28, 2020, Proceedings, Part XXIX, volume 12374 of Lecture Notes in Computer Science, pages 522 539. Springer, 2020.
12

---

[42] Joseph Redmon, Santosh Kumar Divvala, Ross B. Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 779-788, IEEE Computer Society, 2016.
[43] Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruming, In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[44] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fimeet:: Hints for thin deep nets. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[45] Jonathan Schwarz, Siddhant M. Jayakumar, Razvan Pascanu, Peter E. Latham, and Yee whye Teh. Powerpropagation: A sparsity inducing weight reparameterisation. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NewrIPS 2021, December 6- 14, 2021, virtual, pages 28889 28903, 2021.
[46] Baoguang Shi, Xiang Bai, and Cong Yao. An end-to-end trainable neural network for image-based sequence recognition and its application to seene text recognition. IEEE Trans. Pattern Anal. Mach. Intell., 39(11 (9(11):2288228)) 2304, 2017.
[47] Han Shu, Jiahao Wang, Hanting Chen, Lin Li, Yujiu Yang, and Yunhe Wang. Adder attention for vision transformer. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021. virtual, pages 19899-19909, 2021.
[48] Sidak Pal Singh and Dan Alistarh. Woodfisher: Efficient second-order approximation for neural network compression. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
[49] Yang Sui, Miao Yin, Yi Xie, Huy Phan, Saman A. Zonouz, and Bo Yuan. CHIP: channel independence- based pruning for compact neural networks. In Marc' Aurelio Ranzato, Alina Beygelzimer. Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan. Aditors, Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pages 24604-24616, 2021.
Advances in Neural Information Processing
[50] Yaniv Taigman, Ming Yang, Mare' Aurelio Ranzato, and Lior Wolf. Deepface: Closing the gap human-level performance face verification. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2014, Columbus, OH, USA, June 23-28, 2014, pages 1701-1708. IEEE Computer Society, 2014.
[51] Yehui Tang, Yunhe Wang. Yixing Xu, Dacheng Tao, Chunjing Xu, Chao Xu, and Chang Xu. SCOP: scientific control for reliable neural network pruning. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florins Balcan, and Hsuan-Tier Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
[52] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers and distillation through attention. In International Conference on Machine Learning, volume 139, pages 10347-10357, July 2021.
[53] Chaoqi Wang. Guodong Zhang, and Roger B. Grosse. Picking winning tickets before training by preserving gradient flow. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ahaba, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[54] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yer Lo, and Ross Girshick. Detectron2. https /github.
2019.
[55] Wenyuan Zeng and Raquel Urtasun. MLPrune: Multi-layer pruning for automated neural network compression, 2019.
[56] Dongging Zhang, Jiaolong Yang. Dongqiangzi Ye, and Gang Hua. Le-nets: Leamed quantization for highly accurate and compact deep neural networks. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018
13

---

[57] Xiangyu Zhang, Jianhua Zou, Xiang Ming, Kaiming He, and Jian Sun. Efficient and accurate approx- imations of nonlinear convolutional networks. In IEEE Conference Computer Vision and Puttern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pages 1984-1992, IEEE Computer Society, 2015.
[58] Sixiao Zheng. Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo. Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang. Philip H. S. Torr, and Li Zhang. Rethinking semantic segmentation from a sequence-to sequence perspective with transformers. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages -8900 Computer Vision Foundation IEEE, 2021.
[59] Xiao Zhou, Weizhong Zhang, Hang Xu, and Tong Zhang. Effective sparsification of neural networks with global sparsity constraint, In IEEE Conference Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2,21, pages 3599-3608. Computer Vision Foundation [EEE, 2021.
[60] Michael Zhu and Suyog Gupta. To prune, not prune: Exploring the efficacy of pruning for model compression. 6th International Conference on Learning Representations, ICLR 2018, Vancouver BC, Canada, April May 3, 2018, Workshop Track Proceedings. OpenReview.net, 2018
14

---

Appendix
A Proofs of Theorems
A.1 Proof of Theorem
Theorem 3 (The best low-rank approximation). Suppose W is decomposed via SVD and yield W Σ duu,y, where singular values {o,} are sorted in descending order. Given integer T, the best k-rank approximation of W, namely the k-rank matrix that has the smallest l2 distance to W
W W
Σ Couun
This theorem is also the Eckart- Eckart-Young-Ne Yousk-Mirsky Theorem for Frobenius norm.
Though the proof is easily accessible (e.g. from Wikipedia D. we will provide sketch of proof here for reference.
Proof. Denote W = > ii ",uty be the best k-rank approximation of W, (W) be the ith largest singular value of W. Then the low-rank approximation error could be reduced as follows:
[ =
(A.1)
k=1+1
i=++1
F
Given W W + w" according the triangle inequality of spectral norm,
σ1 (W)= = ow(W) (W().
(A.2)
Then for two arbitrary ranks i.j 2 1, we have:
o, (W') to (W")
or (W' - WW;-1 to w
20 0i W -W;- - W3-
(A.3)
2 01 (W Wi+i-2)
+1+)-1(W).
Assume there another k-rank approximation X, Then according to the above formula, for arbitrary
i 121 1,
o - X) = ( - x) + ox+i 2 (W)
(A.4)
Hence,
x 2 Σ W
(W - x)²
Σ of
(A.5)
1=1
k=
which means W is the best k-rank approximation.
A.2 Proof of Theorem 2
Theorem 2 states the effectiveness of rank loss. Before the theorem, recall notations that previously is the normalized weight matrix W; U. Σ, V are matrices reached from the TWT
defined: W == W
SVD of W. where U {ui, u2....) and V {u1, ...) are orthonormal bases; E is diagonal matrix where singular values (01, 02, are sorted in descending order on the diagonal; operator Trun (UEVT)= Σ auu. stands for k-rank truncated SVD, or the k-rank best approximation of W according to Theorem
https://en wttp://www.......ooo.///ooooo.a. wppprooiiaaaaaooiiatitioooooo
15

---

Theorem 4 (Effectiveness of the adversarial rank loss), Given the adversarial rank loss
Frank IW Trun (UEVT) I
(A.6)
If we optimize W in rank loss via gradient descent, the rank of W will increase.
Proof. In gradient descent, the update from weight W W based rank loss Crank could be described as:
Orronk aW
W' = W
(A.7)
We first simplify rank loss. Since U {u,, u2,...) and V = (w), 02,-.. orthonormal bases, could easily rewrite rank loss Lrank as the squared sum of small singular values:
W
k Σ
Drank = ||
of
1|?
|
i=1
--IUEV U. 1:00] V 11
(A.8)
= -- Ep=0l1e = Σ oZ.
1=
The form Equation (A.8) allows us apply chain rule to calculate the gradient of the normalized weight matrix &rank aw
dCrank aw
Orrank do; 0o; Ow
20 20;u,,.
(A.9)
i=++1
Again, the chain rule is applied for the derivative of the weight matrix. For clarity, we show the gradient expression for single weight parameter W [m, n) (the weight value at position (m, n) in the reshaped weight matrix W):
Ohrank OW OW(m,
@Drank Ww
OW OW
=tr
ÖW[m, [m, ml
(E 20 ) [m,
||W||
(A.10)
W I,, n) Σ O EE 20,u,7?
W
Based on the gradient, one step of optimization under learning rate a could be expressed in neat matrix multiplication format, decomposed by orthonormal bases U = {uy u2,...) and (U1,22,...)
Lemma (Weight optimization based on rank loss). Denote Σ
W o (Ei 20,uxu)) = e,
which is scalar constant within each step, step of weight W optimization under learning rate
16

---

> O and rank loss (as defined in Equation (2.6
could be expressed as!
W =W rCank
oW
Ei= 20,
cW
=W
|W]|F
I
-UEVT
2
U ('(4+150)
CY
UEV
(A.11)
|
|F
|
-U (
27 [ E(p
Σ
VT.
[
From the formula of the optimized weight W', we can reach the following conclusions on the optimized weight W firstly, W promises the same set of orthonormal bases U, V after SVD; secondly, comparing small singular values against others, all singular values are penalized by the same amount ety but the small singular values (ranking from & + 1 to r) are awarded with j
increments [| Regardless of swapped singular values due to magnitude change (because of small learning rate y). small singular values will make up more proportion in all the singular values after one step of update. Recall the definition for 6-rank, given fixed sum of squared singular values after normalization, the rank of W will increase.
2-
B
Method Details
B.1 Details on Gradient-Grow.
We explain in details about the procedure of gradient grow evolved from RigL [14. At each gradient grow step, first we calculate the Rank-based Pruning (RPG) objective Then back-propagate L for gradients of network weights. Finally, gradients of pruned network weights are sorted in magnitudes; weights with large gradients are re-activated.
The number of weights to be re-activated are determined by the number remaining weights. The whole pruning framework is detailed in Algorithm [ Grow fraction is function of training iterations that gradually decays for stability of training. Cosine Annealing is used for a, following 14
B.2 Selection of Approximation Rank
Factor k of the truncation operator Trun controls the rank of the low-rank approximation in this adversarial process. However, controlling for each layer is hardly practical, because layers are large in quantity and vary in shapes. We leverage the concept of 8-rank in Definition 1, and tend to control the approximation eиror (also the negative rank loss Crank for a layer) rather than control k. We set a constant å between (0, 1) as the target approximation error between the normalized weight matrix W and its low rank counterpart. Then we find the best that has closest low-rank approximation error as 5 for each layer. Mathematically, this selection process could be written as:
arg min Lrank 8).
(B.1)
C Experiment Details
C.1 CIFAR Experiments
In the CIFAR experiments, we train VGG-19 and ResNer-32 for 300 epoch, which is identical to Zhou al. [59]. We use the SGD optimizer with momentum 0.9, batchsize 128, learning rate 0.1, and weight decay 0.005.
17

---

Algorithm 1: Rank-based Pruning (RPG)
Input :A dense model W with layers; Target density function of iteration) d: Grow
fraction (a function of iteration) o; Mask update interval AT; Total training iteration
T T prume + T finetune
Output: : Sparse Model W @ M
// Initialize a dense mask
2 M + 1;;
// Stage prune and grow; for t t- 1 to T prume do
Forward propagation for minibatch loss Ltaski
if /%A7 then
// Update mask M;
Calculate and sum layerwise rank loss;
Keep top de proportion of weights in W globally to get layerwise density di:
10
for i 4 to n do
Prune to density (1 ory (g)d? based WW;
12
Grow density di again based on VC]:
end for
14
end if
else 15
Train sparse net with task loss;
17
end
end for
19 // Stage 2: keep masks fixed and finetune;
20 for I 4- Tprune + to T do
Keep mask M static, and train weight W till converge; end for
return Weights of the Pruned Model W @ M
C.2 ResNet-50 on ImageNet
We mainly follow Goyal et 18 for ImageNet experiments. ImageNet experiments are run on 8 NVIDIA Tesla V100s. Our RPG method is applied on ResNet-50 for 100 epoch, and compared with 100-epoch baselines various sparsities. We use the SGD optimizer with momentum 0.9, 1024 batchsize, learning rate of 0.4, 0.0001 weight decay, 0.1 label smoothing. 5 epochs of learning rate warmup. We choose 1024 batchsize instead the commonly-uss 4096 batchsize [14 45) due to GPU memory constraints. We did not use augmentation tricks like mixup or cutmix. Standard pretrained model from torchvision is used ImageNet setting for fair comparison with top baselines. is kept at 0.3; AT is set to 100; Trrone is set to 90 epochs.
C.3 Downstream Vision Tasks
In Mask R-CNN experiments, all methods are implemented by us on the identical settings for fair comparison. We follow the training setting of the original Mask R-CNN 21 on Detectron-2 (54 The three methods are applied for 96000 iterations. Notably, certain components in Mask R-CNN ResNet-50 FPN are loaded from pretrained models and left untrained. These components are omitted from pruning and sparsity calculations for RPG and baseline experiments.
C.4 Vision Transformers
In vision transformers, we mainly follow the official setting of DeiT 522 but we extend training epochs from 300 to 600 for fair comparison with SVIT-E (6.. All other training settings are identical to the official training setting of DeiT [52].
18

---

C.5
Replication of AC/DC
AC/DC achieves State-of-the-Are Art performance at high sparsities. Therefore, we hope compare our method extensively with AC/DC on various settings. Accordingly, the schedule of AC/DC need slight modifications based on the original setting. According the original work [40], dense and sparse status are alternated every 5 epochs. We scale the period according to the ratio of the training epochs versus the standard training epochs in the paper [40]. For example, the AC/DC method applied for 300 epochs, since the original ImageNet AC/DC setting has only 100 epochs, we lengthen the period of AC/DC from the original 5 epochs 15 epochs. For warming-uy up and finetuning periods, the similar scaling is performed.
D Additional Experiments
We also tried RPG pruning without loading pretrained models on ResNet-50 ImageNet experiments. Training settings are kept the same with other ResNet-50 pruning experiments. The results provided in Table g The proposed RPG method could surpass all baselines even without loading pretrained models.
Algorithm
Sparsity
FLOPs
Accuracy
ResNet-50
1x
76.80
RPG from scratch
0.9
0.155x
75.35
RPG from scratch
0.95
0.093
73.62
Table 8. RPG-Sparsified ResNet-50 from scratch ImageNct.
19


Reviews:
[{"review_id": "03SzanRZwC", "reviewer": "Reviewer_yERC", "summary": "This paper proposes a novel unstructured pruning method, trying to maximize the matrix rank while trying to remove as many model weights as possible. The paper first demonstrates the phenomenon that unstructured pruning may degrades to structured pruning at large sparsity ratios, which is closely related to the fact that the pruned weight matrices become low-rank matrices after many weights are set to zero. Thus, the objective of the proposed method is trying to on the one hand, minimize the task-related loss, and on the other, maximize the rank of the pruned weight matrices, which forms a min-max problem. This min-max problem is then integrated into model pruning via a matrix rank-boosting regularization term. With the gradual pruning framework, the proposed method (RPG) is examined empirically on CIFAR-10, ImageNet, COCO datasets with CNNs and ViTs model architectures. The results show the effectiveness of the RPG.", "questions": "Below is a summary of my comments in the section \"Weaknesses\".\n1. What is the relationship between the rank of the pruned weight matrix? Will the unstructured pruning with higher rank impairs the further acceleration on hardwares?\n2. How are the SVD-reconstructed terms dealt with during back-propagation?\n3. What are the formulation of $\\mathcal{L}(\\text{task})$?\n4. How are the weights with multi-layers processed to compute Eq. (2.6)?\n5. Why are the baselines not consistent within one table/across different tables on the same tasks?\n6. What how does the training efficiency of different methods look like?", "limitations": "I do not have additional comments on the limitation of this work. Please refer to the \"Weaknesses\" and \"Questions\" sections.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper studies the unstructured pruning from the perspective of rank maintenance, which is very novel. The authors made very informative and helpful illustrations to help the readers understand this paper without difficulty. Therefore, the presentation is also very good. Extensive experiments were conducted on different tasks, and the results look good. In summary, this work is a very good attempt to connect model pruning with weight ranks in a novel perspective.", "weaknesses": "I listed some weakness of this work from different perspectives. I will consider to raise my score if they are addressed properly. \n1. [Motivations] The motivations of this paper by \"removing the structuring patterns in the unstructured pruning\" do not seem very direct to me. In general, currently the results of the unstructured pruning can not be directly used for hardware acceleration. Therefore, one important research direction is to generate structured mask from the unstructured pruning results, such as [ICML22] Coarsening the Granularity: Towards Structurally Sparse Lottery Tickets. Therefore, I wonder if the motivation of this paper will make the unstructured pruning less adaptable to hardwares?\n2. [Presentations] I would suggest the authors to remove some discussions on SVD, which are very basic knowledge in linear algebra. In contrast, more important information in the Appendix can be brought back to the main paper.\n3. [Method] It is a bit vague to me, how the gradients are dealt with when Eq. (2.6) is treated as the regularization term. Is $\\text{Trunc}(U\\Sigma V^T)$ treated as a constant value during back-propagation? Also, I suggest the authors explicitly write down the $\\mathcal{L}(\\text{task})$ as well, which shows how the sparsity of the model weights are imposed.\n4. [Method] In Eq. (2.6), the model weights are denoted by one variable $\\mathbf{W}$. However, as we know the model weights may contain many layers and many types (convolutional kernels, fully connected layers, etc.) Do the authors sum them up in implementation using the same coefficient? Or there are different weights assigned to different layers. It is not clear in the paper regarding this point.\n5. [Experiments] The baselines used in Tab. 1 and Tab. 2 are not consistent. Are their any specific reasons? The same issue is also spotted in Tab. 5, where in the last row block the method \"SViTE\" is missing, but the authors stated \"For fair comparison, all pruning experiments follow the setting of SViTE.\" (Line 351).\n6. [Experiments] It would help the readers to better compare the results if the authors can include the computational (training) time of different methods.\n7. [Minor] There are unnecessary margins under Tab. 2 and Fig. 3. Please consider to remove them and make the layout of the figure better. \n8. [Minor] Line \"is illustrated in\" -> \"**are** illustrated in\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer yERC,\n\nThank you very much for your suggestions. Here are the answers to the questions you raised:\n\nQ1. *RPG will make the unstructured pruning less adaptable to hardwares.* \n\nA1: Sorry for ambiguities in the paper. \"Structured pattern\" does not necessarily mean the acceleration-friendly structural sparsity (which is only a special case of structured pattern); in the context of RPG, \"structured pattern\" is reflected by the low-rank characteristics of weights. The RPG method mainly focuses on improving weight ranks instead of the actual removal of structural sparsity. We conducted speedtest experiments and compared RPG with the competitive pruning baseline of AC/DC [Peste et al.]. Results show that despite higher ranks, RPG won't affect the hardware adaptability of unstructured-sparse networks.\n\n| ResNet-50 | Acc.  | Speedup | Rank  |\n| --------- | ----- | ------- | ----- |\n| Dense     | 76.80 | 1.00x | 263.5 |\n| AC/DC 95% | 73.14 | 1.66x | 234.2 |\n| RPG 95%   | 73.89 | 1.99x | 262.2 |\n\n\n\nQ2. *How are the SVD-reconstructed terms dealt with during back-propagation?* \n\nA2: The term $Trunc(U\\Sigma V)$ is treated as a constant value. The SVD-reconstructed terms are detached during loss calculation.\n\nQ3. *What is the formulation of $\\mathcal{L}(task)$?*\n\nA3: $\\mathcal{L}(task)$ is exactly the training loss for the original dense network. For instance, $\\mathcal{L}(task)$ is cross entropy loss for ResNet-50/DeiT ImageNet classification.\n\nQ4. *How are the weights with multi-layers processed to compute Eq. (2.6)?* \n\nA4: The losses for each layer are summed-up together.\n\nQ5. *Inconsistency of baselines in Tab. 1 and Tab 2; missing SViTE in Tab. 5.* \n\nA5: The inconsistancy between Table 1 and 2 is due to the lack of official reports or code. Sparse ResNet-50 for ImageNet classification is a commonly-used weight pruning benchmark (Table 2), but only a handful of papers report CIFAR results (Table 1). Experiment settings (e.g. models, training epochs) also varies among papers with CIFAR. Hence, we have to follow the setting of one reliable and competitive baseline, namely, ProbMask [Zhou et al.] for CIFAR-10 experiments to guarantee fair comparison. In spite of this, we still tried to include competitive pruning baselines, e.g. we re-implemented AC/DC [Peste et al.] on CIFAR. We attempted to re-implement some other strong baselines but encountered problems (e.g. incomplete open-sourcing; closed-source codebase though requested via emails; failure of replication).\n\nThe method SViTE is missing in the last row block, because the SViTE paper [Chen et al.] only reports values on relatively low sparsities (sparsity 50/60). We compared our RPG method to a more powerful baseline AC/DC to demonstrate the strong capability of our method under a high sparsity regime. For 80\\% sparsity experiments, we still follow the same setting as SViTE except for the pruning rate.\n\nQ6. *Authors should include the computational (training) time of different methods.*\n\nA6: We provide the running time of some pruning algorithms in the table below: \n\n| Baseline | Acc.  | Time |\n| --------- | ----- | ------- |\n| Dense     | 76.80 | 821 min |\n| AC/DC 90% | 75.03 | 861 min |\n| RPG 90% (Ours)   | 75.63 | 866 min |\n\nNotably, we remark that the above statistics could only give a rough estimation, because the actual running time depends on many factors unrelated to the algorithm itself, including how DataParallel is implemented, what supporting packages the codebase is using, et cetera. Statistics show that the RPG method is not significantly time-costly.\n\n\nAdditionally, thank you for your advice on layouts and typos. We will amend them in later revisions.\n\nSincerely,\n\nAuthors\n\nReferences: \n\n[Chen et al.] Chasing sparsity in vision transformers: An end-to-end exploration. NeurIPS 2021.\n\n[Peste et al.] AC/DC: alternating compressed/decompressed training of deep neural networks. NeurIPS 2021.\n\n[Zhou et al.] Effective sparsification of neural networks with global sparsity constraint. CVPR 2021.\n\n"}, {"review_id": "AeupWVOfMl", "reviewer": "Reviewer_jsvG", "summary": "This paper proposes a new weight pruning method for compressing Convolutional Neural Networks (CNNs) called Rank-based PruninG (RPG). The RPG method consists of two steps: first, the low-rank approximation error for the weight matrices is minimized using singular value decomposition, and second, the weight matrices are pushed away from their low-rank approximation to maximize their distance. The authors demonstrate that the RPG method outperforms other state-of-the-art methods in terms of accuracy and compression rate on various datasets and tasks.", "questions": "Please refer to the weaknesses.", "limitations": "The authors have adequately addressed the limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: Pruning weights while maintaining the ranks of sparse weights in an adversarial manner is original and has not been proposed before, which is different from other pruning methods that focus on removing individual weights or neurons.\n\nQuality: The experimental results show that the RPG method achieves higher accuracy and compression rate than the previous baselines. The authors also provide insights into the mechanism of the RPG method and its impact on the network structure, which enhances the quality of the paper.\n\nClarity: The paper is well-written and easy to understand, which enhances its clarity. The authors provide clear explanations of the proposed method and its implementation. They also provide detailed experimental results and analysis.\n\nSignificance: When the pruning rate is high, traditional pruning methods can lead to a structured pattern in the remaining weights, which limits their performance. The proposed rank-based pruning method maintains the ranks of sparse weights in an adversarial manner, which ensures that the pruned network retains its structure and performance even at high pruning rates.", "weaknesses": "1) The adversarial optimization involves a min-max problem that requires additional computation and optimization steps, which can increase the training time and complexity. Similarly, SVD is a computationally expensive operation that requires matrix factorization and singular value decomposition, which can also increase the training cost. Moreover, the proposed method requires the computation of low-rank approximations and the search for the best k-rank approximation, which can further increase the training cost. These additional computations and operations can make the proposed method less practical for large-scale networks or real-time applications.\n\n\n2) Singular value decomposition (SVD) is used in the proposed method to compute the rank and low-rank approximations of weight matrices. In contrast, Canonical Polyadic Decomposition (CPD) and Tucker Decomposition are other methods for decomposing tensors into lower-rank components. It is better to explain why choosing SVD to estimate the rank of weights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer jsvG,\n\nThank you very much for your review. Here are our responses:\n\nQ1. *Additional computations and operations can make the proposed method less practical for large-scale networks or real-time applications.*\n\nA1: In fact, these extra overheads only accounts for a small proportion of training cost. Firstly, the extra cost is amortized because the procedures are carried out once every one hundred iterations; Secondly, weight SVD, which is much more costly than adverserial loss calculation and best k-rank searching, only accounts <<1\\% of the whole pruning cost both in terms of time and FLOPs (according to Sec. 3.6). In a nutshell, additional computations are minimal and won't impact the application value of RPG on large models and other applications. \n\n\nQ2. *Why choosing SVD instead of CPD or Tucker Decomposition?*\n\nA2: Unlike Singular Value Decomposition of matrices, Canonical Polyadic Decomposition and Tucker Decomposition are decomposition methods of high-order tensors (in fact, CPD could be viewed as a high-order extension of SVD). Matrix-form (2-dimensional) weights (rather than high-order tensors) are the most general and widely-applied form in all sorts of neural networks. Hence, we adopt SVD as the decomposition method.\n\nSincerely,\n\nAuthors"}, {"review_id": "tdBccubHmQ", "reviewer": "Reviewer_P9XJ", "summary": "This work propose a novel objective for performing element-wise pruning of DNN model. The work identifies the loss of weight rank as the key factor influencing the performance of DNN when pruned to high sparsity. This issue is tackled by including a rank loss in the pruning criteria, so that weight elements contributing to the weight matrix rank are preserved. Experiment results show the proposed method outperform previous work under high sparsity.", "questions": "See weakness", "limitations": "The limitation is adequately discussed. No potential negative social impact is observed.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The motivation of preserving high rank in element-wise pruning is novel, and it motivates the proposed method well\n2. The proposed method is well formulated and is technically sound\n3. Adequate ablation study and extensive experiments are conducted, showing promising results", "weaknesses": "One major concern of this paper is whether aiming for higher element-wise sparsity is useful for the deployment of efficient DNN model. As mentioned in the limitation, element-wise sparsity is not well supported on GPU. Even on CPU, as shown in Tab.3 only 2x speedup can be achieved with 95% sparsity, at the cost of 3% accuracy drop. While a 50% structural sparsity can lead to 4x FLOPs reduction, potentially 3x speedup, with less accuracy drop. This would indicate that aiming towards a high-rank element-wise sparsity is not as useful as directly having structural sparsity. I would suggest the author to provide a comparison of accuracy-speed tradeoff of the proposed method and previous structural pruning results, to show the importance of the proposed method.\n\nMinor issue: It would be better to have a pseudo code for the pruning procedure performed in Sec. 2.5", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer P9XJ,\n\nThank you very much for your review. Here are our responses:\n\nQ1. *The author should provide a comparison of accuracy-speed tradeoff of the proposed method and previous structural pruning results.*\n\nA1: Thanks for your suggestion. Here we attach a table comparing the CPU speedup of one of the latest structural baseline TPP [Wang et al.] with RPG. The table shows that the gap between the two methods are narrow. \n\n| Baseline | Accuracy  | Speed |\n| --------- | ----- | ----- |\n| RPG | 76.58 | 1.08x  |\n| TPP | 76.44 | 1.13x  |\n| RPG | 75.63 | 1.56x  |\n| TPP | 75.60 | 1.45x  |\n| TPP | 74.51 | 1.86x  |\n| RPG | 73.89 | 1.99x  |\n\n\nBesides, while we admit structural baselines could achieve good accerlation effects on GPU (while unstructured sparsity has little effect), unstructured-sparse networks are more compact in terms of size. The size advantage of unstructured sparsity allows applications in storage or bandwidth-limited scenarios.\n\nQ2. *It would be better to have a pseudo code for the pruning procedure performed in Sec. 2.5.*\n\nA2: Thanks for your advice. The pseudo code is included in the Appendix B.1, pp. 4 due to page limits. We will consider moving it to a prominent place in later revision.\n\nSincerely,\n\nAuthors\n\nReferences:\n\n[Wang et al.] Trainability Preserving Neural Pruning. ICLR 2023."}, {"review_id": "IE7H8O98Fl", "reviewer": "Reviewer_GHQR", "summary": "This paper proposes a novel Rank-based PruninG (RPG) method for network pruning that maintains the ranks of sparse weights in an adversarial manner, leading to high-rank topology and improved performance. The proposed method is evaluated on various datasets and tasks, including image classification, object detection, and semantic segmentation, and compared to state-of-the-art pruning methods. The experimental results show that the proposed RPG method outperforms the existing methods in terms of accuracy and efficiency. The paper also provides insights into the importance of rank preservation in network pruning and the potential of adversarial training for improving the performance of pruned networks.", "questions": "This paper mentions that the proposed method is based on the rank of the weights, but it would be helpful to have more details on how exactly the rank is determined and how it affects the pruning process. Additionally, are there any limitations or potential drawbacks to using rank-based pruning compared to other pruning methods that use different criteria for selecting weights to prune?", "limitations": "Yes.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "+ This paper is well-written and organized, with clear explanations of the proposed method and experimental results. The authors provide detailed insights into the importance of rank preservation in network pruning and the potential of adversarial training for improving the performance of pruned networks.\n\n+ This paper proposes a novel Rank-based PruninG (RPG) method that maintains the ranks of sparse weights in an adversarial manner, leading to high-rank topology and improved performance. This approach is original and creative, as it combines the ideas of rank preservation and adversarial training to address the limitations of existing pruning methods.\n\n+ This paper provides a comprehensive evaluation of the proposed RPG method on various datasets and tasks, including image classification, object detection, and semantic segmentation. The experimental results show that the proposed method outperforms the existing methods in terms of accuracy and efficiency, which demonstrates the quality and effectiveness of the proposed approach.\n\nOverall, this paper has a high-quality and significant contribution to the field of network pruning, with original ideas, rigorous evaluation, clear explanations, and practical implications.\n", "weaknesses": "- One potential weakness of the proposed Rank-based Pruning (RPG) method is that it may only achieve better performance than existing methods when the sparsity rate is high. As mentioned in the paper, the RPG method outperforms existing methods such as WoodFisher, PowerPropagation, and AC/DC at sparsity rates of 90%, 95%, and 98%. However, at lower sparsity rates such as 80%, the RPG method is slightly lower than WoodFisher in terms of ImageNet accuracy. This suggests that the RPG method may not be as effective at lower sparsity rates, and may require higher sparsity rates to achieve better performance than existing methods. This could be a limitation for some applications where lower sparsity rates are preferred due to memory or computational constraints.\n\n- The proposed method can only be applied on weight pruning, which is slower than filter pruning when their pruning rates are same.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer GHQR,\n\nThank you very much for your review. Here are our reponses to the weaknesses and questions you raised:\n1. *The RPG method is not performant at low sparsities, impairing low-sparsity applications*: We admit the limitation of the RPG pruning methods under low-sparsity regimes, because the rank-collapse effect is not manifested on low-sparsity networks. But we hold that highly-sparse networks are more valuable in terms of application. As shown in the ResNet-50 example (cf. Table 3), intermediate-sparse models could achieve little CPU acceleration effect; while the speedup is significant for models at high sparsities. Hence, we could find a better accuracy-speed trade-off on highly-sparse networks, and the application value of the proposed RPG method is demonstrated.\n2. *About how the rank is determined*: Due to page limits, we put details of rank determination in appendix B.2. Considering weights of different layers vary significantly in shapes, we designed an automatic selection mechanism for ranks.\n3. *Any limitations or potential drawbacks to using rank-based pruning*: Our method is targeted at the \"rank collapse\" effect that only occurs on highly-sparse networks. It has limited effect on models with relatively low sparsity, because the \"rank collapse\" effect is not so serve on low sparsity networks.\n\n\nSincerely,\n\nAuthors"}]

Author rebuttal:

Meta-review: This paper proposes a novel Rank-based PruninG (RPG) method for network pruning that maintains the ranks of sparse weights in an adversarial manner, leading to high-rank topology and improved performance. The proposed method is evaluated on various datasets and tasks, and compared to state-of-the-art methods. The experimental results show that the RPG method outperforms other pruning methods in terms of accuracy and efficiency. The main strengths of this paper are its novel combination of rank preservation and adversarial training ideas to address the limitations of existing pruning methods, the thorough experimental evaluation, and its clear presentation of results and analysis.

Reviewers initially raised several legitimate concerns including the hardware practical acceleration, the baseline inconsistency, and missing training time. The authors actively addressed the raised concerns during rebuttal by supplying many additional experiments and managed to convince even the most negative reviewer to turn positive. Eventually, based on the all-positive reviewer consensus, AC recommends acceptance.
Decision: Accept (poster)