SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yuchuan Tian $^{1}$ , Hanting Chen $^{2}$ , Tianyu Guo $^{2}$ , Chao Xu $^{1}$ , Yunhe Wang $^{2*}$
$^{1}$  National Key Lab of General AI, School of Intelligence Science and Technology, Peking University.
$^{2}$  Huawei Noah's Ark Lab.
tianyc@stu.pku.edu.cn, {chenhanting,tianyu.guo,yunhe.wang}@huawei.com,
xuchao@cis.pku.edu.cn
Convolutional Neural Networks (CNNs) are hard to deploy on edge devices due to its high computation and storage complexities. As a common practice for model compression, network pruning consists of two major categories: unstructured and structured pruning, where unstructured pruning constantly performs better. However, unstructured pruning presents a structured pattern at high pruning rates, which limits its performance. To this end, we propose a Rank-based PruninG (RPG) method to maintain the ranks of sparse weights in an adversarial manner. In each step, we minimize the low-rank approximation error for the weight matrices using singular value decomposition, and maximize their distance by pushing the weight matrices away from its low rank approximation. This rank-based optimization objective guides sparse weights towards a high-rank topology. The proposed method is conducted in a gradual pruning fashion to stabilize the change of rank during training. Experimental results on various datasets and different tasks demonstrate the effectiveness of our algorithm in high sparsity. The proposed RPG outperforms the state-of-the-art performance by  $1.13\%$  top-1 accuracy on ImageNet in ResNet-50 with  $98\%$  sparsity. The codes are available at https://github.com/huawei-noah/Efficient-Computing/tree/master/Pruning/RPG and https://citee.com/mindspore/models/tree/master/research/cv/RPG.
As Convolutional Neural Networks (CNNs) are adapted to various tasks at better performance, their sizes also explode accordingly. From shallow CNNs like LeNet [30], larger CNNs like AlexNet [27], to deeper modern CNNs like ResNets [22] and DenseNets [25], CNNs are growing larger for more complex tasks and representations, including large-scale image classification and downstream tasks like object detection [42], segmentation [21], etc. The evolution of CNN gives rise to various real-world applications, such as autonomous driving [2], camera image processing [3], optical character recognition [46], and facial recognition [50]. However, it is difficult to deploy large CNN models on mobile devices since they require heavy storage and computation. For example, deploying a ResNet-50 [22] model costs 8.2G FLOPs for processing a single image with  $224 \times 224$  size, which is unaffordable for edge devices with limited computing power such as cellphones and drones.
In order to compress heavy deep models, various methodologies have been proposed, including Weight quantization [19, 56], knowledge distillation [24, 44], and network pruning. Network pruning prunes the redundant weights in convolutional neural networks to shrink models. Weight (or unstructured) pruning [20] and filter (or structured) pruning [32] are two main pathways to prune CNNs. Weight
pruning sparsifies dense kernel weight tensors in convolutional layers in an unstructured manner including iterative pruning [20], gradual pruning [60, 17], and iterative rewinding [15, 16, 43]. Some other works [31, 53, 48] propose gradient or hessian based weight saliencies that proved effective in certain scenarios. Filter pruning [37, 23, 35, 36, 51] prunes filters in convolutional layers as a whole, reducing the redundant width of network layers.
Although structured pruning algorithms can be well supported by existing hardwares and bring large runtime acceleration benefits, their performance is much lower than that of unstructured pruning. For example, SOTA unstructured pruning methods could achieve  $80\%$  sparsity on ResNet-50 with little performance drop [48, 45] while structured pruning could only reach less than  $50\%$  [49], since filter pruning is a subset of weight pruning by further imposing structural constraints. However, under circumstances of high sparsities, we observe that unstructured pruning partially degrades to structured pruning. When weights are with a large proportion of zeros, it is highly likely that a structured pattern appears, where a whole channel or filter is almost completely pruned. Therefore, existing weight pruning methods usually meet dramatic performance decay at high sparsities.
Inspired by the comparison of the two pruning categories, we propose to reduce structural patterns in weight pruning. Structured pruning is factually a reduction of weight rank in deep Convnets. Thus, rank could be adopted as a metric for evaluating the "structuredness" of unstructured sparse weights: a sparse weight is considered highly structured if it possesses low rank. To keep unstructured pruning from being too structured, we hope to maintain weight ranks under high sparsities in pruning. Based on the goal of rank improvement, we propose an adversarial Rank-based PruninG (RPG) approach for unstructured pruning. First, we find a low-rank approximation of the weight by minimizing the approximation error. The best low-rank approximation is found via singular value decomposition. Second, to enhance weight ranks, we maximize the distance between the weight and its low-rank counterpart to increase weight rank. This adversarial rank-based optimization objective guides sparse weights towards a high-
rank topology. The proposed method is conducted in a gradual pruning fashion to stabilize the change of rank during training. The advantage of the proposed RPG method is evaluated through extensive experiments on image classification and downstream tasks, and Figure 1 demonstrates that our method gains a matrix rank advantage compared to baselines.
Figure 1. Average weight matrix rank of ResNet-32 [22] pruning baselines versus sparsity. Our rank-based method is effective in maintaining weight ranks at high sparsities compared with baselines.
Figure 2. An illustrative diagram of our Rank-based Pruning (RPG) method.
2 Maintaining Rank via Adversarial Pruning
In conventional terms of supervised neural network learning, given a target loss function  $\mathcal{L}$ , neural network weight  $W$ , and input output pairs  $X = \{x_{i}\}_{i = 1\dots n}, Y = \{y_{i}\}_{i = 1\dots n}$ , weight training of a
neural network  $W$  is formulated as:
$$
\underset {W} {\arg \min } \mathcal {L} (Y, W X), \tag {2.1}
$$
Weight pruning limits the total number of non-zero weights in weight  $W$ ; or mathematically, weight pruning imposes a  $l_0$ -norm constraint on neural network learning. Given sparsity budget  $c$ , the constraint is described as:
$$
\| W \| _ {0} \leq c, \tag {2.2}
$$
A common practice is to reparameterize weight  $W$  with the Hadamard elementwise product of a weight tensor  $W$  and a binary mask  $M$ . The binary mask  $M$  has the same shape as  $W$ , and each element in  $M$  represents whether its corresponding parameter in  $W$  is pruned. After reparametrization, the weight pruning problem is then formulated as:
$$
\underset {W \odot M} {\arg \min } \mathcal {L} (Y, (W \odot M) X) \text {s . t .} \| M \| _ {0} \leq c. \tag {2.3}
$$
In Equation (2.3),  $\odot$  is the Hadamard elementwise product of matrices.
At high sparsities in unstructured pruning, the rank of sparse networks could decrease substantially. In the following sections, we will demonstrate the problem and propose a solution to maintain sparse weight ranks.
2.2 Analyzing Weight Pruning in High Sparsity
Unstructured and structured pruning are two major pruning methodologies. In unstructured pruning practices, weight tensors of CNNs are pruned in a fine-grained manner: each and every solitary weight parameters could be turned off (i.e. set to zero) within the network, but the whole weight tensor structure is left unchanged. In contrast, structured pruning focuses on the pruning of filters: filters are cut-off as the smallest prunable unit in the pruning process. Comparing the two pruning paradigms under the same sparsity budget, Zhu and Gupta [60] illustrate that unstructured pruning performs much better than structured pruning under the same pruning budget.
This phenomenon could be explained from the perspective of matrix ranks. In fact, structured pruning is a direct rank reduce imposed on weight matrices, which means filter pruning is basically weight pruning with low rank. The rank of a matrix represents the upper bound of the amount of information contained in the matrix. A powerful network should be rich in information, and we hope features of the sparse network could have high ranks. Feature ranks is closely related to ranks of sparse weight matrices because of the formula below that describes the relationship of ranks in matrix multiplication:
$$
\operatorname {R a n k} (Y) = \operatorname {R a n k} (W X) \leq \min  (\operatorname {R a n k} (W), \operatorname {R a n k} (X)). \tag {2.4}
$$
According to Equation (2.4), when filter pruning is applied on weight  $W$  that directly impacts its rank, the rank of the output feature will also degrade, causing dramatic loss in information richness. On the other hand, unstructured pruning is free from the structural constraint of filter pruning, and thus maintain more amount of information.
However, under circumstances of high sparsities, we observe that unstructured pruning partially degrades to structured pruning. When weights are filled with a large proportion of zeros, it is very probably that some filters or channels are almost entirely turned-off: "quasi-structured" sparse weight pattern is then formed. A baseline evaluation of matrix ranks in Figure 1 illustrates this concern. Therefore, existing weight pruning methods usually meet dramatic performance decay at high sparsities. Inspired by the properties of the two categories of pruning, we propose to reduce the structured pattern in unstructured pruning, and therefore to maintain weight ranks under high sparsities.
2.3 Low Rank Approximation through SVD
Now that weight ranks are important in weight pruning, we need a practical way to compute ranks in the context of deep neural networks. Previous deep learning works on ranks apply matrix rank theories to CNN low-rank tensor decomposition. In these works, low-rank approximations are proposed to fit
weight tensors. Denil et al. [10] decomposes  $W \in \mathbb{R}^{m \times n}$  into the multiplication of  $U \in \mathbb{R}^{m \times r}$  and  $V \in \mathbb{R}^{r \times n}$  where  $r$  is much smaller than  $m$  and  $n$ .  $UV$  provides a low-rank approximation (rank bounded by  $r$ ) of weight  $W$ . Denton et al. [11] uses the sum of  $k$  rank-one approximations to provide the  $k$ -rank approximation of a feature tensor. Zhang et al. [57] multiplies a low-rank matrix  $M$  to weight matrix  $W$ , and solves the low-rank  $M$  with Singular Value Decomposition (SVD). In modern works [41, 7], low-rank approximations are widely studied as well.
Since the weight values are always discrete, as an alternative solution and inspired by low-rank approximation works, we converge to an approximated rank rather than compute a precise rank solution. Hence, we define the approximated rank as following:
Definition 1 ( $\delta$ -rank of a matrix). Given a matrix  $W$  and a small error tolerance  $\delta > 0$ , the  $\delta$ -rank of  $W$  is defined as the smallest positive integer  $k$  such that there exist a  $k$ -rank matrix, whose  $l_2$  distance to  $W$  is smaller than  $\delta$ .
In previous works, ranks are evaluated via singular values computed from Singular Value Decomposition (SVD). Zhang et al. [57] uses the sum of the top-k PCA eigenvalues to approximate ranks of layer responses; Lin et al. [33] defines rank as the number of non-negligible singular values and does SVD analysis on feature maps; Shu et al. [47] performs SVD on attention maps and augment model performance by keeping a fatter tail in the singular value distribution. These discoveries all acknowledge that singular values from SVD estimates ranks of matrices. We also leverage SVD to compute  $\delta$ -rank as defined in Definition 1. First, we illustrate that SVD could generate the best low-rank approximation:
Theorem 1 (The best low-rank approximation). Suppose  $W$  is decomposed via SVD and yield  $W = \sum_{i=1}^{r} \sigma_i u_i v_i^T$  where singular values  $\{\sigma_i\}$  are sorted in descending order. Given integer  $k < r$ , the best  $k$ -rank approximation of  $W$ , namely the  $k$ -rank matrix that has the smallest  $l_2$  distance to  $W$  is
$$
\widetilde {W} = \sum_ {i = 1} ^ {k} \sigma_ {i} u _ {i} v _ {i} ^ {T}.
$$
The proof of Theorem 1 will be shown in Appendix. Since SVD could yield the best low-rank approximation, we could use this property to solve  $\delta$ -rank defined in Definition 1. Given weight matrix  $W$ , we search for the smallest  $k$  such that the  $l_{2}$  approximation error of best  $k$ -rank approximation  $\widetilde{W}$  as formulated in Theorem 1 is below the error tolerance  $\delta$ . In this way, we are able to solve the rank of  $W$ .
2.4 Adversarial Optimization for Rank Maintenance
Equipped with the method for matrix rank computation, we hope to formulate a target loss function according to this heuristic such that optimization of the loss could maintain weight ranks.
In contrast to low-rank approximations, high-rank matrices should be hard for low-rank matrices to approximate. Assume  $S$  is the set of all low-rank matrices,  $W$  should keep its distance away from this set  $S$  to increase its rank. But this is a hard problem, for we have to figure out all low-rank matrices. To further simplify the problem, we find the best low-rank approximation rather than all low-rank approximations.  $W$  should estrange itself from the best low-rank approximation whose distance is the farthest from  $W$ . This simplification is valid and will be proved later.
Using this heuristic as motivation, we design an adversarial mechanism that increases the difficulty for  $W$  to be approximated by low-rank matrices, and consequently to advocate higher matrix ranks of  $W$  while pruning. At first, the best low-rank approximation  $\widetilde{W}$  of a small rank  $k$  is generated via Singular Value Decomposition, for the purpose of minimizing its distance to weight  $W$ ; next,  $W$  is optimized to increase the distance from  $\widetilde{W}$ . The procedures could be understood as an adversarial combat between  $W$  and  $\widetilde{W}$ : as the low-rank  $\widetilde{W}$  tries to fit  $W$ ,  $W$  is optimized to keep itself far away from  $\widetilde{W}$ . Mathematically, the combat could be expressed as a min-max problem.
But unlucky, the problem may suffer the risk of not getting converged. When  $\widetilde{W}$  is fixed, the best  $W$  is taken when  $W\to \infty$ . To resolve this issue during optimization, we constrain  $W$  within a euclidean norm ball. In other words, we plug  $\frac{W}{\|W\|_F}$  instead of  $W$  into the max-min problem. The reasons we use  $l_{2}$  normalization are: 1.  $W$  is bounded rather than growing to infinity; 2. the rank of
$W$  could increase if we  $l_{2}$  normalize  $W$  when optimizing the min-max problem, which will be shown in the mathematical proof in the appendix; 3.  $l_{2}$  normalization on weight is equivalent to imposing  $l_{2}$  normalization on its singular values, providing a fair standard for rank comparisons based on the definition of rank in Definition 1 given fixed error tolerance.
Before the introduction of this min-max problem, we introduce several notations:  $\| \cdot \|_F$  is the Frobenius norm (2-norm) of matrices;  $I$  is the identity matrix;  $\overline{W} \coloneqq \frac{W}{\|W\|}$  is the  $l_2$  normalized weight matrix  $W$ ;  $U, \Sigma, V$  are matrices reached from the SVD of  $\overline{W}$ , where  $U = \{u_1, u_2, \ldots\}$  and  $V = \{v_1, v_2, \ldots\}$  are orthonormal bases;  $\Sigma$  is a diagonal matrix where singular values  $\{\sigma_1, \sigma_2, \ldots\}$  are sorted in descending order on the diagonal; operator  $\mathrm{Trun}(U\Sigma V^T) = \sum_{i=1}^{k} \sigma_i u_i v_i^T$  stands for  $k$ -rank truncated SVD, or the  $k$ -rank best approximation of  $\overline{W}$ .
Then formally, we express the min-max problem as follows:
$$
\begin{array}{l} \min  _ {W} \max  _ {U, \Sigma , V} - \| \overline {{W}} - \operatorname {T r u n} \left(U \Sigma V ^ {T}\right) \| _ {F} ^ {2}, \\ \text {s . t .} \quad U ^ {T} U = I, \quad V ^ {T} V = I, \quad \bar {W} = \frac {W}{\| W \|}. \tag {2.5} \\ \end{array}
$$
The optimization target is defined as the adversarial rank loss:
$$
\mathcal {L} _ {\text {r a n k}} = - \left\| \bar {W} - \operatorname {T r u n} \left(U \Sigma V ^ {T}\right) \right\| _ {F} ^ {2}. \tag {2.6}
$$
In deep learning, gradient descent is the most widely applied method for optimization problems, and we also adopt gradient descent for our experiments. Hence in this context, we propose the following theorem, stating that our adversarial rank loss could guide weight  $W$  towards higher rank:
Theorem 2 (Effectiveness of the adversarial rank loss). Given the adversarial rank loss as defined in Equation (2.6). If we optimize  $W$  in rank loss via gradient descent, the rank of  $W$  will increase.
The theorem could be mathematically proved, and the detailed proof will be provided in the appendix.
With the proposed adversarial rank loss, our optimization objective consists of two goals: 1. we hope to reduce the loss for a certain task (e.g. classification, detection, etc.) for good sparse network performance; 2. we hope to reduce rank loss for higher weight ranks. We formulate the Rank-based Pruning objective by doing affine combination of the two goals. Given affine hyperparameter  $\lambda$ , the loss for a certain task  $\mathcal{L}_{task}$ , the adversarial rank loss  $\mathcal{L}_{rank}$ , the Rank-based Pruning (RPG) objective  $\mathcal{L}$  is defined as:
$$
\mathcal {L} := \mathcal {L} _ {\text {t a s k}} + \lambda \mathcal {L} _ {\text {r a n k}}. \tag {2.7}
$$
2.5 The Gradual Pruning Framework
Previous works have proposed various pruning framework, including One-shot Pruning [20], Sparse-to-sparse Pruning [8, 1, 39, 12, 14], and Iterative Magnitude Pruning for Lottery Tickets [15, 43]. Compared with these frameworks, Gradual Pruning (GP) [60] could reach better performance with modest training budget. We adopt Gradual Pruning as the pruning framework, which is a usual practice in many works [48, 59, 34]. GP prunes a small portion of weights once every  $\Delta T$  training steps, trying to maintain sparse network performance via iterative "pruning and training" procedures.
However, it is hard to associate rank loss with Gradual Pruning; we hope the factor of rank could be considered in the choice of weights via the proposed rank loss. Loss gradients are widely-applied weight saliency criteria, because gradient magnitudes reflect the potential importance of pruned weights: if a turned-off weight possesses large gradients with respect to the objective loss function, it is expected for significant contributions to loss reduction [14]. We use periodic gradient-based weight grow similar to previous pruning works [14, 34, 6], i.e. the weights are periodically grown at each binary mask update step. But differently, the rank-based pruning objective (defined as Equation (2.7)) is used for gradients computation with respect to each model weight in our case. In this way, the rank factor is considered during the selection of active weights: there is a tendency that RPG chooses an active set of weights that features high-rank.
Models	VGG19	ResNet32
Sparsity	99%	99.5%	99.9%	99%	99.5%	99.9%
Dense	93.84	94.78
PBW [19]	90.89	10.00	10.00	77.03	73.03	38.64
MLPrune [55]	91.44	88.18	65.38	76.88	67.66	36.09
ProbMask [59]	93.38	92.65	89.79	91.79	89.34	76.87
AC/DC [40]	93.35	80.38	78.91	91.97	88.91	85.07
RPG (Ours)	93.62	93.13	90.49	91.61	91.14	89.36
Table 1. Sparsified VGG-19 and ResNet-32 on CIFAR-10. Baseline results are obtained from [59].
An embedded benefit of periodic gradient-based weight grow lies in computation cost considerations. Singular Value Decomposition (SVD) that is essential for rank computation is costly for large weight tensors. Calculating rank loss for each optimization step is hardly affordable. The adoption of periodic weight updating, however, amortizes the cost of rank loss computations. We also provide an SVD overhead analysis in Sec 3.6.
In summary, Our Rank-based Pruning (RPG) method is formulated as follows: once every  $\Delta T$  training steps, the prune-and-grow procedures that updates binary mask  $M$  is performed. Firstly, we plan the number of parameters to prune and to grow, such that after mask updating, the whole network will reach the target sparsity at the current iteration. Target sparsity will increase gradually as training goes on, which is identical to GP. Secondly, we globally sort all parameters based on magnitude and perform the pruning operation. Thirdly, we grow the parameters based on gradient. For other training steps, mask  $M$  is left unchanged; the active weight values are updated.
Specifically, HRank [33] also leverages matrix rank evaluations in pruning. Our idea is significantly different from HRank [33] in the following aspects: 1. HRank performs filter pruning while our work focuses on weight pruning; 2. HRank evaluates ranks of feature maps, but we evaluate ranks of weight tensors; 3. HRank uses feature rank as filter saliency; our work uses weight rank to guide the update of a sparse network topology.
Our Rank-based PruninG (RPG) method is evaluated on several benchmarks and proved outstanding among recent unstructured pruning baselines. This section presents the experiment results to empirically prove the effectiveness of our RPG method, especially on high sparsities. First, we will show the results of RPG on two image classification datasets: the comparatively small-scaled CIFAR-10, and the large-scaled ImageNet. Then, we will present the results of RPG on downstream vision tasks. Finally, an ablation study will be given.
3.1 CIFAR Experiments
Experiment settings. We first compare our RPG pruning method with other methods on CIFAR-10 classification. CIFAR-10 is one of the most widely used benchmark for image classification. It consists of  $60000 \times 32 \times 32$  images: 50000 for training, and 10000 for validation. We hope to try our RPG method first on this relatively small dataset and look for heuristic patterns.
Among the pruning baselines, we choose ProbMask [59] and AC/DC [40] for comparison because these two methods are intended for high-sparsity pruning. Additionally, ProbMask is a recent baseline that provides both CIFAR and ImageNet classification results, enabling us for further investigation on larger-scale datasets. Other baselines including PBW [19] and MLPrune [55] are earlier conventional pruning baselines for references. For fair comparison, our RPG method is applied to modern CNN structures, i.e. VGG-19 and ResNet-32, and prune for 300 epochs, according to the setting of ProbMask [59]. The results are shown in Table 1.
Results analysis. At relatively low sparsities, the gap between recent baselines are small. ProbMask [59], AC/DC [40], and RPG all give satisfactory results at  $99\%$  compared with early pruning works. But as sparsity further increases, the three methods undergo significant performance decay on
either network. At  $99.5\%$  and  $99.9\%$ , our RPG method shows great advantage over the other two baselines. This discovery inspires us further investigate the high-sparsity potential of RPG on the large-scale ImageNet dataset.
3.2 ResNet-50 on ImageNet
Experiment settings. Sparse ResNet-50 networks evaluated on the ImageNet dataset are the most commonly-used and recognized weight pruning benchmarks. ImageNet ISLVRC2012 [9] is a large scale image classification dataset. It contains 1281K images in the training set and 50K images in the validation set. All the images are shaped  $224 \times 224$  and distributed in 1000 classes. ResNet-50 [22] is a medium-size canonical CNN with 25.5M parameters and 8.2G FLOPs, designed for ImageNet classification.