SlowGuess/NeurIPS_2023_OCR huggingface dataset:
SUBP: Soft Uniform Block Pruning for  $1 \times N$  Sparse CNNs Multithreading Acceleration
Jingyang Xiang $^{1}$  Siqi Li $^{1}$  Jun Chen $^{1}$  Shipeng Bai $^{1}$  Yukai Ma $^{1}$  Guang Dai $^{2,3}$  Yong Liu $^{1*}$
$^{1}$ APRIL Lab, Zhejiang University, Hangzhou, China
$^{2}$ State Grid Corporation of China
$^{3}$ SGIT AI Lab, Shaanxi, China
{jingyangxiang,lsq4747,junc,shipengbai,yukaima}@zju.edu.cn guang.gpai@gmail.com,yongliu@iipc.zju.edu.cn
The study of sparsity in Convolutional Neural Networks (CNNs) has become widespread to compress and accelerate models in environments with limited resources. By constraining N consecutive weights along the output channel to be group-wise non-zero, the recent network with  $1 \times N$  sparsity has received tremendous popularity for its three outstanding advantages: 1) A large amount of storage space saving by a Block Sparse Row matrix. 2) Excellent performance at a high sparsity. 3) Significant speedups on CPUs with Advanced Vector Extensions. Recent work requires selecting and fine-tuning  $1 \times N$  sparse weights based on dense pre-trained weights, leading to the problems such as expensive training cost and memory access, sub-optimal model quality, as well as unbalanced workload across threads (different sparsity across output channels). To overcome them, this paper proposes a novel Soft Uniform Block Pruning (SUBP) approach to train a uniform  $1 \times N$  sparse structured network from scratch. Specifically, our approach tends to repeatedly allow pruned blocks to regrow to the network based on block angular redundancy and importance sampling in a uniform manner throughout the training process. It not only makes the model less dependent on pre-training, reduces the model redundancy and the risk of pruning the important blocks permanently but also achieves balanced workload. Empirically, on ImageNet, comprehensive experiments across various CNN architectures show that our SUBP consistently outperforms existing  $1 \times N$  and structured sparsity methods based on pre-trained models or training from scratch. Source codes and models are available at https://github.com/JingyangXiang/SUBP.
In recent years, convolutional neural networks (CNNs) have achieved great success in image classification [56, 22], object detection [21, 16], semantic segmentation [17, 4] and other fields. The remarkable performance owes to the deeper and wider architectures, also leading to the prohibitively expensive computational cost and colossal memory footprint. Although some efficient architectures are proposed, such as residual connection [22], inception module [58], etc., it is still difficult to deploy the state-of-the-art CNNs on the available CPUs embedded devices with limited resource. Therefore, the network pruning is emerging by pruning the redundancy in CNNs. Due to the appealing perfor
Figure 1: Four mainstream types of pruning in the literature. (a) Unstructured weight pruning removes individual weights at arbitrary locations. (b) Structured filter pruning removes entire convolutional filters. (c) N:M weight pruning (2:4 case) requires at most N out of M consecutive weights along input channels to be non-zero. (d)  $1 \times N$  weight pruning ( $1 \times 4$  case) constrains N consecutive weights along the output channel to be group-wise non-zero.
mance, it has received extensive attention from industry and academia. It is necessary to maintain the model with small size, low memory footprint, low computational cost and high inference speed.
According to the pruning granularity, most existing works about network pruning mainly focus on weight pruning [10, 19, 34] and filter pruning [64, 40, 25, 24, 46]. Weight pruning deletes the weight of filters directly, which may always result in the unstructured sparsity of filters. The expensive memory accesses is also less efficient in saving memory usage and computational cost, since the unstructured model needs a large number of indices to record the positions of the reserved weights and can't take full advantage of Advanced Vector Extensions (AVX), Single Instruction Multiple Data (SIMD) and poorly utilizes memory caches [63]. In contrast, filter pruning tends to prune at the level of filter or even layer. Since filter pruning still preserves the original convolution structure, it enables the model with structured sparsity and more efficient memory usage. Therefore, filter pruning can take full advantage of the high-performance computing library i.e. Basic Linear Algebra Subprogram (BLAS) to achieve apparent acceleration and is more advocated in accelerating the networks. More recently, the development of hardware and operators has given rise to new pruning patterns. The most famous work is N:M fine-grained pruning [55], where N out of M weights are zeros for every continuous M weights along input channels. It also can be seen as a special pattern of weight pruning. Currently, this pattern achieves acceleration only in the case of 2:4. However, it is impossible to be utilized on other types of devices since the instructions of sparse Matrix Multiply-Accumulate (MMA) are specially designed for NVIDIA Ampere Core [55]. Not to mention, in realistic deployment scenarios, GPUs on mobile and embedded devices are not always accessible, and such GPUs are more difficult to be satisfied.
Above all, how to retain the performance and achieve realistic acceleration on mobile CPUs becomes a challengeable but valuable problem. In order to solve this issue, Lin et al. [65] proposed a novel pattern of  $1 \times N$  weight pruning with its merits in realizing both high-performing accuracy and apparent CPUs acceleration for practical model deployment. The  $1 \times N$  pruning pattern provides an intermediate granular level for network pruning, which is coarser as compared to the fine-grained weight but finer as compared to the coarse-grained filter. Fig. 1(d) shows an example of  $1 \times N$  pruning pattern that satisfies  $N = 4$ , the core distinction lies in that  $1 \times N$  pruning consists of  $N$  consecutive weights along the output channel to be group-wise non-zero. These consecutive weights can be stored continuously in the memory cache and the convolution with the inputs can proceed using a block-wise vectorized operation in parallel thanks to AVX and SIMD. The indices memory of the weight positions can also benefit from Block Sparse Row (BSR) matrix and be greatly saved.
However, Lin et al. [65] permanently pruned blocks based on "smaller-norm-less-important" criterion in a non-uniform manner. On the one hand, it reduced the capacity of original model and thus harmed the performance. On the other hand, it left the blocks redundancy untouched and always caused unbalanced workload across threads. What's more, it still followed a traditional pre-training, pruning
and fine-tuning (PPF) pipeline, which depended on pre-trained model and still suffered from the expensive pre-training burden.
To address the above-mentioned limitations, we propose a novel block pruning approach named Soft Uniform Block Pruning (SUBP) to obtain high accuracy sub-model without PPF pipeline. In contrast to the traditional pruning approaches that non-uniformly and permanently remove blocks, we prune and regrow the blocks uniformly via importance sampling, which allows the pruned blocks to be recovered and balanced workload across threads. From intra-layer's perspective, we propose a new Block Pruning criterion by taking Angular Redundancy (BPAR) into account. BPAR identifies the angular redundancy between blocks, so we can prune blocks with redundancy, rather than those with "relatively less" importance. With only one training procedure from scratch, our obtained sub-models yield better accuracy than the previous methods under the similar FLOPs constraints.
To sum up, the contributions of this paper are highlighted as follows:
- We propose a novel block pruning approach named Soft Uniform Block Pruning (SUBP) with three appealing characteristics: (1) a periodic block pruning and regrowing technique via importance sampling, (2) a pruning criterion based on angular redundancy across blocks, and (3) a uniform  $1 \times \mathrm{N}$  sparse pattern for multithreading acceleration.
- Our approach trains a uniform  $1 \times N$  sparse CNNs from scratch, effectively reducing the training cost and achieving better inference latencies in multithreading scenarios, since it circumvents the expensive PPF pipeline and balances workload across threads.
- Extensive experiments on the large ImageNet dataset have demonstrated the effectiveness of our SUBP under different FLOPs constraints. SUBP obtains consistent accuracy improvement across various N and networks, achieving a better trade-off between accuracy and inference latencies. For example, ResNet50  $(1 \times 16)$  model yields  $4 \times$  FLOPs reduction while still achieving  $76.3\%$  top-1 accuracy and suppressing the previous results.
Fully connected operators are widely used in various types of neural networks, and they can be mathematically represented by one matrix multiplication. The natural idea is that all elements of a matrix are not equally important. Removing unimportant elements from the fully connected operators not only reduces the size and the amount of computation, but also potentially improves the generalization performance of the model. Weight pruning (Fig. 1(a)) and filter pruning (Fig. 1(b)) are traditional pruning methods. There are also some special pruning methods that require a high degree of integration with hardware or operators as shown in Fig. 1(c,d). In what follows, we will briefly review some related works.
Weight Pruning. Weight pruning is one of the most widely studied model pruning methods, which removes individual weights at any position of the network. The study of weight pruning could date back to optimal brain damage [33] and optimal surgeon [20], which prune weights based on the Hessian within the loss function. Previous studies removed unimportant weights by using gradient [34], momentum [10], magnitude [19], etc. Han et al. [19] proposed to discard the small weights whose values are below the threshold through an iterative method. Recent some works also payed attention to unstructured sparsity in an adaptive training manner. Ding et al. [10] gradually zeroed out the redundant weights by categorizing weights into two parts and updating them according to different rules. Frankle et al. [13] presented an algorithm to identify subnetworks that were capable of training effectively. Although weight pruning can maintain most of the accuracy of the model with high sparsity, it is difficult to leverage the existing high-efficiency BLAS libraries in practice to accelerate on general hardware due to its irregular weight distribution. Therefore, weight sparsity pruning is rarely used in practical applications.
Filter Pruning. Filter pruning gains achieve noticeable speedup on general hardware after pruning. The filter importance and the pruned network structure are the two most important issues widely studied by researchers. Typical works solve the former issue by devising a certain criterion to measure the importance of filters, including output activation [64], scale factor amplitude of Batch Normalization layer [46], ranks of feature map [40], norm or geometric median of filters [24, 25], etc.. To be specific, he et al. [25] leveraged geometric median to prune filters with redundancy. As for the latter, most works are based on rules of thumb [7, 24], or use evolutionary algorithms [41],
Figure 2: An illustration of our SUBP method, which optimizes the weight values and regrows the removed blocks in one training pass from scratch jointly. In SUBP, both retained and regrown blocks are active, participating in the training iterations. After the last iteration, SUBP exports the pruned model and optimized block weights.
reinforcement learning [26], meta learning [45] and other methods [9, 43] to predict the layer-wise sparsity. For instance, he et al. [26] applied reinforcement learning to compression and acceleration models on mobile devices automatically. However, filter pruning removes entire convolution filters, which may damage the information learned by the network and cause serious accuracy degradation with a high pruning rate. Therefore, filter pruning is still difficult to apply in practical tasks.
Special Pruning. In order to achieve the maximum balance between accuracy and model performance, researchers have proposed various special pruning types [49, 50, 3, 12]. These convolution modes often require the combination of special operators or hardwares. Supported by the NVIDIA Ampere Core, N:M sparsity has gained tremendous attention due to its attractive storage and computation efficiency. Asit et al. [50] and Pool et al. [53] followed a traditional PPF pipeline to implement N:M sparsity. Although retraining can improve the accuracy of the N:M sparsity network, the pre-training-based approach still incurs expensive training costs, which hinders the deployment of N:M sparsity techniques. To address this, zhou et al. [66] proposed a Sparse-refined straight-through estimator (SR-STE) to train N:M sparsity network from scratch. Zhang et al. [65] characterized N:M sparsity as a combinatorial problem and assigned each combination a learnable score to obtain the best subset. Apart from these, researchers have also turned their attention to  $1 \times \mathbf{N}$  sparse networks, which can be accelerated on CPUs. Lin et al. [42] proposed  $1 \times \mathbf{N}$  pruning pattern for CNNs firstly, which achieved better accuracy and speed trade-off than both weight and filter pruning.
3.1 Uniform  $1 \times N$  Block Pruning
We start by introducing symbols and notations in this subsection. Without losing generality, we assume that a CNN has  $L$  layers. We parameterize the tensor connection of CNN with  $\{W^i\in \mathbb{R}^{C_{i + 1}\times C_i\times K_i^h\times K_i^w}|1\leq i\leq L\}$ , where  $C_i,C_{i + 1},K_i^h$  and  $K_{i}^{w}$  represent the numbers of input channels, output channels, kernel height and kernel width of  $i$ -th layer, respectively.
A  $1 \times \mathrm{N}$  pruning pattern partitions the whole  $W^{i}$  into a collection of small blocks. Considering the  $W^{i}$  in the  $i$ -th layer, the  $1 \times \mathrm{N}$  pruning pattern can be achieved by partitioning  $W^{i}$  into a collection of  $C_{i}$  col-groups and then further partitioning  $N_{i}$  into a collection of  $\frac{C_{i+1}}{N}$  row-groups. Consequently, each block is a  $1 \times \mathrm{N}$  matrix along the input channel, which includes  $\mathbf{N}$  consecutive output kernel with the same input channel index. We denote the matrix block set as  $\{B_{j,k}^{i} \in \mathbb{R}^{N \times 1 \times K_{i}^{h} \times K_{i}^{w}} | 1 \leq i \leq L, 0 \leq j \leq \frac{C_{i+1}}{N} - 1, 0 \leq k \leq C_{i} - 1\}$ , to stand for the  $W_{j:N: (j+1): N, k,:}^{i}$ . Based on this partition pattern, the basic pruning granularity of  $1 \times \mathrm{N}$  sparsity falls into these blocks. According to network pruning which can be implemented by imposing a mask  $M^{i}$  upon  $W^{i}$ , we introduce  $\{M_{j,k}^{i} \in \{0,1\} | 1 \leq i \leq L, 0 \leq j \leq \frac{C_{i+1}}{N} - 1, 0 \leq k \leq C_{i} - 1\}$ , to define the objective function of
pre-existing work:
$$
\arg \max  _ {M ^ {i}} \sum_ {i = 1} ^ {L} \mathcal {F} \left(\left\{B _ {j, k} ^ {i} \cdot M _ {j, k} ^ {i} \mid 0 \leq j \leq \frac {C _ {i + 1}}{N} - 1, 0 \leq k \leq C _ {i} - 1 \right\}\right), s. t. \frac {\| M _ {: :} ^ {i} \| _ {0}}{K} = 1 - p \tag {1}
$$
where  $\mathcal{F}(\cdot)$  measures the importance of its input,  $K = \frac{C_{i + 1}}{N}\cdot C_i$  and  $p$  is the expected prune rate for the model.
To reduce the latency of CNNs, multi-core devices are often adopted in most practical scenarios. On a single-core CPU, the most important aspect is the continuity and locality of memory access and calculation in order to fully utilize the cache and vectorization unit. However, nearly all modern processors have multiple cores, and the computation can benefit from multithreading parallelism. Even though the continuity and locality of memory access and calculation are also important to multi-core CPU, the most important aspect of CPU multi-core computing is to achieve balanced workload to fully utilize the computing power of each core and improve computational efficiency. As can be seen from Fig. 3, if the sparsity among different output channel blocks keeps the same, the workload across different threads will be the same.
However, it is easy to know that the conditions in Eq. (1) often lead to different sparsity levels among different output channel blocks, e.g.,  $\| M_{1,:}^{i}\|_{0} = 1$  and  $\| M_{2,:}^{i}\|_{0} = C_{i} - 1$ , which will cause workload imbalance [60] among threads, degrade performance and waste resources during multicore execution. Therefore, to overcome these issues, we propose a uniform  $1\times \mathrm{N}$  block pruning type on the basis of the analysis in above, which is equivalent to constraining the sparsity levels among different output channel blocks to be the same. In another word, the uniform  $1\times \mathrm{N}$  block pruning is a special case of  $1\times \mathrm{N}$  block pruning. Therefore, we define the objective function of uniform  $1\times \mathrm{N}$  block pruning as:
Figure 3: Balanced workload across threads.
$$
\arg \max  _ {M ^ {i}} \sum_ {i = 1} ^ {L} \sum_ {j = 0} ^ {\frac {C _ {i + 1}}{N} - 1} \mathcal {F} \left(\left\{B _ {j, k} ^ {i} \cdot M _ {j, k} ^ {i} \mid 0 \leq k \leq C _ {i} - 1 \right\}\right), s. t. \frac {\left\| M _ {j ;} ^ {i} \right\| _ {0}}{C _ {i}} = 1 - p \tag {2}
$$
3.2 Block Pruning via Angular Redundancy
To get rid of the constraints in the norm-based criterion, we propose a new block pruning criterion by taking block angular redundancy into account and name it as BPAR here. The central idea of angular redundancy is as follows: given a non-zero vector  $X \in \mathbb{R}^{N \times M}$  and a non-zero vector  $W \in \mathbb{R}^{N \times 1}$ , we have
$$
\left\{ \begin{array}{l l} W _ {1} ^ {T} X = \alpha W _ {2} ^ {T} X, & <   W _ {1}, W _ {2} > = 0 \\ W _ {1} ^ {T} X = - \alpha W _ {2} ^ {T} X, & <   W _ {1}, W _ {2} > = \pi \end{array} , \alpha = \frac {\| W _ {1} \| _ {2}}{\| W _ {2} \| _ {2}} \right. \tag {3}
$$
where  $< \cdot, \cdot >$  denotes the angle between two vectors. In particular, we define the  $< 0, \cdot> = 0$ . According to the above analyses, even though a  $1 \times N$  block may be relatively important, this block still has an angular redundancy if there exists another block with a similar orientation angle. Similarly, a relatively less important  $1 \times N$  block can also map the vectors to a meaningful space and extract valuable information if its orientation angle is very different from others.
In order to simplify the explanations, firstly, we vectorize the representation of  $B_{j,k}^{i} \in \mathbb{R}^{N \times 1 \times K_{i}^{h} \times K_{i}^{w}}$  as  $\Omega_{j,k}^{i} \in \mathbb{R}^{N*1*K_{i}^{h}*K_{i}^{w}}$ . Then, we compute the importance score  $S$  of our block as:
$$
S _ {j, k} ^ {i} = \frac {\left\| \Omega_ {j , k} ^ {i} \right\| _ {1}}{\sum_ {m = 0} ^ {C _ {i} - 1} \left\| \Omega_ {j , m} ^ {i} \right\| _ {1}} - \lambda \frac {\sum_ {m = 0} ^ {C _ {i} - 1} \left| \operatorname {C o s i n e S i m} \left(\Omega_ {j , k} ^ {i} , \Omega_ {j , m} ^ {i}\right) \right|}{\sum_ {n = 0} ^ {C _ {i} - 1} \sum_ {m = 0} ^ {C _ {i} - 1} \left| \operatorname {C o s i n e S i m} \left(\Omega_ {j , n} ^ {i} , \Omega_ {j , m} ^ {i}\right) \right|}, \tag {4}
$$
where  $\lambda$  is a balanced hyper-parameter and CosineSim denotes cosine distance between two vectors. Based on Eq. (4), it is obvious that unlike previous criterion solely based on norm, our approach might retain some blocks with smaller norm but larger angular differences in comparison with other blocks. Simultaneously, it may discard the blocks with larger norm but smaller angular differences.
3.3 Soft Uniform Block Pruning
Most of the previous pruning [27, 40, 42] works followed a traditional PPF pipeline, which compressed CNNs in a hard manner. However, once filters or weights are pruned, they can not be recovered, which limits the model capacity and leads to unacceptable accuracy degradation.
To rectify the aforementioned limitations, here we propose a novel soft block pruning approach i.e. Soft Uniform Block Pruning (SUBP) to obtain high accuracy sub-model without a pre-trained super-model or fine-tuning the pruned model. In contrast to the traditional hard pruning approaches, we recover the pruned blocks through an importance sampling approach, which enables the compressed network to have a large model capacity and thus achieves a higher accuracy than others. As shown in Fig. 2, the details of our SUBP can be divided into two stages in what follows.
1) Block Pruning Stage: Firstly, we init all elements in  $M_{j,k}^{i}$  to 1. For the  $i$ -th layer, we use Eq. (4) to evaluate the importance of each block and identify the set of important blocks to retain as:
$$
\mathcal {T} _ {j} ^ {i} = \operatorname {A r g T o p K} \left(\left\{S _ {j, k} ^ {i} \mid 1 \leq k \leq C _ {i} \right\}\right) \in \mathbb {R} ^ {\lceil C _ {i} (1 - p _ {i}) \rceil} \tag {5}
$$
which gives the indices of blocks with the top  $\lceil C_i(1 - p_i)\rceil$  scores of  $B_{j}^{i}$ . Then, we prune the bottom-ranking block by zeroizing  $\{M_{j,k}^i |k\notin \mathcal{T}_j^i\}$ .
2) Block Regrowing Stage: To determine the blocks to regrow, we introduce an importance sampling strategy based on the block importance score. We compute the importance of sampling probabilities by  $p_{j,k}^{i} = \exp \left(\frac{S_{j,k}^{i}}{\tau}\right) / \sum_{m\notin \mathcal{T}_{j}^{i}}\exp \left(\frac{S_{j,m}^{i}}{\tau}\right)$ , where the temperature  $\tau$  is to balance the sampling attention between the different blocks. Then we select the regrow block indices by performing importance sampling  $\mathcal{G}_j^i = \mathrm{Multinomial}(\{p_{j,k}^i |k\notin \mathcal{T}_j^i\} ,\delta_tC_i)$  without replacement based on the regrowing factor  $\delta_{t}$  at  $t$ -th epoch and reset  $\{M_{j,k}^{i}|k\in \mathcal{G}_{j}^{i}\}$  to 1.
To compute the regrowing factor  $\delta_t$ , we employ a gradually schedule [67] to gradually reduce the regrown blocks so that the subnet can converge to the target uniform  $1\times \mathbf{N}$  sparsity stably at the end of training. Specially, the regrowing factor at  $t$ -th epoch is computed as:
$$
\delta_ {t} = \left\{ \begin{array}{l l} 1 - p, & t \leq t _ {s} \\ \delta_ {0} \left(1 - \frac {t - t _ {s}}{t _ {e} - t _ {s}}\right) ^ {3}, & t _ {s} <   t \leq t _ {e} \\ 0, & t _ {e} <   t \end{array} \right. \tag {6}
$$
where  $\delta_0$  is the initial regrowing factor,  $t_s$  and  $t_e$  denote the start and end of the training epochs in the block pruning-regrowing stage respectively. Since the model is trained from scratch and the pruning and regrowing decision based on the weights may not be sufficient enough, therefore, we don't prune blocks in the early stage of training. When  $t \geq t_e$ , the blocks will stop regrowing and we finish pruning the block. The remaining blocks are considered to be the best candidates in the end.
4.1 Experiment Settings and Implementation Details
In this section, we evaluate our SUBP on the largescale dataset ImageNet with the representative networks, including ResNet18, ResNet34, ResNet50 and MobileNetV1. We implement SUBP using the PyTorch framework and NVIDIA RTX 3090 GPUs. The SGD optimizer with a momentum of 0.875 and a weight decay of 3e-5 is adopted. We train all compared networks for 250 epochs with a mini-batch size of 512 and an initial learning rate of 0 which is linearly increased to 0.512 during the first 8 epochs and then decayed to 0 by the cosine learning rate schedule. Same as previous methods [18, 45], we also use label smoothing with the factor 0.1 and apply the standard data augmentation. To keep our method simple and generic, the hyper-parameters  $(\tau, \lambda, \delta_0, t_s, t_e)$  in above are set to (1.0, 1.0, 0.2, 10, 180) and kept constant in our experiments. By adjusting appropriate