Title: SUBP: Soft Uniform Block Pruning for 1$\times$N Sparse CNNs Multithreading Acceleration
Venue: NeurIPS 2023
Accepted: yes
Keywords: Soft Uniform Block Pruning

Block Angular Redundancy

Hardware Acceleration
Abstract: The study of sparsity in Convolutional Neural Networks (CNNs) has become widespread to compress and accelerate models in environments with limited resources. By constraining N consecutive weights along the output channel to be group-wise non-zero, the recent network with 1$\times$N sparsity has received tremendous popularity for its three outstanding advantages: 1) A large amount of storage space saving by a \emph{Block Sparse Row} matrix. 2) Excellent performance at a high sparsity. 3) Significant speedups on CPUs with Advanced Vector Extensions. Recent work requires selecting and fine-tuning 1$\times$N sparse weights based on dense pre-trained weights, leading to the problems such as expensive training cost and memory access, sub-optimal model quality, as well as unbalanced workload across threads (different sparsity across output channels). To overcome them, this paper proposes a novel \emph{\textbf{S}oft \textbf{U}niform \textbf{B}lock \textbf{P}runing} (SUBP) approach to train a uniform 1$\times$N sparse structured network from scratch. Specifically, our approach tends to repeatedly allow pruned blocks to regrow to the network based on block angular redundancy and importance sampling in a uniform manner throughout the training process. It not only makes the model less dependent on pre-training, reduces the model redundancy and the risk of pruning the important blocks permanently but also achieves balanced workload. Empirically, on ImageNet, comprehensive experiments across various CNN architectures show that our SUBP consistently outperforms existing 1$\times$N and structured sparsity methods based on pre-trained models or training from scratch. Source codes and models are available at \url{https://github.com/JingyangXiang/SUBP}.

SUBP: Soft Uniform Block Pruning for 1 × N Sparse CNNs Multithreading Acceleration

Abstract

The study of sparsity in Convolutional Neural Networks (CNNs) has become widespread to compress and accelerate models in environments with limited resources. By constraining N consecutive weights along the output channel to be group-wise non-zero, the recent network with 1 × N sparsity has received tremendous popularity for its three outstanding advantages: 1) A large amount of storage space saving by a Block Sparse Row matrix. 2) Excellent performance at a high sparsity. 3) Significant speedups on CPUs with Advanced Vector Extensions. Recent work requires selecting and fine-tuning 1 × N sparse weights based on dense pre-trained weights, leading to the problems such as expensive training cost and memory access, sub-optimal model quality, as well as unbalanced workload across threads (different sparsity across output channels). To overcome them, this paper proposes a novel S oft U niform B lock P runing (SUBP) approach to train a uniform 1 × N sparse structured network from scratch. Specifically, our approach tends to repeatedly allow pruned blocks to regrow to the network based on block angular redundancy and importance sampling in a uniform manner throughout the training process. It not only makes the model less dependent on pre-training, reduces the model redundancy and the risk of pruning the important blocks permanently but also achieves balanced workload. Empirically, on ImageNet, comprehensive experiments across various CNN architectures show that our SUBP consistently outperforms existing 1 × N and structured sparsity methods based on pre-trained models or training from scratch. Source codes and models are available at https://github.com/JingyangXiang/SUBP .

1 Introduction

In recent years, convolutional neural networks (CNNs) have achieved great success in image classification [56, 22], object detection [21, 16], semantic segmentation [17, 4] and other fields. The remarkable performance owes to the deeper and wider architectures, also leading to the prohibitively expensive computational cost and colossal memory footprint. Although some efficient architectures are proposed, such as residual connection [22], inception module [58], etc., it is still difficult to deploy the state-of-the-art CNNs on the available CPUs embedded devices with limited resource. Therefore, the network pruning is emerging by pruning the redundancy in CNNs. Due to the appealing perfor-

Deploy

Deploy

Deploy

Deploy

None-zero

data

None-zero

data

None-zero

data

None-zero

data

Indptr

Indices

Indices

Indices

(a)  Unstructured

weight pruning

(b)  Structured filter

pruning

(c)  N:M weight

pruning

(d)  1xN weight

pruning

Unstructured  sparse

weight

Structured sparse

filter

N:M structured

sparse weight

1xN structured

sparse weight

mance, it has received extensive attention from industry and academia. It is necessary to maintain the model with small size, low memory footprint, low computational cost and high inference speed.

According to the pruning granularity, most existing works about network pruning mainly focus on weight pruning [10, 19, 34] and filter pruning [64, 40, 25, 24, 46]. Weight pruning deletes the weight of filters directly, which may always result in the unstructured sparsity of filters. The expensive memory accesses is also less efficient in saving memory usage and computational cost, since the unstructured model needs a large number of indices to record the positions of the reserved weights and can't take full advantage of Advanced Vector Extensions (AVX), Single Instruction Multiple Data (SIMD) and poorly utilizes memory caches [63]. In contrast, filter pruning tends to prune at the level of filter or even layer. Since filter pruning still preserves the original convolution structure, it enables the model with structured sparsity and more efficient memory usage. Therefore, filter pruning can take full advantage of the high-performance computing library i.e. Basic Linear Algebra Subprogram (BLAS) to achieve apparent acceleration and is more advocated in accelerating the networks. More recently, the development of hardware and operators has given rise to new pruning patterns. The most famous work is N:M fine-grained pruning [55], where N out of M weights are zeros for every continuous M weights along input channels. It also can be seen as a special pattern of weight pruning. Currently, this pattern achieves acceleration only in the case of 2:4. However, it is impossible to be utilized on other types of devices since the instructions of sparse Matrix Multiply-Accumulate (MMA) are specially designed for NVIDIA Ampere Core [55]. Not to mention, in realistic deployment scenarios, GPUs on mobile and embedded devices are not always accessible, and such GPUs are more difficult to be satisfied.

Above all, how to retain the performance and achieve realistic acceleration on mobile CPUs becomes a challengeable but valuable problem. In order to solve this issue, Lin et al. [65] proposed a novel pattern of 1 × N weight pruning with its merits in realizing both high-performing accuracy and apparent CPUs acceleration for practical model deployment. The 1 × N pruning pattern provides an intermediate granular level for network pruning, which is coarser as compared to the fine-grained weight but finer as compared to the coarse-grained filter. Fig. 1(d) shows an example of 1 × N pruning pattern that satisfies N=4, the core distinction lies in that 1 × N pruning consists of N consecutive weights along the output channel to be group-wise non-zero. These consecutive weights can be stored continuously in the memory cache and the convolution with the inputs can proceed using a block-wise vectorized operation in parallel thanks to A VX and SIMD. The indices memory of the weight positions can also benefit from Block Sparse Row (BSR) matrix and be greatly saved.

However, Lin et al. [65] permanently pruned blocks based on 'smaller-norm-less-important' criterion in a non-uniform manner. On the one hand, it reduced the capacity of original model and thus harmed the performance. On the other hand, it left the blocks redundancy untouched and always caused unbalanced workload across threads. What's more, it still followed a traditional pre-training, pruning

and fine-tuning (PPF) pipeline, which depended on pre-trained model and still suffered from the expensive pre-training burden.

To address the above-mentioned limitations, we propose a novel block pruning approach named S oft U niform B lock P runing (SUBP) to obtain high accuracy sub-model without PPF pipeline. In contrast to the traditional pruning approaches that non-uniformly and permanently remove blocks, we prune and regrow the blocks uniformly via importance sampling, which allows the pruned blocks to be recovered and balanced workload across threads. From intra-layer's perspective, we propose a new B lock P runing criterion by taking A ngular R edundancy (BPAR) into account. BPAR identifies the angular redundancy between blocks, so we can prune blocks with redundancy, rather than those with 'relatively less' importance. With only one training procedure from scratch, our obtained sub-models yield better accuracy than the previous methods under the similar FLOPs constraints.

To sum up, the contributions of this paper are highlighted as follows:

We propose a novel block pruning approach named S oft U niform B lock P runing (SUBP) with three appealing characteristics: (1) a periodic block pruning and regrowing technique via importance sampling, (2) a pruning criterion based on angular redundancy across blocks, and (3) a uniform 1 × N sparse pattern for multithreading acceleration.

Our approach trains a uniform 1 × N sparse CNNs from scratch, effectively reducing the training cost and achieving better inference latencies in multithreading scenarios, since it circumvents the expensive PPF pipeline and balances workload across threads.

Extensive experiments on the large ImageNet dataset have demonstrated the effectiveness of our SUBP under different FLOPs constraints. SUBP obtains consistent accuracy improvement across various N and networks, achieving a better trade-off between accuracy and inference latencies. For example, ResNet50 (1 × 16) model yields 4 × FLOPs reduction while still achieving 76.3% top-1 accuracy and suppressing the previous results.

2 Related Work

Fully connected operators are widely used in various types of neural networks, and they can be mathematically represented by one matrix multiplication. The natural idea is that all elements of a matrix are not equally important. Removing unimportant elements from the fully connected operators not only reduces the size and the amount of computation, but also potentially improves the generalization performance of the model. Weight pruning (Fig. 1(a)) and filter pruning (Fig. 1(b)) are traditional pruning methods. There are also some special pruning methods that require a high degree of integration with hardware or operators as shown in Fig. 1(c,d). In what follows, we will briefly review some related works.

Weight Pruning. Weight pruning is one of the most widely studied model pruning methods, which removes individual weights at any position of the network. The study of weight pruning could date back to optimal brain damage [33] and optimal surgeon [20], which prune weights based on the Hessian within the loss function. Previous studies removed unimportant weights by using gradient [34], momentum [10], magnitude [19], etc . Han et al. [19] proposed to discard the small weights whose values are below the threshold through an iterative method. Recent some works also payed attention to unstructured sparsity in an adaptive training manner. Ding et al. [10] gradually zeroed out the redundant weights by categorizing weights into two parts and updating them according to different rules. Frankle et al. [13] presented an algorithm to identify subnetworks that were capable of training effectively. Although weight pruning can maintain most of the accuracy of the model with high sparsity, it is difficult to leverage the existing high-efficiency BLAS libraries in practice to accelerate on general hardware due to its irregular weight distribution. Therefore, weight sparsity pruning is rarely used in practical applications.

Filter Pruning. Filter pruning gains achieve noticeable speedup on general hardware after pruning. The filter importance and the pruned network structure are the two most important issues widely studied by researchers. Typical works solve the former issue by devising a certain criterion to measure the importance of filters, including output activation [64], scale factor amplitude of Batch Normalization layer [46], ranks of feature map [40], norm or geometric median of filters [24, 25], etc. . To be specific, he et al. [25] leveraged geometric median to prune filters with redundancy. As for the latter, most works are based on rules of thumb [7, 24], or use evolutionary algorithms [41],

Layer i

Layer i+1

Randomly

initialized model

Layer i

Layer i+1

Block pruning stage

Layer i

Block regrowing stage

Step 1

Layer i

Block pruning stage

Layer i

Block regrowing stage

Step 2

Training

Training

Step≤N？

Training

Yes

Training

No

Step 3 to N

BPAR prune

criterion

Prune blocks

with a layer

Regrow

scheduler

Importance

sampling

BPAR prune

criterion

Prune blocks

with a layer

Regrow

scheduler

Importance

sampling

Retained weight

Regrown weight

Pruned weight

Training

Pruned model &

optimized block

weights

Layer i+1

Layer i+1

Layer i+1

reinforcement learning [26], meta learning [45] and other methods [9, 43] to predict the layer-wise sparsity. For instance, he et al. [26] applied reinforcement learning to compression and acceleration models on mobile devices automatically. However, filter pruning removes entire convolution filters, which may damage the information learned by the network and cause serious accuracy degradation with a high pruning rate. Therefore, filter pruning is still difficult to apply in practical tasks.

Special Pruning. In order to achieve the maximum balance between accuracy and model performance, researchers have proposed various special pruning types [49, 50, 3, 12]. These convolution modes often require the combination of special operators or hardwares. Supported by the NVIDIA Ampere Core, N:M sparsity has gained tremendous attention due to its attractive storage and computation efficiency. Asit et al. [50] and Pool et al. [53] followed a traditional PPF pipeline to implement N:M sparsity. Although retraining can improve the accuracy of the N:M sparsity network, the pre-training-based approach still incurs expensive training costs, which hinders the deployment of N:M sparsity techniques. To address this, zhou et al. [66] proposed a Sparse-refined straight-through estimator (SR-STE) to train N:M sparsity network from scratch. Zhang et al. [65] characterized N:M sparsity as a combinatorial problem and assigned each combination a learnable score to obtain the best subset. Apart from these, researchers have also turned their attention to 1 × N sparse networks, which can be accelerated on CPUs. Lin et al. [42] proposed 1 × N pruning pattern for CNNs firstly, which achieved better accuracy and speed trade-off than both weight and filter pruning.

3 Methodology

3.1 Uniform 1 × N Block Pruning

We start by introducing symbols and notations in this subsection. Without losing generality, we assume that a CNN has L layers. We parameterize the tensor connection of CNN with { W i ∈ R C i +1 × C i × K h i × K w i | 1 ≤ i ≤ L } , where C i , C i +1 , K h i and K w i represent the numbers of input channels, output channels, kernel height and kernel width of i -th layer, respectively.

A 1 × N pruning pattern partitions the whole W i into a collection of small blocks. Considering the W i in the i -th layer, the 1 × N pruning pattern can be achieved by partitioning W i into a collection of C i col-groups and then further partitioning N i into a collection of C i +1 N row-groups. Consequently, each block is a 1 × N matrix along the input channel, which includes N consecutive output kernel with the same input channel index. We denote the matrix block set as { B i j,k ∈ R N × 1 × K h i × K w i | 1 ≤ i ≤ L, 0 ≤ j ≤ C i +1 N -1 , 0 ≤ k ≤ C i -1 } , to stand for the W i j · N :( j +1) · N,k, : , : . Based on this partition pattern, the basic pruning granularity of 1 × N sparsity falls into these blocks. According to network pruning which can be implemented by imposing a mask M i upon W i , we introduce { M i ∈ { 0 , 1 }| 1 ≤ i ≤ L, 0 ≤ j ≤ C i +1 -1 , 0 ≤ k ≤ C i -1 } , to define the objective function of j,k N

pre-existing work:

where F ( · ) measures the importance of its input, K = C i +1 N · C i and p is the expected prune rate for the model.

To reduce the latency of CNNs, multi-core devices are often adopted in most practical scenarios. On a single-core CPU, the most important aspect is the continuity and locality of memory access and calculation in order to fully utilize the cache and vectorization unit. However, nearly all modern processors have multiple cores, and the computation can benefit from multithreading parallelism. Even though the continuity and locality of memory access and calculation are also important to multi-core CPU, the most important aspect of CPU multi-core computing is to achieve balanced workload to fully utilize the computing power of each core and improve computational efficiency. As can be seen from Fig. 3, if the sparsity among different output channel blocks keeps the same, the workload across different threads will be the same.

However, it is easy to know that the conditions in Eq. (1) often lead to different sparsity levels among different output channel blocks, e.g., ∥ M i 1 , : ∥ 0 = 1 and ∥ M i 2 , : ∥ 0 = C i -1 , which will cause workload imbalance [60] among threads, degrade performance and waste resources during multicore execution. Therefore, to overcome these issues, we propose a uniform 1 × N block pruning type on the basis of the analysis in above, which is equivalent to constraining the sparsity levels among different output channel blocks to be the same. In another word, the uniform 1 × N block pruning is a special case of 1 × N block pruning. Therefore, we define the objective function of uniform 1 × N block pruning as:

Indices

3

1334

135

>

3

4

×

Paralleized Block-wise

VectorizedOperation

Thread2

ndus

Sparse Weight

Balanced loading

across threads

Indices

Indptr

3

133

135

3

4

×

Paralleized Block-wise

4

VectorizedOperation

Thread1

3.2 Block Pruning via Angular Redundancy

To get rid of the constraints in the norm-based criterion, we propose a new block pruning criterion by taking block angular redundancy into account and name it as BPAR here. The central idea of angular redundancy is as follows: given a non-zero vector X ∈ R N × M and a non-zero vector W ∈ R N × 1 , we have

where < · , · > denotes the angle between two vectors. In particular, we define the < 0 , · > = 0 . According to the above analyses, even though a 1 × N block may be relatively important, this block still has an angular redundancy if there exists another block with a similar orientation angle. Similarly, a relatively less important 1 × N block can also map the vectors to a meaningful space and extract valuable information if its orientation angle is very different from others.

In order to simplify the explanations, firstly, we vectorize the representation of B i j,k ∈ R N × 1 × K h i × K w i as Ω i j,k ∈ R N ∗ 1 ∗ K h i ∗ K w i . Then, we compute the importance score S of our block as:

where λ is a balanced hyper-parameter and CosineSim denotes cosine distance between two vectors. Based on Eq. (4), it is obvious that unlike previous criterion solely based on norm, our approach might retain some blocks with smaller norm but larger angular differences in comparison with other blocks. Simultaneously, it may discard the blocks with larger norm but smaller angular differences.

3.3 Soft Uniform Block Pruning

Most of the previous pruning [27, 40, 42] works followed a traditional PPF pipeline, which compressed CNNs in a hard manner. However, once filters or weights are pruned, they can not be recovered, which limits the model capacity and leads to unacceptable accuracy degradation.

To rectify the aforementioned limitations, here we propose a novel soft block pruning approach i.e. S oft U niform B lock P runing (SUBP) to obtain high accuracy sub-model without a pre-trained supermodel or fine-tuning the pruned model. In contrast to the traditional hard pruning approaches, we recover the pruned blocks through an importance sampling approach, which enables the compressed network to have a large model capacity and thus achieves a higher accuracy than others. As shown in Fig. 2, the details of our SUBP can be divided into two stages in what follows.

1) Block Pruning Stage: Firstly, we init all elements in M i j,k to 1. For the i -th layer, we use Eq. (4) to evaluate the importance of each block and identify the set of important blocks to retain as:

which gives the indices of blocks with the top ⌈ C i (1 -p i ) ⌉ scores of B i j . Then, we prune the bottom-ranking block by zeroizing { M i j,k | k / ∈ T i j } .

2) Block Regrowing Stage: To determine the blocks to regrow, we introduce an importance sampling strategy based on the block importance score. We compute the importance of sampling probabilities by p i j,k = exp ( S i j,k τ ) / ∑ m/ ∈T i j exp ( S i j,m τ ) , where the temperature τ is to balance the sampling attention between the different blocks. Then we select the regrow block indices by performing importance sampling G i j = Multinomial ( { p i j,k | k / ∈ T i j } , δ t C i ) without replacement based on the regrowing factor δ t at t -th epoch and reset { M i j,k | k ∈ G i j } to 1.

To compute the regrowing factor δ t , we employ a gradually schedule [67] to gradually reduce the regrown blocks so that the subnet can converge to the target uniform 1 × N sparsity stably at the end of training. Specially, the regrowing factor at t -th epoch is computed as:

where δ 0 is the initial regrowing factor, t s and t e denote the start and end of the training epochs in the block pruning-regrowing stage respectively. Since the model is trained from scratch and the pruning and regrowing decision based on the weights may not be sufficient enough, therefore, we don't prune blocks in the early stage of training. When t ≥ t e , the blocks will stop regrowing and we finish pruning the block. The remaining blocks are considered to be the best candidates in the end.

4 Experiments

4.1 Experiment Settings and Implementation Details

In this section, we evaluate our SUBP on the largescale dataset ImageNet with the representative networks, including ResNet18, ResNet34, ResNet50 and MobileNetV1. We implement SUBP using the PyTorch framework and NVIDIA RTX 3090 GPUs. The SGD optimizer with a momentum of 0.875 and a weight decay of 3e-5 is adopted. We train all compared networks for 250 epochs with a mini-batch size of 512 and an initial learning rate of 0 which is linearly increased to 0.512 during the first 8 epochs and then decayed to 0 by the cosine learning rate schedule. Same as previous methods [18, 45], we also use label smoothing with the factor 0.1 and apply the standard data augmentation. To keep our method simple and generic, the hyper-parameters ( τ , λ , δ 0 , t s , t e ) in above are set to (1.0, 1.0, 0.2, 10, 180) and kept constant in our experiments. By adjusting appropriate

parameters for different models and pruning rates, better results should be obtained in general. In this paper, we calculate FLOPs by counting multiplication and addition as one operation as He [22] did.

4.2 Influence of Pruning Criterion

In this subsection, we will investigate the influence of 1 × N pruning criterion. Table 1 shows the compared results with respect to ℓ 1 norm [42] and BPAR. As suggested by [37, 48, 51, 28], the pruning criterion selects the appropriate sub-model on the basis of its importance or redundancy, which plays an important role in the traditional PPF approach. Table 1 tells us that our BPAR

outperforms the norm-based method on the ImageNet dataset. For MobileNetV1 and ResNet50, our BPAR can achieve consistent accuracy improvements with the same sparsity and inference speedup. On MobileNetV1(1 × 16), it obtains 0.74% and 0.38% top1-accuracy and top5-accuracy improvement when BPAR is applied. In particular, for pruning a pre-trained MobileNetV1, BPAR achieves better results for N=8 than ℓ 1 norm for N=2, which indicates 1 × N pruning based on BPAR can obtain a better trade-off between accuracy and inference latencies. Similar results can also be observed on the ResNet50 in Table 1. In essence, our BPAR explicitly utilizes the relationship and identifies the mutual angular redundancy between blocks, giving rise to its superior performance.

4.3 Results on ImageNet

To verify the effectiveness of our SUBP, we apply it to the heavyweight CNN model (i.e. ResNet [22]) with different depths and the lightweight CNN model (i.e. MobileNet) initialized with random weights on ImageNet [5] dataset. Here, ResNet-18/34/50 and MobileNetV1 are used as the baseline models and have 1.8/3.7/4.1 GFLOPs and 572 MFLOPs.

The results in Table 2 show that SUBP achieves noticeably higher accuracy than the state-of-the-art pruning methods under the same FLOPs constraints. For example, our SUBP on ResNet50(1 × 16) with 2 × FLOPs reduction achieves 77.6% top-1 accuracy, which is 3.1%, 1.6%, 0.7% and 0.2% higher than Taylor[51], SCOP [59], CafeNet [57] and CHEX [29] respectively. The results on 1 × 32 sparsity also demonstrate the superiority against the other methods. On the other hand, at the same target accuracy, our SUBP also achieves higher FLOPs reduction. For example, our SUBP achieves 4 × FLOPs reduction and 76.0% top-1 accuracy on ResNet50(1 × 32) compared to the SCOP, which only yields 1.9 × FLOPs reduction. We further observe that SUBP also achieves higher accuracy at a fraction of the training cost for MobileNetV1. For instance, SUBP achieves 71.1% top-1 accuracy on MobileNetV1(1 × 32, 250 epoch) under the 279M FLOPs constraint, which is 0.5% higher than MetaPruing [45](320 epoch). This is because SUBP can dynamically explore the optimal sub-model in one training pass from scratch, circumventing the expensive PPF cycles.

4.4 Results on Object Detection and Instance Segmentation

To further explore the performance of SUBP in downstream tasks, we conduct experiments on object detection and instance segmentation on the challenging COCO dataset [44]. The results are shown in Table 3 and Table 4. We adopt the classical method Faster RCNN [54] for object detection and Mask RCNN [21] for instance segmentation. We use ResNet50 with different sparse ratios and block sizes as backbone. All the experiments are conducted based on MMDetection [1]. Compared to Dense ResNet50, the SUBP can achieve competitive results, which further demonstrates itsrobustness and superiority on downstream computer vision tasks.

4.5 SUBP from Pre-trained Models

In order to further investigate the generality property of our approach, we apply SUBP to the model initialized with pre-trained weights. For an equitable comparison with other PPF methods, we adopt the pre-trained ResNet18 provided by the torchvision 2 and run SUBP for 120 epochs as the previous one did. The results in Table 5 show that our SUBP still achieves competitive top-1 and top-5 accuracy under the same FLOPs compared to the previous state-of-the-art PPF methods. Furthermore, when ResNet18 is trained for 90+120 epochs, it achieves 69.5% top-1 accuracy, which is only 0.4% lower than training it from scratch for 250 epochs in Table 2.

4.6 Deployment Efficiency Analysis

0.125

0.25

0.375

0.5

0.625

0.75

0.875

Pruning Rate

p

40

65

90

115

140

165

190

Network Latency (ms)

Dense

1x32(Uniform)

1x16(Uniform)

1x8(Uniform)

1x32(Non-Uniform)

1x16(Non-Uniform)

1x8(Non-Uniform)

0.125

0.25

0.375

0.5

0.625

0.75

0.875

Pruning Rate

p

30

55

80

105

130

155

180

Network Latency (ms)

Dense

1x32(Uniform)

1x16(Uniform)

1x8(Uniform)

1x32(Non-Uniform)

1x16(Non-Uniform)

1x8(Non-Uniform)

To further explore the acceleration capacity with respect to different N and prune rates on CPUs-based platforms, we adopt TVM [2] to compile the uniform and non-uniform 1 × Npruned models. For a fair comparison, we also consider TVM for dense model to obtain baseline latency. We deploy the 1 × N pruned models to obtain network latencies on the arm platform of Apple M1 Pro CPU @ 3.20GHz. From Fig. 4, we observe 1 × N pruned model achieves noticeable latency reductions across various pruning rates and N. For example, when the pruning rate is greater than 0.25, the inference latencies of different N are all the better than their dense baseline under the single thread scenario. When the thread num is set to 2, the inference speed of different configurations has been improved. Due to the workload imbalance between threads in non-uniform 1 × N, it always lags behind its uniform counterpart. From Fig. 4, there are also two points worth noting: 1) the improvement of 1 × N is not as significant as a vanilla convolution in a multithreading context; 2) N=16 achieves a faster inference speed than other N on M1 Pro. Therefore, optimizing multithreading inference with 1 × N pruning and selecting appropriate N based on suitable platforms are directions that can be further explored.

5 Limitation and Discussion

Firstly, our SUBP masks weights throughout the training and still needs dense computations. Therefore, while SUBP has advantages over methods that depend on PPF pipeline, there is still room for improving its training efficiency. Secondly, there are still missing experiments, including applying 1 × N sparsity on other types of DNNs like RNN, transformer and other tasks including object detection, natural language processing, etc. . We will design corresponding high-performance operators to improve the training efficiency of SUBP, and explore the performance of 1 × N sparsity in other types of DNNs and tasks to verify its broader applicability in our future work.

6 Conclusion

Uniform 1 × Nsparsity is an important technique that allows fast inference on multithreading scenarios under multi-core architecture. This paper proposes soft uniform block pruning, SUBP, to efficiently train a uniform 1 × N sparsity network from scratch. SUBP dynamically adjusts the pruning blocks based on a periodic pruning and regrowing process, which prevents the important blocks from being prematurely pruned and keeps the model's capacity. We also present a new block pruning criterion named BPAR, which explicitly considers the mutual angular redundancy between blocks rather than 'relatively less' importance only. By proposing a BPAR based block subset selection approach for pruning and an importance sampling based strategy for regrowing, we can obtain a sub-model with high accuracy by end-to-end implementation without pre-training a large model or requiring extra fine-tuning. Extensive experiments have exhibited our method can effectively reduce the FLOPs and inference latencies while achieving superior accuracy over several SOTAs.

Acknowledgments and Disclosure of Funding

This work is funded in part by the Key Research and Development Project of Zhejiang Province under Grant 2021C01035.

References

Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:1906.07155 , 2019.

Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, et al. Tvm: An automated end-to-end optimizing compiler for deep learning. In Symposium on Operating Systems Design and Implementation (OSDI) , pages 578-594, 2018.

Zhuangzhi Chen, Jingyang Xiang, Yao Lu, Qi Xuan, Zhen Wang, Guanrong Chen, and Xiaoniu Yang. Rgp: Neural network pruning through regular graph with edges swapping. IEEE Transactions on Neural Networks and Learning Systems , pages 1-13, 2023.

Ioana Croitoru, Simion-Vlad Bogolin, and Marius Leordeanu. Unsupervised learning of foreground object segmentation. International Journal of Computer Vision (IJCV) , 127(9):12791302, 2019.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 248-255, 2009.

Xiaohan Ding, Guiguang Ding, Yuchen Guo, and Jungong Han. Centripetal sgd for pruning very deep convolutional networks with complicated structure. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 4943-4953, 2019.

Xiaohan Ding, Guiguang Ding, Yuchen Guo, Jungong Han, and Chenggang Yan. Approximated oracle filter pruning for destructive cnn width optimization. In International Conference on Machine Learning (ICML) , pages 1607-1616. PMLR, 2019.

Xiaohan Ding, Tianxiang Hao, Jianchao Tan, Ji Liu, Jungong Han, Yuchen Guo, and Guiguang Ding. Lossless cnn channel pruning via decoupling remembering and forgetting. Proceedings of the IEEE/CVF International Conference on Com-puter Vision , 2021.

Xiaohan Ding, Tianxiang Hao, Jianchao Tan, Ji Liu, Jungong Han, Yuchen Guo, and Guiguang Ding. Resrep: Lossless cnn pruning via decoupling remembering and forgetting. In IEEE International Conference on Computer Vision (ICCV) , pages 4510-4520, 2021.

Xiaohan Ding, Xiangxin Zhou, Yuchen Guo, Jungong Han, Ji Liu, et al. Global sparse momentum sgd for pruning very deep neural networks. In Advances in Neural Information Processing Systems (NeurIPS) , pages 6382-6394, 2019.

Xuanyi Dong and Yi Yang. Network pruning via transformable architecture search. Proceedings of Advances in Neural Information Processing Systems , pages 759-770, 2019.

Erich Elsen, Marat Dukhan, Trevor Gale, and Karen Simonyan. Fast sparse convnets. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 14629-14638, 2020.

Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations (ICLR) , 2019.

Shangqian Gao, Feihu Huang, Weidong Cai, and Heng Huang. Network pruning via performance maximization. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9270-9280, 2021.

Shangqian Gao, Feihu Huang, Jian Pei, and Heng Huang. Discrete model compression with resource constraint for deep neural networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1899-1908, 2020.

Ross Girshick. Fast r-cnn. In IEEE International Conference on Computer Vision (ICCV) , pages 1440-1448, 2015.

Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In IEEE International Conference on Computer Vision (ICCV) , pages 580-587, 2014.

Shaopeng Guo, Yujie Wang, Quanquan Li, and Junjie Yan. Dmcp: Differentiable markov channel pruning for neural networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1539-1547, 2020.

Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems (NeurIPS) , pages 1135-1143, 2015.

Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain surgeon. Advances in neural information processing systems , 5, 1992.

Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick. Mask r-cnn. In IEEE International Conference on Computer Vision (ICCV) , 2017.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 770-778, 2016.

Yang He, Yuhang Ding, Ping Liu, Linchao Zhu, Hanwang Zhang, and Yi Yang. Learning filter pruning criteria for deep convolutional neural networks acceleration. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 2009-2018, 2020.

Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. In International Joint Conference on Artificial Intelligence (IJCAI) , pages 2234-2240, 2018.

Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 4340-4349, 2019.

Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. Amc: Automl for model compression and acceleration on mobile devices. In Proceedings of the European conference on computer vision (ECCV) , pages 784-800, 2018.

Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE international conference on computer vision , pages 1389-1397, 2017.

Zejiang Hou and Sun-Yuan Kung. A feature-map discriminant perspective for pruning deep neural networks. arXiv preprint arXiv:2005.13796 , 2020.

Zejiang Hou, Minghai Qin, Fei Sun, Xiaolong Ma, Kun Yuan, Yi Xu, Yen-Kuang Chen, Rong Jin, Yuan Xie, and Sun-Yuan Kung. Chex: Channel exploration for cnn model compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12287-12298, 2022.

Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 , 2017.

Zehao Huang and Naiyan Wang. Data-driven sparse structure selection for deep neural networks. Proceedings of the European Conference on Computer Vision , pages 304-320, 2018.

Minsoo Kang and Bohyung Han. Operation-aware soft channel pruning using differentiable masks. Proceedings of International Conference on Machine Learning , pages 5122-5131, 2020.

Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems , 2, 1989.

Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. Snip: Single-shot network pruning based on connection sensitivity. In International Conference on Learning Representations (ICLR) , 2019.

Bailin Li, Bowen Wu, Jiang Su, and Guangrun Wang. Eagleeye: Fast sub-net evaluation for efficient neural network pruning. Proceedings of European Conference on Computer Vision , pages 639-654, 2020.

Changlin Li, Guangrun Wang, Bing Wang, Xiaodan Liang, Zhihui Li, and Xiaojun Chang. Dynamic slimmable network. Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021.

Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. Proceedings of International Conference on Learning Representations , 2017.

Yawei Li, Shuhang Gu, Christoph Mayer, Luc Van Gool, and Radu Timofte. Group sparsity: The hinge between filter pruning and decomposition for network compression. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8018-8027, 2020.

Lucas Liebenwein, Cenk Baykal, Harry Lang, Dan Feldman, and Daniela Rus. Provable filter pruning for efficient neural networks. Proceedings of International Conference on Learning Representations , 2020.

Mingbao Lin, Rongrong Ji, Yan Wang, Yichen Zhang, Baochang Zhang, Yonghong Tian, and Ling Shao. Hrank: Filter pruning using high-rank feature map. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1529-1538, 2020.

Mingbao Lin, Rongrong Ji, Yuxin Zhang, Baochang Zhang, Yongjian Wu, and Yonghong Tian. Channel pruning via automatic structure search. In International Joint Conference on Artificial Intelligence (IJCAI) , pages 673-679, 2020.

Mingbao Lin, Yuxin Zhang, Yuchao Li, Bohong Chen, Fei Chao, Mengdi Wang, Shen Li, Yonghong Tian, and Rongrong Ji. 1xn pattern for pruning convolutional neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022.

Shaohui Lin, Rongrong Ji, Chenqian Yan, Baochang Zhang, Liujuan Cao, Qixiang Ye, Feiyue Huang, and David Doermann. Towards optimal structured cnn pruning via generative adversarial learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2790-2799, 2019.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision (ECCV) , pages 740-755, 2014.

Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang, Tim Kwang-Ting Cheng, and Jian Sun. Metapruning: Meta learning for automatic neural network channel pruning. In IEEE International Conference on Computer Vision (ICCV) , pages 3296-3305, 2019.

Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In Proceedings of the IEEE international conference on computer vision , pages 2736-2744, 2017.

Jian-Hao Luo and Jianxin Wu. Neural network pruning with residual-connections and limiteddata. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 14581467, 2020.

Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In IEEE International Conference on Computer Vision (ICCV) , pages 5058-5066, 2017.

Fanxu Meng, Hao Cheng, Ke Li, Huixiang Luo, Xiaowei Guo, Guangming Lu, and Xing Sun. Pruning filter in filter. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 17629-17640. Curran Associates, Inc., 2020.

Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu, and Paulius Micikevicius. Accelerating sparse deep neural networks. arXiv preprint arXiv:2104.08378 , 2021.

Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, and Jan Kautz. Importance estimation for neural network pruning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 11264-11272, 2019.

Xuefei Ning, Tianchen Zhao, Wenshuo Li, Peng Lei, Yu Wang, and Huazhong Yang. Dsa: More efficient budgeted pruning via differentiable sparsity allocation. In European Conference on Computer Vision (ECCV) , pages 592-607. Springer, 2020.

Jeff Pool and Chong Yu. Channel permutations for n: M sparsity. Advances in Neural Information Processing Systems (NeurIPS) , 34, 2021.

Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems , pages 91-99, 2015.

Olivier Giroux et al. Ronny Krashinsky. Nvidia ampere sparse tensor core. https: //developer.nvidia.com/blog/nvidia-ampere-architecture-in-depth/ , 2020.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations (ICLR) , 2015.

Xiu Su, Shan You, Tao Huang, Fei Wang, Chen Qian, Changshui Zhang, and Chang Xu. Locally free weight sharing for network width search. Proceedings of International Conference on Learning Representations , 2021.

Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1-9, 2015.

Yehui Tang, Yunhe Wang, Yixing Xu, Dacheng Tao, Chunjing Xu, Chao Xu, and Chang Xu. Scop: Scientific control for reliable neural network pruning. Proceedings of Advances in Neural Information Processing Systems , 2020.

Carl Yang, Aydın Buluç, and John D Owens. Design principles for sparse matrix multiplication on the gpu. In Euro-Par 2018: Parallel Processing: 24th International Conference on Parallel and Distributed Computing, Turin, Italy, August 27-31, 2018, Proceedings , pages 672-687. Springer, 2018.

Tien-Ju Yang, Andrew Howard, Bo Chen, Xiao Zhang, Alec Go, Mark Sandler, Vivienne Sze, and Hartwig Adam. Netadapt: Platform-aware neural network adaptation for mobile applications. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 285-300, 2018.

Mao Ye, Chengyue Gong, Lizhen Nie, Denny Zhou, Adam Klivans, and Qiang Liu. Good subnetworks provably exist: Pruning via greedy forward selection. Proceedings of International Conference on Machine Learning , pages 10820-10830, 2020.

Zihao Ye, Ruihang Lai, Junru Shao, Tianqi Chen, and Luis Ceze. Sparsetir: Composable abstractions for sparse compilation in deep learning. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3 , ASPLOS 2023, page 660-678, New York, NY, USA, 2023. Association for Computing Machinery.

Yuxin Zhang, Mingbao Lin, Chia-Wen Lin, Jie Chen, Yongjian Wu, Yonghong Tian, and Rongrong Ji. Carrying out cnn channel pruning in a white box. IEEE Transactions on Neural Networks and Learning Systems (TNNLS) , 2022.

Yuxin Zhang, Mingbao Lin, ZhiHang Lin, Yiting Luo, Ke Li, Fei Chao, YONGJIAN WU, and Rongrong Ji. Learning best combination for efficient n: M sparsity. In Advances in Neural Information Processing Systems , 2022.

Aojun Zhou, Yukun Ma, Junnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hongsheng Li. Learning n: M fine-grained structured sparse neural networks from scratch. In International Conference on Learning Representations (ICLR) , 2021.

Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. In International Conference on Learning Representations Workshop (ICLRW) , 2017.

Tao Zhuang, Zhixuan Zhang, Yuheng Huang, Xiaoyi Zeng, Kai Shuang, and Xiang Li. Neuronlevel structured pruning using polarization regularizer. Proceedings of Advances in Neural Information Processing Systems , 2020.

Zhuangwei Zhuang, Mingkui Tan, Bohan Zhuang, Jing Liu, Yong Guo, Qingyao Wu, Junzhou Huang, and Jinhui Zhu. Discrimination-aware channel pruning for deep neural networks. Proceedings of Advances in Neural Information Processing Systems , pages 883-894, 2018.

A Supplementary Material

This Supplementary Material is structured as follows. We provide a formulation of our algorithm in Section B. To investigate the effectiveness of different components of our SUBP, we conduct ablation studies and provide additional experimental results in Section C. In Section D, we provide deployment results on the x86 platform of Intel(R) Xeon(R) Platinum 8260L CPU @ 2.30GHz to further explore the performance of 1 × N sparse on different platforms. Finally, in Section E, we discuss the societal impact of our method.

B Algorithm Formulation

Algorithm 1: Overview of the SUBP method.

1 Input : An L -layer CNN model with weights W = { W i | 1 ≤ i ≤ L } ; block binary mask matrices M = { M i j,k ∈ { 0 , 1 }| 1 ≤ i ≤ L, 0 ≤ j ≤ C i +1 N -1 , 0 ≤ k ≤ C i -1 } ; indices of activate blocks with the top scores T ; indices of regrow blocks based on importance sampling G ; target prune rate p ; initial regrowing factor δ 0 ; importance balance coefficient λ ; sampling attention balance factor τ ; training epochs T total; start and end epoch in the pruning-regrowing stage t s , t e ; training set D ; 2 Output : A sub-model satisfying the target prune rate p , its optimal weight values W ∗ and binary mask M ∗ ; 3 Randomly initialize the model weights W ; 4 Initialize { M i j,k | ∀ i, ∀ j, ∀ k } to 1 ; 5 Reformat W to B according to Section 3 ; 6 for each training epoch t ∈ [ T total ] do 7 Sample a mini-batch from D and update the model weights W ; 8 if t s < t ≤ t e then 9 Reset { M i j,k | ∀ i, ∀ j, ∀ k } to 1 ; Compute the importance score S of block by Eq. 4 ; Get the indices of activate blocks with the top scores by Eq. 5 ; Prune the bottom-ranking block by set { M i j,k | k / ∈ T i j } to 0; Compute the importance sampling probabilities by p i j,k = exp ( S i j,k τ ) / ∑ m/ ∈T i j exp ( S i j,m τ ) ; Compute the regrowing factor by Eq. 6 ; Get the indices of regrow blocks based on importance sampling by G i j = Multinomial ( { p i j,k | k / ∈ T i j } , δ t C i ) without replacement ; Regrow the blocks by resetting { M i j,k | k ∈ G i j } to 1 ;

C Ablation Analysis

In Table 6, we investigate the effectiveness of different design choices in our block regrowing stage. All the experiments are based on the TinyImageNet with ResNet18(1×32). Compared to the random

baseline with 57.0% top-1 accuracy, our SUBP achieves consistent improvement under the different settings.

We find that regrowing factor δ 0 significantly impacts the final quality of the model. Intuitively, a larger regrowing factor can provide a more extensive sampling space during training and retain the model's capacity to a greater extent. However, a sizeable regrowing factor may also cause drastic sub-model structure changes, affecting stability during training. As shown, the accuracy is improved by 0.8% as the δ 0 increases from 0.1 to 0.2. When δ 0 increases again, the model's accuracy drops until δ 0 is the full model size. This suggests that the relationship between the regrowing factor and the final quality of the model is varied, and selecting an appropriate regrowing factor in specific circumstances can improve the final quality.

We also investigate the decay scheduler for the block regrowing stage. We compare several decay schedulers, including default (Eq. 6), constant, linear, and cosine. The experiments show SUBP has good robustness to different decay schedulers, as default, linear, and cosine decay schedulers all show similar performance. With a decay scheduler, the sampling space can be gradually decreased, and the sub-model under training can converge stably.

To achieve a fairer and fuller comparison with previous methods, we conduct ablation study with the same epoch and different block sizes. It can be seen in Table 7 that both longger training epochs and finer pruning granularity can improve the final results. We can weight trade offs in practical applications.

D Deployment on x86 Platform

0.125

0.25

0.375

0.5

0.625

0.75

0.875

Pruning Rate

p

30

55

80

105

130

155

180

Network Latency (ms)

Dense

1x32(Uniform)

1x16(Uniform)

1x8(Uniform)

1x32(Non-Uniform)

1x16(Non-Uniform)

1x8(Non-Uniform)

0.125

0.25

0.375

0.5

0.625

0.75

0.875

Pruning Rate

p

20

45

70

95

120

145

170

Network Latency (ms)

Dense

1x32(Uniform)

1x16(Uniform)

1x8(Uniform)

1x32(Non-Uniform)

1x16(Non-Uniform)

1x8(Non-Uniform)

In order to further explore the performance of 1 × N sparse DNNs on different platforms, as shown in Fig. 5, we also conducted corresponding experiments on the x86 platform of Intel(R) Xeon(R) Platinum 8260L CPU @ 2.30GHz and obtain the similar results in general: 1) The gain of vanilla convolution in multithreading scenarios is much greater than that of 1 × N sparse convolution. 2) The

inference speed of uniform 1 × N is slightly faster than that of non-uniform in the case of multithreading, indicating the importance of workload balance again. However, unlike the performance on the arm platform of Apple M1 Pro CPU @ 3.20GHz, the 1 × N sparse DNNs are significantly accelerated when N is set to 16 and 32 on the Platinum 8260L CPU @ 2.30GHz. We can also notice that in most cases, N=32 achieves a fast inference speed.

E Societal Impact

Our method can reduce the computational overhead of training and inferencing stages while achieving satisfactory accuracy on modern CNN models. This can facilitate the application of CNN models on edge devices and is of high value for the community and society to realize Green AI. At the same time, our 1 × N sparse DNNs are based on new sparse operators, which can promote the progress of related hardware and algorithms to a certain extent.

Peer reviews:
comments: SUMMARY:
This paper proposes a new pruning approach to train a uniform 1×N sparse structured network from scratch, dubbed as SUBP. SUBP pruned blocks according to block angular redundancy. The blocks are pruned in a uniform manner to make full use of multithread parallelism computation. During the training process, the pruned blocks can be regrown through importance sampling. They conduct experiments across various CNN architectures to show the superiority of  SUBP.

STRENGTHS:
- This paper is well-written and its motivation is reasonable.
- Code seems to be available, and it'll be helpful if the authors could open-source the work if the paper is accepted.
- They deploy the pruned model in both the arm platform and x86 platform to get actual inference speed. These results well-validate that uniform pruning methods are faster than non-uniform pruning methods.

WEAKNESSES:
-  The authors are encouraged to conduct experiments on other tasks, not just classification.
-  The authors only report the FLOPs in Table 2, the authors are encouraged to report the training time.
-  Compared with method CHEX, the improvements in accuracy are limited.

QUESTIONS:
- Please see the weaknesses part.
- Why the inference latency of the sparse model is lower than the inference latency of the dense model in low pruning rates?

LIMITATIONS:
As discussed in Section 5, there are still missing experiments, including applying  1×N sparsity on other types of DNNs like RNN, transformer, and other tasks including object detection, natural language processing, etc.. The generalization on downstream tasks still needs to be verified.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-27

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSUBP trains a uniform 1×N sparse CNNs from scratch, with three characteristics: (1) a periodic block pruning and regrowing technique via importance sampling, (2) a pruning criterion based on angular redundancy across blocks,  and (3) a uniform 1×N sparse pattern for multithreading acceleration. The authors also provide latency results on hardware platforms.\n\nSTRENGTHS:\n1. The method achieves competitive performance against many existing methods. \n\n2. SUBP trains a uniform 1×N sparse CNNs from scratch.\n\nWEAKNESSES:\n1. Novelty concern. This paper uses a prune-and-grow idea to train a sparse model from scratch, which is been well-studied by many papers in sparse training [1-2]. DMCP and CHEX also don’t require pre-training. Work [3] studied how to prune the network from scratch with a grow-and-prune methodology. Besides, the 1xN pruning technique needs compiler optimization (like TVM) to achieve real inference acceleration. Therefore, the acceleration part of this work highly relies on TVM.\n\n2. The performance improvements are marginal. In Table 1, it’s hard to tell if SUBP is better than CHEX. More important, CHEX uses channel pruning, which could achieve a faster acceleration rate when the pruning ratio is the same. From my point of view, CHEX is basically faster and has the same performance in terms of accuracy. DMCP (channel pruning) is only trained 150 epochs, which is also hard to tell if SUBP has better performance (69.0%,150 epochs vs 69.7%, 250 epochs). Overall, when compared with many SOTA methods, SUBP could not achieve a faster acceleration rate or has an obviously better performance in terms of accuracy. Table 2 has the same problem with many outdated baselines. The authors should provide error bars when the accuracy improvements are marginal.\n \n3. The paper is written poorly. Sections 1 and 3 should be organized better, and section 3 is not clearly written. Table 1 first appeared on Page 6 (section 3) and was first mentioned at the end of Page 7 (section 4). Table 2 has an error with vertical lines exceeding the table. Also, the font size of the figures in this paper is very small, especially in Figure 2. All these things make this paper hard to read.\n\n4. There is no ablation study to show the effectiveness of each components of SUBP.\n\n[1] Rigging the Lottery: Making All Tickets Winners, ICML 2020.\n\n[2] MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edge, Neurips 2021.\n\n[3] Effective Model Sparsification by Scheduled Grow-and-Prune Methods, ICLR 2022.\n\nQUESTIONS:\nPlease refer to “Weaknesses\".\n\nLIMITATIONS:\nPlease refer to “Weaknesses\" for limitations. The authors adequately addressed the potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new fine-grained structured pruning method, which can train from scratch to reduce the training cost and prunes the weights in a uniform manner to handle the load imbalance problem. During pruning, the method uses block angular redundancy as the criterion to discard the redundant blocks and allows pruned blocks to regrow to the network based on the importance sampling to search for a better sparse pattern. Experiments show that the SUBP method can achieve a significant accuracy improvement compared with other methods under the same FLOPs constraint.\n\nSTRENGTHS:\n- The paper is well-written.\n- The proposed SUBP method can greatly improve the model accuracy than kernel pruning methods under a similar pruning ratio.\n- The authors propose uniform pruning to handle the load imbalance problem and show the real speedup on the CPU to evaluate the effectiveness of this method.\n\nWEAKNESSES:\n- Experiments are insufficient. The authors only evaluate the proposed method on the classification task. It is better to discuss the generalization of the method and evaluate on different tasks.\n- The paper lacks a comparison of accuracy with the non-uniform block pruning method. Figure 4 shows that the speedup of the uniform 1xN method is limited. If there is an accuracy drop, I wonder if it is worth adding the uniform constraint at the cost of such a model accuracy loss.\n- Experiments only compare with the kernel pruning methods and lack comparisons with other block pruning methods, e.g. PatDNN [1] and 1xN pruning [2].\n- This paper only focuses on CPU, but GPU on mobile is also important. For example, the implementation of the algorithms related to autonomous driving currently heavily relies on Orin GPUs. It is better to analyze the efficiency of this method on GPU, and the uniform pruning method should achieve better speedups.\n\n[1] Wei Niu, et al. PatDNN: Achieving Real-Time DNN Execution on Mobile Devices with Pattern-based Weight Pruning, ASPLOS, 2020.  \n[2] Mingbao Lin, et al. 1xN Pattern for Pruning Convolutional Neural Networks, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.\n\nQUESTIONS:\n- If use the same data augmentation and hyper-parameters to train the dense model with the same 250 epochs, what’s the dense model accuracy? Because hyper-parameters and data augmentation are different from other methods, the comparison is unfair. If you can compare the accuracy loss of different methods compared with their dense baselines, it may better demonstrate the effectiveness of SUBP.\n- Is this method effective on vision transformers? Transformer models have been widely used in various computer vision tasks.\n- Does the regrow method apply to the experiments in Table 1, including the l1 norm and BPAR?\n\nLIMITATIONS:\nThe author discussed the limitation of the method and the societal impacts. Other limitations of my concern can refer to the weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
Structural weight pruning methods have gain popularity due to the hardware friendly pattern. This work improves 1xN pruning with prune-and-grow and uniform block pruning. Evaluation shows outperforming accuracy results at similar levels of FLOPs than prior work.

STRENGTHS:
- This paper presents a new contribution on block-wise pruning with soft (prune-and-grow) in channel-wise uniformly.
- The evaluation shows potential significance of the proposed block pruning method in achieve accuracy with reduced FLOPs.

WEAKNESSES:
- The paper could be improve in clarity and sufficient implemantion details to help readability.

QUESTIONS:
- Regarding the key design idea of uniform block pruning, what is being done in Section 3.1 and how different compared with prior work of block pruning? Figure 2 could be modified to better illustrate and/or an algorithm description of the steps. Is it every output channel has the same number of pruned 1xN blocks? 
- In Section 3.2, block pruning vis angular redundancy could be better positioned in the overall pruning.
- What are the sparsity ratios used in Table 2? While showing FLOPs is informative, it is not the end product, and information on the sparsity ratio could help evaluation the proposed method.

LIMITATIONS:
Not adequate. From the evaluation, the proposed method, among many other pruning work, significantly increase the training epochs. The negative impacts from more training costs need to be discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-10