SUBP: Soft Uniform Block Pruning for 1$\times$N Sparse CNNs Multithreading Acceleration
Authors: Jingyang Xiang
Siqi Li
Jun Chen
Guang Dai
Shipeng Bai
Yukai Ma
Yong Liu
Venue: NeurIPS 2023 poster (2023)
Keywords: Soft Uniform Block Pruning
Block Angular Redundancy
Hardware Acceleration
TL;DR: We propose a new block pruning criterion based on angular redundancy and a soft block pruning approach to a train 1xN pruned network from scratch.

Abstract: The study of sparsity in Convolutional Neural Networks (CNNs) has become widespread to compress and accelerate models in environments with limited resources. By constraining N consecutive weights along the output channel to be group-wise non-zero, the recent network with 1$\times$N sparsity has received tremendous popularity for its three outstanding advantages: 1) A large amount of storage space saving by a \emph{Block Sparse Row} matrix. 2) Excellent performance at a high sparsity. 3) Significant speedups on CPUs with Advanced Vector Extensions. Recent work requires selecting and fine-tuning 1$\times$N sparse weights based on dense pre-trained weights, leading to the problems such as expensive training cost and memory access, sub-optimal model quality, as well as unbalanced workload across threads (different sparsity across output channels). To overcome them, this paper proposes a novel \emph{\textbf{S}oft \textbf{U}niform \textbf{B}lock \textbf{P}runing} (SUBP) approach to train a uniform 1$\times$N sparse structured network from scratch. Specifically, our approach tends to repeatedly allow pruned blocks to regrow to the network based on block angular redundancy and importance sampling in a uniform manner throughout the training process. It not only makes the model less dependent on pre-training, reduces the model redundancy and the risk of pruning the important blocks permanently but also achieves balanced workload. Empirically, on ImageNet, comprehensive experiments across various CNN architectures show that our SUBP consistently outperforms existing 1$\times$N and structured sparsity methods based on pre-trained models or training from scratch. Source codes and models are available at \url{https://github.com/JingyangXiang/SUBP}.

SUBP: Soft Uniform Block Pruning for 1xN Sparse CNNs Multithreading Acceleration
Jingyang Xiang'
Siqi Li
Jun Chen
Shipeng Bai!
Yukai Ma¹ Guang Dai
Yong Liu
APRIL Lab, Zhejiang University, Hangzhou, China State Grid Corporation of China
SSIT AI Lab, Shaanxi, China
{ dinggnggxiang ang, 18q4747,jume shi
shi pengbai, yukaima edu. cn guang gdai @gmail com, yong. yongliuGiipc.c wzj.:iiipp............. edu. cn
Abstract
The study of sparsity in Convolutional Neural Networks (CNNs) has become widespread to compress and accelerate models in environments with limited re- sources. By constraining N consecutive weights along the output channel be group-wise non-zero the recent network with 1 XXN sparsity has received tremen- dous popularity for its three outstanding advantages: space saving by Block Sparse Row matrix. 2) Excellent performance at a high sparsity. 3) Significant speedups CPUs with Advanced Vector Extensions. Re- cent work requires selecting and fine-tuning I X sparse weights based on dense pre-trained weights, leading to the problems such as expensive training cost and memory access, sub-optimal model quality, as well as unbalanced workload across threads (different sparsity across output channels). To overcome them, this pa- per proposes a novel Soft Uniform Block Pruning (SUBP) approach to train a uniform xN sparse structured network from scratch. Specifically, our approach tends to repeatedly allow pruned blocks to regrow to the network based on block angular redundancy and importance sampling in a uniform manner throughout the training process. It not only makes the model less dependent pre-training, reduces the model redundancy and the risk of pruning the important blocks per- manently but also achieves balanced workload. Empirically, ImageNet, com- prehensive experiments across various CNN architectures show that our SUBP consistently outperforms existing I XxN and structured sparsity methods based on pre-trained models or training from scratch. Source codes and models are available at hhttps:://gtthhhe com/ Jingyangxiiagg/SUUB
A large amount of storage
Introduction
In recent years, convolutional neural networks (CNNs) have achieved great success in image clas- sification [56] 22). object detection 21 16), semantic segmentation [17] 4 and other fields. The remarkable performance owes to the deeper and wider architectures, also leading to the prohibitively expensive computational cost and colossal memory footprint. Although some efficient architectures are proposed, such as residual connection [22]. inception module (58), etc., it is still difficult to deploy the state-of-the-art CNNs the available CPUs embedded devices with limited resource. Therefore, the network pruning is emerging by pruning the redundancy in CNNs. Due to the appealing perfor-
Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Unstrutuure qparso
Structured sparse liter
NM structored
Jaft struttere
Deplay
E
data
Indices
data
data
Indicao
dada
Indper
Indious
(a) Unstracttured
(V) Structured fitter
NiS weight
lult weight
pruning
Figure 1: Four mainstream types of pruning in the literature. (a) Unstructured weight pruning removes individual weights at arbitrary locations. (b) Structured filter pruning removes entire convolutional of M consecutive weights along input channels to be non-zero. (d)1 xN weight pruning (I x case) constrains N consecutive weights along the output channel be group-wise non-zero.
filters. (c) N:M weight pruning (2:4 case) requires at most N
mance, it has received extensive attention from industry and academia. necessary maintain the model with small size, low memory footprint, low computational cost and high inference speed.
According to the pruning granularity, most existing works about network pruning mainly focus weight pruning [T0] 19 34] and filter pruning [64] 40 25 24 446. Weight pruning deletes the weight of filters directly, which may always result in the unstructured sparsity of filters. The expensive memory accesses is also less efficient in saving memory usage and computational cost, since the unstructured model needs large number of indices record the positions of the reserved weights and can't 't take full advantage of Advanced Vector Extensions (AVX), Single Instruction Multiple Data (SIMD) and poorly utilizes memory caches [63] In contrast, filter pruning tends prune at the level of filter or even layer. Since filter pruning still preserves the original convolution structure, enables the model with structured sparsity and more efficient memory usage. Therefore, filter pruning can take full advantage of the nigh-performance computing library i.e. Basic Linear Algebra Subprogram (BLAS) achieve apparent acceleration and is more advocated in accelerating the networks. More recently, the development of hardware and operators has given rise to new pruning patterns. The most famous work is N:M fine-graines pruning (55), where N out M weights are zeros for every continuous M weights along input channels. It also can be seen as a special pattern of weight pruning. Currently, this pattern achieves acceleration only in the case of 2:4. However, it is impossible to be utilized other types of devices since the instructions of sparse Matrix Accumulate (MMA) are specially designed for NVIDIA Ampere Core (55). Not to mention, in realistic deployment scenarios, GPUs on mobile and embedded devices not always accessible. and such GPUs are more difficult to be satisfied.
Above all, how to retain the performance and achieve realistic acceleration on mobile CPUs becomes
challengeable but valuable problem. order solve this issue, Lin er al. (65] proposed novel pattern of xN weight pruning with its merits in realizing both high-performing accuracy and apparent CPUs acceleration for practical model deployment. The xxN pruning pattern provides an intermediate granular level for network pruning, which is coarser as compared to the fine-grained weight but finer as compared to the coarse-grand grauiee filter. Fig. d shows an example of IxN pruning pattern that satisfies N=4, the core distinction lies in that I 1xN pruning consists of N consecutive weights along the output channel to be group-wise non-zero. These consecutive weights can be stored continuously in the memory cache and the convolution with the inputs can proceed using block-wise vectorized operation in parallel thanks to AVX and SIMD. The indices memory of the weight positions also benefit from Block Sparse Row (BSR) matrix and be greatly saved.
criterion in non-uniform manner. On the one hand, it reduced the capacity of original model and thus harmed the performance. On the other hand, it left the blocks redundancy untouched and always caused unbalanced workload across threads. What's more, it still followed traditional pre- pre-training, pruning
However, Lin al. 65 permanently pruned blocks based on

---

and fine-tuning (PPF) pipeline, which depended on pre- pre-trained model and still suffered from the expensive pre-training burden.
To address the above-mentioned limitations, we propose a novel block pruning approach named Soft Uniform Block Pruming (SUBP) to obtain high accuracy sub-model without PPF pipeline. In contrast to the traditional pruning approaches that non-unifornly and permanently remove blocks, we prune and regrow the blocks uniformly via importance sampling, which allows the pruned blocks to be recovered and balanced workload across threads. From intra-layer's perspective, we propose a new Block Pruning criterion by taking Angular Redundancy (BPAR) into account. BPAR identifies the angular redundancy between blocks, so we can prune blocks with redundancy, rather than those with "relatively less" importance. With only one training procedure from scratch, our obtained sub-models yield better accuracy than the previous methods under the similar FLOPs constraints.
To sum up, the contributions of this paper are highlighted follows:
We propose a novel block pruning approach named Soft Uniform Block Pruning (SUBP) with three appealing characteristics: (1) a periodic block pruning and regrowing technique via importance sampling, (2) a pruning criterion based on angular redundancy across blocks, and (3) a uniform X sparse pattern for multithreading acceleration.
Our approach trains a uniform I XN sparse CNNs from scratch, effectively reducing the training cost and achieving better inference latencies in multithreading scenarios, since it circumvents the expensive PPF pipeline and balances workload across threads.
Extensive experiments on the large ImageNet dataset have demonstrated the effectiveness of our SUBP under different FLOPs constraints. SUBP obtains consistent accuracy im- provement across various N and networks, achieving a better trade-off between accuracy and inference latencies. For example, ResNet50 (1 x 16) model yields 4x FLOPs reduction while still achieving 76.3% top-1 accuracy and suppressing the previous results.
2 Related Work
Fully connected operators are widely used in various types of neural networks, and they can be mathematically represented by one matrix multiplication. The natural idea is that all elements of a matrix are not equally important. Removing unimportant elements from the fully connected operators not only reduces the size and the amount of computation, but also potentially improves the generalization performance of the model. Weight pruning (Fig. (Fig.Ta)) and filter pruning (Fig. (b)) are traditional pruning methods. There are also some special pruning methods that require a high degree of integration with hardware or operators as shown Fig. (,,d). what follows, we will briefly review some related works.
Weight Pruning. Weight pruning is one of the most widely studied model pruning methods, which removes individual weights at any position of the network. The study of weight pruning could date back to optimal brain damage (33] and optimal surgeon (20). which prune weights based on the Hessian within the loss function. Previous studies removed unimportant weights by using gradient 34 momentum [0) magnitude [19. etc. Han er al. 19 proposed to discard the small weights whose values are below the threshold through an iterative method. Recent some works also payed attention to unstructured sparsity in an adaptive training manner. Ding et al. [10] gradually zeroed out the redundant weights by categorizing weights into two parts and updating them according to different rules. Frankle el al. [33 presented an algorithm to identify subnetworks that were capable of training effectively. Although weight pruning can maintain most of the accuracy of the model with high sparsity, is difficult to leverage the existing high-efficiency BLAS libraries in practice to accelcrate on general hardware due to its irregular weight distribution. Therefore, weight sparsity pruning is rarely used in practical applications.
Filter Pruning. Filter pruning gains achieve noticeable speedup on general hardware after pruning. The filter importance and the pruned network structure are the two most important issues widely studied by researchers. Typical works solve the former issue by devising certain criterion to
measure the importance of filters, including output activation [64], scale factor amplitude of Batch
Normalization layer [46), ranks of feature map (40), norm or geometric median of filters 244 25).
etc. To be specific, he el al. (25) leveraged geometric median to prune filters with redundancy. As for the latter, most works are based on rules of thumb 77 24] or evolutionary algorithms [1].
3

---

An illustration of our SUBP method, which optimizes the weight values and regrows the removed blocks in one training pass from scratch jointly. In SUBP, both retained and regrown blocks are active, participating in the training iterations. After the last iteration, SUBP exports the pruned model and optimized block weights.
Figure 2:
reinforcement learning 26], meta learning 45 and other methods 9 to predict the layer- wise sparsity. For instance, he ef al. (26) applied reinforcement learning to compression and acceleration models on mobile devices automatically. However, filter pruning removes entire convolution filters, which may damage the information learned by the network and cause serious accuracy degradation with high pruning rate. Therefore, filter pruning is still difficult to apply in practical tasks.
Special Pruning, In order to achieve the maximum balance between accuracy and model performance, researchers have proposed various special pruning types [4] 50 12). These convolution modes often require the combination of special operators or hardwares. Supported by the NVIDIA Ampere Core, N:M sparsity has gained tremendous attention due to its attractive storage and computation efficiency. Asit et al. [50] and Pool et al. [53] followed traditional PPF pipeline to implement N:M sparsity. Although retraining can improve the accuracy of the N:M sparsity network, the pre-training-baa based approach still incurs expensive training costs, which hinders the deployment of N:M sparsity techniques. To address this, zhou el al. [66] proposed a Sparse-refined straight- estimator (SR-STE) to train N:M sparsity network from scratch. Zhang el al. (65] characterized N:M sparsity as a combinatorial problem and assigned each combination a learnable score to obtain the best subset. Apart from these, researchers have also turned their attention to 1 xxN sparse networks, which can be accelerated on CPUs. Lin et al. [42] proposed I XxN pruning pattern for CNNs firstly, which achieved better accuracy and speed trade-off than both weight and filter pruning.
3 Methodology
3.1 Uniform 1 1xN Block Pruning
We start by introducing symbols and notations in this subsection. Without losing generality, we assume that a CNN has layers. We parameterize the tensor connection of CNN with (Wi E RG11xC, Kt | < i < L}. where C., C++1, Kth and K," represent the numbers of input channels, output channels, kernel height and kernel width of i-th layer, respectively.
A XxN pruning pattern partitions the whole W into collection of small blocks. Considering the W in the i-th layer, the xxN pruning pattern can achieved by partitioning W into a collection of C col- col-groups and then further partitioning N; into a collection of C.++ row- row-groups. Consequently,
N each block is I XN matrix along the input channel, which includes N consecutive output kernel with the same input channel index. We denote the matrix block set as (Bj NR.Ixx xK |1 S
i S L. < < N
Ca+1 1,0 k C. 1}, to stand for the W J.N:(i+1)-NN.::: Based on this
partition pattern, the basic pruning granularity of xN sparsity falls into these blocks. According to network pruning which can be implemented by imposing a mask M' upon W we introduce
(M (0, 1}|1 < L. o C.11 1,0 < k < C. 1}. to define the objective function of

---

pre-existing work:
ET((B) M).10 Sj C++1 - 1,0 - 1)), s.l.
[
arg max
p (1)
K
M
where F(-) measures the importance of its input, K Cisl C₁ and p is the expected prune rate for the model.
To reduce the latency of CNNs, multi-core devices are often adopted in most practical scenarios. On single-core CPU, the most important aspect is the continuity and locality of memory access and calculation in order to fully utilize the cache and vectorization unit. However, nearly all modern processors have multiple cores, and computation can benefit from multithreading parallelism. Even though the continuity and locality of memory access and calculation are also important to multi-core CPU, the most important aspect of CPÚ multi-core computing is achieve balanced workload to fully utilize the computing power of each core and improve computational efficiency. As can be seen from Fig. 3 if the sparsity among different output channel blocks keeps the same, the workload across different threads will be the same.
However, it easy know that the condi- tions in Eq. (I) often lead to different spar- sity levels among different output channel blocks, e.g.. M lo = 1 and ||M llo = G. which will cause workload imbal- ance [60] among threads, degrade perfor- mance and waste resources during multi- core execution. Therefore, to overcome these issues, we propose a uniform I xxN block pruning type the basis of the anal- ysis in above, which equivalent to con- straining the sparsity levels among different output channel blocks to be the same. In another word, the uniform 1 XxN block prun- ing is a special case xN block pruning. Therefore, define the objective function of uniform xN block pruning as:
Vecteried Operstion
Throel?
Iepart
Spars Wedgl
Oapet
loading threads
X
BSA Formet
Inclow
Vertorteed Opereties Thrwelt
Figure 3: Balanced workload across threads.
L
M
arg max M
F([B), M, 1)), s.t. =
C2
1=
3.2 Block Pruning via Angular Redundancy
To get rid of the constraints in the norm-based criterion, we propose a new block pruning criterion by taking block angular redundancy into account and name it as BPAR here. The central idea of angular redundancy as follows: given a non-zero vector E Nx and non-zero vector W E Xx1 we have
W WTx=-aw77 < W1, W2 >=
W1, Wi,W0 W2 >= a
W ll W2ll
where <.. denotes the angle between vectors. In particular, we define the 0, According to the above analyses, even though I xxN block may be relatively important, this block still has angular redundancy if there exists another block with similar orientation angle. Similarly, a relatively less important xN block can also map the vectors to a meaningful space and extract valuable information if its orientation angle is very different from others.
In order to simplify the explanations, firstly, we vectorize the representation of B), NXIRR IR K Then, compute the importance score S of our block as:
Qi
C, m==
CosineSSim(()
52 j.k
Cosine.Sim(Q St 5mmm)
J.m
m
5

---

where À is a balanced hyper- parameter and CosineSim denotes cosine distance between two vectors. Based on Eq. 44) it is obvious that unlike previous criterion solely based on norm, our approach might retain some blocks with smaller norm but larger angular differences in comparison with other blocks. Simultaneously, it may discard the blocks with larger norm but smaller angular differences.
3.3 Soft Uniform Block Pruning
Most of the previous pruning 40 pressed CNNs in a hard manner. However, once filters or weights are pruned, they can not be recovered, which limits the model capacity and leads to unacceptable accuracy degradation.
works followed traditional PPF pipeline, which com-
To rectify the aforementioned limitations, here propose novel soft block pruning approach i.c. Soft Uniform Block Pruning (SUBP) to obtain high accuracy sub-model without a pre-trained super- model or recover the pruned blocks through an importance sampling approach, which enables the compressed network to have a large model capacity and thus achieves higher accuracy than others. As shown in Fig. the details of our SUBP can be divided into two stages in what follows.
the pruned model. In contrast to the traditional hard pruning approaches,
1) Block Pruning Stage: Firstly, we init all elements in Mj,x to 1. For the i-th layer, we use Eq. (4) to evaluate the importance of each block and identify the set of important blocks to retain as:
Tj ArgTopK (15) S k C.)) E R [C((-~))]
(5)
which gives the indices of blocks with the top [C;(1 PP]] scores of Bj. Then, we prune the bottom-ranking block by zeroizing { (Mi.elk lk & }-
2) Block Regrowing Stage: To determine the blocks to regrow, we introduce an importance sampling strategy based on the block importance score. We compute the importance of sampling probabilities
$100
/EmęT; exp $i where the temperature T is to balance the sampling
by PI,a- exp
attention between the different blocks. Then we select the regrow block indices by performing importance sampling 9j = Multinomial (P) & T; 5,0,) without replacement based on the regrowing factor & at t-th epoch and reset {M) 1k E 9j} to
To compute the regrowing factor dr, employ gradually schedule [67 to gradually reduce the regrown blocks so that the subnet can converge to the target uniform 1xN sparsity stably at the end of training. Specially, the regrowing factor at /-th epoch is computed as:
P. 50 0,
i t
8,
to < stst. te
(6)
to <t
where do is the initial regrowing factor, la and le denote the start and end of the training epochs in the block pruning-regrowwng stage respectively. Since the model is trained from seratch and the pruning and regrowing decision based on the weights may not be sufficient enough, therefore, we don't t prune blocks in the early stage of training. When te, the blocks will stop regrowing and we finish pruning the block. The remaining blocks are considered to be the best candidates in the end.
4 Experiments
4.1 Experiment Settings and Implementation Details
In this section, we evaluate our SUBP on the largescale dataset ImageNet with the representative networks, including ResNet18. ResNet34, ResNet50 and MobileNetV1. We implement SUBP using the PyTorch framework and NVIDIA RTX 3090 GPUs. The SGD optimizer with momentum of 0.875 and a weight decay of 3e-5 adopted. We train all compared networks for 250 epochs with mini-batch size of 512 and an initial learning rate of 0 which is linearly increased to 0.512 during the first epochs and then decayed previous methods [18 45 we also use label smoothing with the factor 0.1 and apply the standard data augmentation. To keep our method simple and generic, the hyper-paramette (T, X, 6p. Ls, 1.) in above are set to (1.0, 1.0, 0.2, 10, 180) and kept constant in experiments. By adjusting appropriate
by the cosine learning rate schedule. Same as

---

Table 1: Performance comparison of our BPAR prune criterion against L norm. The experiment is conducted using MobileNetV1 and ResNet50 with the pruning rate = 50% on ImageNet dataset. We test our BPAR on three cases, which N is set 8, 16 and 32 respectively.
MobileNetV1
ResNet50
(p =p= 50%)
(p 50%)
Accuracy
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Origin
71.15
89.83
71.15
89.83
77.00
93.65
77.01
93.65
Weight Pruning
70.76
89.59
70.76
89.59
77.09
93.61
77.09
93.61
Filter Pruning
65.35
86.26
65.35
86.26
75.38
92.52
75.38
92.52
Prune Criterion
41 norm
BPAR
E norm
BPAR
Accuracy
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
x2 Pattern
70.28
89.37
76.65
93,47
x Pattern
70.05
89.06
76.51
93.24
8 Pattern
69.91
89.08
70.55
89.42
76.15
93.13
76.70
93.51
X Pattern
69.56
88.93
70.30
89.35
76.25
93.08
76.52
93.20
x32 Pattern
69.54
88.80
70.03
89.23
75.96
92.95
76.31
93.18
Table 2: Results of R8sNet18, ResNet34, ResNet50 and MobileNetV1 ImageNet dataset. "PT": require pre-training. "SR": sparse ratio.
Method
FLOPs
Top-1 Epochs
Method
PT FLOPs
SR
Top-1
Epochs
ResNet- 18
ResNet-50
PFP 39
1.27G
43.8%
67.4%
270
SSS 33
X
2.3G
38.8%
71.8%
100
SCOP 59
V
1.10G
39.3%
69.2%
230
TAS 11
X
2.3G
43.5%
76.2%
240
SFP 24
V
1.04G
47.6%
67.1%
200
GAL
43
/
2.3G
16.8%
72.0%
150
FPGM 25
V 1.04G
47.6%
68.4%
200
Hrank
40
2.3G
36.7%
75.0%
570
DMCP 18
X 1.04G
17.6% 69.0%
150
Taylor 51
2.2G
44.3%
74.5%
CHEX 29
1.03G
38.7% 69.6%
250
C-SGD 6
2.2G
42.8%
74.9%
SUBP x
1.03G
44.1% 69.9%
250
SCOP 59
V
2.2G
42.8%
76.0%
230
SUBP(Ix32)
1.03G
44.1%
69.7%
250
DSA S2
2.0G X
74.7%
120
ResNet-34
CafeNet 57
X 2.0G
27.8%
76.9%
300
Taylor 51
2.8G
21.1%
72.8%
CHEX 29
X 2.0G
35.8%
77.4%
250
SFP 24
2.2G
49.
71.8%
200
SUBP(1 x16
X 2.0G
45.5%
77.6 %
250
FPGM 25
2.2G
49.1%
72.5%
200
SUBP(1x32)
2.0G X
45.5%
77.4% %
250
GFS 62
2.1G
32.5%
72.9%
240
SCP 32
X 1.9G
75.3%
200
DMC 15
2.1G
72.6%
490
Hinge 38
1.9G
74.7%
NPPM 14
2.1G
73.0%
390
AdaptDCP
69
1.9G
51.5%
75.2%
210
SCOP 599
2.0G
45.6%
72.6%
230
LFPC 23
1.6G
74.5%
235
CafeNet
5
1.8G
21.1%
73.1%
300
ResRep
8
1.5G
75.3%
270
CHEX 29
2.0G
29.2%
73.5%
250
Polarize
68
1.2G
74.2%
248
SUBP(1x16
2.0G
43.8%
73.7%
250
DSNel
36
1.2G
74.6%
150
SUBP(Ix32)
2.0G
43.8%
73.6%
250
CURL
47
1.1G
73.8%
73.4%
190
MobileNetVV
DMCP
18
X
1.1G
43.6%
74.1%
150
0,75x
325M
38.1%
68.4%
MetaPrune
45
X
1.0G
53.5%
73.4%
160
NetAdapl 61
284M
69.1%
EagleEye 35
/
1.0G
69.4%
74.2%
240
AMC 26
28.5M
70.5%
CaleNet [7
X
1.0G
52.9%
75.3%
300
MctaPruning 45
281M
50.9%
70.6%
320
CHEX 29
X
1.0G
67.1%
76.0%
250
SUBP(I 16
X
279M
40.0%
70.8%
250
SUBP(1 x
X
1.0G
68.3%
76.3%
250
SUBP(1x32)
X
279M
40.0%
71.1%
250
SUBP(1x32)
X
1.0G
68.3%
76.0%
250
parameters for different models and pruning rates, better results should be obtained in general. In this paper, we calculate FLOPss by counting multiplication and addition as one operation as He [22 did.
4.2 Influence of Pruning Criterion
In this subsection, we will investigate the influence of x pruning criterion. Table shows the compared results with respect to ε norm 42 and BPAR. As suggested by [37 48 51 28 the pruning criterion selects the appropriate sub-model on the basis of its importance redundancy, which plays an important role in the traditional PPF approach. Table [ tells us that our BPAR

---

outperforms the norm bumm-asedd method on the ImageNet dataset. For MobileNetVi and ResNet50, our BPAR can achieve consistent accuracy improvements with the same sparsity and inference speedup. On MobileNetV [(1 16), it obtains 0.74% and 0.38% topl-accurayy and topS-aceuracy improvement when BPAR is applied. particular, for pruning pre-trained MobileNetVi BPAR achieves better results for N=8 than & norm for N=2, which indicates 1 xN pruning based on BPAR can obtain better trade-off between accuracy and inference latencies. Similar results can also be observed on the ResNer50 in Table In essence, BPAR explicitly utilizes the relationship and identifies the mutual angular redundancy between blocks, giving rise its superior performance.
4.3 Results on ImageNet
To verify the effectiveness our SUBP, we apply to the heavyweight CNN model (i.e. ResNet (22 with different depths and the lightweight CNN model (i.e. MobileNet) initialized with random weights on ImageNet (5] dataset. Here, BesNet-18/34/55 and MobileNetVI are used as the baseline models and have 1.8/3.7/4. GFLOPs and 572 MFLOPs.
The results Table show that SUBP achieves noticeably higher accuracy than the state-of-the-arr pruning methods under the same FLOPs constraints. For example, our SUBP on ResNet50(1 x 16) with 2x FLOPs reduction achieves 77.6% top-1 accuracy, which is 3.1% 1%, 1.6%, 0.7% and 0.2% higher than Taylor[51], SCOP [59] CafeNet [57 and CHEX (29] respectively. The results on I xx32 sparsity also demonstrate the superiority against the other methods. On the other hand, at the same target accuracy, our SUBP also achieves higher FLOPs reduction. For example, our SUBP achieves 4x FLOPs reduction and 76.0% top-1 accuracy on ResNet50(1 x32) compared to the SCOP, which only yields 1.9x FLOPs reduction. We further observe that SUBP also achieves higher accuracy at fraction of the training cost for MobileNetV1. For instance, SUBP achieves 71.1 % top-1 accuracy on MobileNetV (1 32, 250 epoch) under the 279M FLOPs constraint, which is 0.5% higher than MetaPruing (320 epoch). This is because SUBP can dynamically explore the optimal sub-model in one training pass from scratch, circumventing the expensive PPF cycles.
4.4 Results on Object Detection and Instance Segmentation
To further explore the performance of SUBP in downstream tasks, we conduct experiments on object detection and instance segmentation on the challenging COCO dataset 44 The results are shown in Table and and Table We adopt the classical method Faster RCNN [54] for object detection and Mask RCNN (21) for instance segmentation. We use ResNet50 with different sparse ratios and block sizes as backbone. All the experiments are conducted based MMDetection [I]. Compared to Dense ResNet50, the SUBP can achieve competitive results, which further demonstrates itsrobustness and superiority downstream computer vision tasks.
Table 3: Object detcction results COCO.
Table 4: Instance segmentation results COCO.
Model
Block Size
mAP
Model
Block Size
Box mAP
Mask mAP
F-RCNN WONNN-550(411)
37.4
M-RCNN-RSON R50(4.1G)
38.2
34.7
ARCNN-R50(2.00)
1x
38.4
M-RCNN FRCNN-R50(200C
1x 32
39.2
35.4
FRCNN-R50(2.0G)
1x
38.5
M-RCNN-R5OK R50(2.0G)
1x 16
39.4
35.5
TRCNN-R50(1.00)
Ix
37.1
M-RCNN-RS0N RRN(()0500
1x 32
37.4
33.8
#RCNN-R55(1.00)
1x 37.3
M-RCNN-RSUN R50((.G))
1x 16
37.5
33.8
4.5 SUBP from Pre-trained Models
In order to further investigate the generality property of our approach, apply SUBP to the model initialized with pre-trained weights. For an equitable comparison with other PPF methods, we adopt the pre-trained ResNet18 provided by the torchvision and run SUBP for 120 epochs the previous one did. The results in Table that our SUBP still achieves competitive top- and top-5 accuracy under the same FLOPs compared to the previous state-of-the of-the-art PPF methods. Furthermore, when ResNet1 8 is trained for 90+1 120 epochs, it achieves 69.5% top-1 accuracy, which is only 0.4% lower than training it from scratch for 250 epochs in Table 2

---

Table 5: ResNet18 starting from the pre-trained models on ImageNet dataset. "Epochs" are reported as: pre-training epochs plus all subsequent training epochs needed to obtain the final pruned model.
Model
Method
Params
FLOPs
Top-1
Top-5
Epochs
Baseline
11.7M
188G
69.8%
89.1%
90
PFP
139
6.6M
1.27G
67.4%
87.9%
90+180
SCOP [59
7.1M
1.10G
69.2%
88.9%
90+140
ResNet- 18
SFP 224
7.1M
1.04G
67.1%
87.8%
100+100
FPGM 25
7.1M
1.04G
68.4%
88.5%
100+100
CHEX 29
1.04G
69.2%
90+120
SUBP(1 x 16)
7.1M
1.04G
69.5%
89.0%
90+120
SUBP(1X32)
7.1M
1.04G
69.3%
88.9%
90+120
4.6 Deployment Efficiency Analysis
Jones
Denea balussfam
Lol Loltun
bddunfamm
Loa Unfferr)
Network Latency (ms)
bolaken Dolaka-undee Unifer))
Network Latency
1.0
LeB Un
0.155
0.25
0.375
0.5
0.825
0.75
0.875
30
5.125
025
0.375
05
0.525
0.75
0.875
Pruming Rate P
Pruning Rate p
Figure 4: Network latency comparison between uniform x sparse against non-uniform and dense model with varying N and prune rates. The experiment is conducted using ResNet18 and set the input shape as (4, 3, 224, 224) on the arm platform of Apple M1 Pro CPU 3.20GHz with single thread (left) and two threads (right). Best viewd in colors.
To further explore the acceleration capacity with respect different N and prune rates CPUs-based platforms, we adopt TVM [2] to compile the uniform and non-uniform xxN pruned models. For a fair comparison, we also consider TVM for dense model to obtain baseline latency. We deploy the 1xN pruned models to obtain network latencies on the arm platform of Apple MI Pro CPU @ 3.20GHz. From Fig. 4 observe xN pruned model achieves noticeable latency reductions across various pruning rates and N. For example, when the pruning rate is greater than 0.25, the inference latencies of different are all the better than their dense baseline under the single thread scenario. When the thread num is set to 2, the inference speed of different configurations has been improved. Due the workload imbalance between threads in non-uniform 1 XxN, it always lags behind its uniform counterpart. From Fig.Pe there are also two points worth noting: 1) the improvement of 1 x is not as significant as a vanilla convolution a multithreading context; 2) N=16 achieves faster inference speed than other N on M1 Pro. Therefore, optimizing multithreading inference with 1 xN pruning and selecting appropriate N based on suitable platforms are directions that can further explored.
Limitation and Discussion
Firstly, our SUBP masks weights throughout the training and still needs dense computations. There- fore, while SUBP has advantages over methods that depend on PPF pipeline, there still room for improving its training efficiency. Secondly, there are still missing experiments, including applying IxN sparsity on other types of DNNs like RNN, transformer and other tasks including object detec- tion, natural language processing, efc. We will design corresponding nigh-performance operators to improve the training efficiency SUBP, and explore the performance of X sparsity other types of DNNs and tasks to verify its broader applicability in our future work.

---

Conclusion
Uniform 1xN sparsity an important technique that allows fast inference on multithreading scenarios under multi-core architecture. This paper proposes soft uniform block pruning, SUBP, to efficiently train a uniform xXN sparsity network from seratch. SUBP dynamically adjusts the pruning blocks based on periodic pruning and regrowing process, which prevents the important blocks from being prematurely pruned and keeps model's capacity. We also present a new block pruning criterion named BPAR, which explicitly considers the mutual angular redundancy between blocks rather than "relatively less" importance only. By proposing BPAR based block subset selection approach for pruning and importance sampling based strategy for regrowing, we can obtain sub-model with high accuracy by end-to-end implementation without pre- vre-trannig a large model or requiring extra fine-tuning. Extensive experiments have exhibited our method can effectively reduce the FLOPs and inference latencies while achieving superior accuracy over several SOTAs.
10

---

Acknowledgments and Disclosure of Funding
This work is funded in part by the Key Research and Development Project of Zhejiang Province under Grant 2021
References
[1] Kai Chen, Jiaqi Wang. Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprini vrXiv:/906.070 2019.
[2] Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, Tvm: An automated end-to-end optimizing compiler for deep learning. In Symposium on Operating Systems Design and Implementation (OSDI), pages 578-594, 2018.
[3] Zhuangzhi Chen, Jingyang Xiang. Yao Lu, Qi Xuan, Zhen Wang, Guanrong Chen, and Xiaoniu Yang. Rgp: Neural network pruning through regular graph with edges swapping. IEEE Transactions on Neural Networks and Learning Systems, pages 1-13, 2023.
[4] Ioana Croitoru, Simion-Vlas Bogolin, and Marius Leordeanu. Unsupervised learning of foreground object segmentation. International Journal of Computer Vision (IJCV), 127(9):1279- 1302, 2019.
[5] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 248-255, 2009.
[6] Xiaohan Ding, Guiguang Ding, Yuchen Guo, and Jungong Han. Centripetal sgd for pruning very deep convolutional networks with complicated structure. In JEEE Conference Computer Vision and Pattern Recognition (CVPR), pages
2019.
[7] Xiaohan Ding, Guiguang Ding, Yuchen Guo, Jungong Han, and Chenggang Yan. Approximated oracle filter pruning for destructive enn width optimization. In International Conference on Machine Learning (ICML), pages 1607-1616. PMLR, 2019.
[8] Xiaohan Ding, Tianxiang Hao, Jianchao Tan, Ji Liu, Jungong Han, Yuchen Guo, and Guiguang Ding, Lossless enn channel pruning via decoupling remembering and forgetting. Proceedings of the JEEE/CVF International Conference on Com-puter Vision, 2021.
[9] Xiaohan Ding, Tianxiang Hao, Jianchao Tan, Ji Liu, Jungong Han, Yuchen Guo, and Guiguang Ding. Resrep: Lossless cnn pruning via decoupling remembering and forgetting. In IEEE International Conference on Computer Vision (ICCV) pages 45 4510-4520, 2021.
[10) Xiaohan Ding, Xiangxin Zhou, Yuchen Guo, Jungong Han, Ji Liu, et al. Global sparse momentum sgd for pruning very deep neural networks. In Advances Neural Information Processing Systems (NeurIPS), pages 6382-6394 2019.
[11] Xuanyi Dong and Yi Yang. Network pruming via transformable architecture search. Proceedings of Advances in Neural Information Processing Systems, pages 759-770, 2019.
[12] Erich Elsen, Marat Dukhan, Trevor Gale, and Karen Simonyan. Fast sparse convnets. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14629-14638, 2020.
[13] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. International Conference on Learning Representations (ICLR), 2019.
[14] Shangqian Gao, Feihu Huang, Weidong Cai, and Heng Huang. Network pruning via perfor- mance maximization. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 9270-9280, 2021.

---

[15] Shangqian Gao, Feihu Huang, Jian Pei, and Heng Huang, Discrete model compression with resource constraint for deep neural networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1899-1908, 2020.
[16] Ross Girshick. Fast r-cnn. In IEEE International Conference on Computer Vision (ICCV), pages 1440-1448, 2015.
[17] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. IEEE International Conference on Computer Vision (ICCV), pages 580-587, 2014.
[18] Shaopeng Guo, Yujie Wang, Quanquan Li, and Junjie Yan. Dmep: Differentiable markov channel pruning for neural networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1539-1547, 2020.
[19] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems (NeurIPS), pages 1135-1143, 2015.
[20] Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain surgeon. Advances in neural information processing systems, 5, 1992.
[21] Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick. Mask r-cnn. In IEEE International Conference on Computer Vision (ICCV), 2017.
[22] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[23] Yang He, Yuhang Ding, Ping Liu, Linchao Zhu, Hanwang Zhang, and Yi Yang. Learning filter pruning criteria for deep convolutional neural networks acceleration. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages
2020.
[24] Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. In International Joint Conference on Artificial Intelligence (IJCAI), pages 2234-2240, 2018.
[25] Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 4340-43449, 2019.
[26] Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. Ame: Automl for model compression and acceleration on mobile devices. Proceedings of the European conference on computer vision (ECCV), pages 784 800, 2018.
[27] Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. Proceedings of the IEEE international conference on computer vision, pages 1389-1397, 2017.
[28] Zejiang Hou and Sun- Yuan Kung. A feature-map discriminant perspective for pruning deep neural networks. arXiv preprint arXiv:2005.13796 2020.
[29] Zejiang Hou, Minghai Qin, Fei Sun, Xiaolong Ma. Kun Yuan, Yi Xu, Yen-Kuang Chen, Rong Jin, Yuan Xie, and Sun- Yuan Kung. Chex: Channel exploration for enn model compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12287-12298, 2022.
[30] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: /704.v:7704.044 04861, 2017.
[31] Zehao Huang and Naiyan Wang. Data-driven sparse structure selection for deep neural networks. Proceedings of the European Conference Computer Vision, pages 304-320, 2018.
12

---

[32] Minsoo Kang and Bohyung Han. Operation- aware soft channel pruning using differentiable masks. Proceedings of International Conference on Machine Learning, pages 5122-5131, 2020.
[33] Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
[34] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr Snip:
Single-shot network pruning based on connection sensitivity. In International Conference on Learning Representations (ICLR), 2019.
[35] Bailin Li, Bowen Wu, Jiang Su, and Guangrun Wang. Eagleeye: Fast sub-net evaluation for efficient neural network pruning, Proceedings of European Conference on Computer Vision, pages
2020.
[36] Changlin Li, Guangrun Wang, Bing Wang. Xiaodan Liang, Zhihui Li, and Xiaojun Chang. Dynamic slimmable network. Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
[37] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. Proceedings of International Conference on Learning Representations, 2017.
[38] Yawei Li, Shuhang Gu, Christoph Mayer, Luc Van Gool, and Radu Timofte. Group sparsity: The hinge between filter pruning and decomposition for network compression. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 8018-8018 88-807, 2020.
[39] Lucas Liehenwein, Cenk Baykal, Harry Lang, Dan Feldman, and Daniela Rus. Provable filter pruning for efficient neural networks. Proceedings of International Conference on Learning Representations, 2020.
[40] Mingbao Lin, Rongrong Ji, Yan Wang, Yichen Zhang, Baochang Zhang, Yonghong Tian, and Ling Shao. Hrank: Filter pruning using high-rank feature map. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1529-1538, 2020.
[41] Mingbao Lin, Rongrong Ji, Yuxin Zhang, Baochang Zhang, Yongjian Wu, and Yonghong Tian. Channel pruning via automatic structure search. In International Joint Conference on Artificial Intelligence (IJCAI), pages 673-679,70 2020.
[42] Mingbao Lin, Yuxin Zhang, Yuchao Li, Bohong Chen, Fei Chao, Mengdi Wang, Shen Li, Yonghong Tian, and Rongrong Ji. Ixn pattern for pruning convolutional neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[43] Shaohui Lin, Rongrong Ji, Chenqian Yan, Baochang Zhang, Liujuan Cao, Qixiang Ye, Feiyue Huang, and David Doermann. Towards optimal structured enn pruning via generative adver- sarial learning. Proceedingy of the IEEE/CVE conference on computer vision and pattern recognition, pages 2790-2799, 2019.
[44] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision (ECCV), pages 740-755, 2014.
[45] Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang. Tim Kwang-Ting Cheng, and Jian Sun. Metapruning: Meta learning for automatic neural network channel pruning. In IEEE International Conference on Computer Vision (ICCV) pages 3296-3305, 2019.
[46] Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In Proceedings of the IEFE international conference on computer vision, pages 2736-2744, 2017.
[47] Jian-Hao Luo and Jianxin Wu. Neural network pruning with residual-connection and limited- data. In IEFE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1458- 1467, 2020.
13

---

[48] Jian-Hao Luo. Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In IEEE International Conference on Computer Vision (ICCV). pages 5058-5066, 2017.
[49] Fanxu Meng, Hao Cheng, Ke Li. Huixiang Luo, Xiaowei Guo, Guangming Lu, and Xing Sun. Pruning filter in filter. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 17629-17640. Curran Associates, Inc., 2020.
[50] Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu, and Paulius Micikevicius. Accelerating sparse deep neural networks. arXiv preprint wwXiv:2104008375 2021.
[51] Pavlo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 11264-12272 2019.
[52] Xuefei Ning, Tianchen Zhao, Wenshuo Li, Peng Lei, Yu Wang, and Huazhong Yang. Dsa: More efficient budgeted pruning via differentiable sparsity allocation. In European Conference on Computer Vision (ECCV), pages 592-607. Springer, 2020.
[53] Jeff Pool and Chong Yu. Channel permutations for n: M sparsity. Advances in Neural Information Processing Systems (NeurIPS), 34, 2021.
[54] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances neural information processing systems, pages 91-99, 2015.
[55] Olivier Giroux et al. Ronny Krashinsky. Nvidia ampere sparse tensor core. https //developer nvidia. ceaidda..combboog/ haall///widiiaaaaaarrraarraeeeee inn-eetth 2020.
[56] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations (ICLR), 2015.
[57] Xiu Su, Shan You, Tao Huang, Fei Wang, Chen Qian, Changshui Zhang, and Chang Xu. Locally free weight sharing for network width search. Proceedings of International Conference Learning Representations, 2021.
[58] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1-9, 2015.
[59] Yehui Tang, Yunhe Wang, Yixing Xu, Dacheng Tao, Chunjing Xu, Chao Xu, and Chang Xu. Scop: Scientific control for reliable neural network pruning. Proceedings of Advances in Neural Information Processing Systems, 2020.
[60] Carl Yang, Aydin Buluç, and John D Owens. Design principles for sparse matrix multiplication on the gpu. In Euro-Par 2018: Parallel Processing: 24th International Conference on Parallel and Distributed Computing, Turin, Italy, August 27-31, 2018, Proceedings, pages 672-687. Springer, 2018
[61] Tien-Ju Yang, Andrew Howard, Bo Chen, Xiao Zhang, Alec Go, Mark Sandler, Vivienne Sze, and Hartwig Adam. Netadapt: Platform-aware neural network adaptation for mobile applications. In Proceedings of the European Conference on Computer Vision (ECCV), pages 285-300, 2018.
[62] Mao Ye, Chengyue Gong, Lizhen Nie, Denny Zhou, Adam Klivans, and Qiang Liu. Good subnetworks provably exist: Pruning via greedy forward selection. Proceedings of International Conference Machine Learning, pages 10820-10830, 2020.
14

---

[63] Zihao Ye, Ruihang Lai, Junru Shao, Tianqi Chen, and Luis Ceze. Sparsetir: Composable abstractions for sparse compilation deep learning, In Proceedings of the 28th ACM Inter- national Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, ASPLOS 2023, page 660-678, New York, NY, USA, 2023. Association for Computing Machinery.
[64] Yuxin Zhang, Mingbao Lin, Chia- Wen Lin, Jie Chen, Yongjian Wu, Yonghong Tian, and Rongrong Ji. Carrying out cnn channel pruning in a white box. IEEE Transactions on Neural Networks and Learning Systems (TNNLS), 2022.
[65] Yuxin Zhang, Mingbao Lin, ZhiHang Lin, Yiting Luo, Ke Li, Fei Chao, YONGJIAN WU, and Rongrong Ji. Learning best combination for efficient n: M sparsity. In Advances in Neural Information Processing Systems, 2022.
[66] Aojun Zhou, Yukun Ma, Junnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hongsheng Li. Learning n: M fine seee-graned structured sparse neural networks from scratch. In International Conference on Learning Representations (ICLR). 2021.
[67] Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. In International Conference Learning Representations Workshop (ICLRW), 2017.
[68] Tao Zhuang, Zhixuan Zhang, Yuheng Huang, Xiaoyi Zeng, Kai Shuang, and Xiang Li. Neuron- level structured pruning using polarization regularizer. Proceedings of Advances in Neural Information Processing Systems, 2020.
[69] Zhuangwei Zhuang, Mingkui Tan, Bohan Zhuang, Jing Liu, Yong Guo, Qingyao Wu, Junzhou Huang, and Jinhui Zhu Discrimination-wware channel pruning for deep neural networks. Proceedings of Advances in Neural Information Processing Systems, pages 883-894, 2018.
15

---

A Supplementary Material
This Supplementary Material is structured follows. We provide formulation of our algorithm in Section B To investigate the effectiveness of different components of our SUBP, we conduct ablation studies and provide additional experimental results Section C In Section D we provide deployment results on the x86 platform of Intel(R) Xeon(R) Platinum 8260L CPU @ 2.30GHz to
further explore the performance of xxN sparse on different platforms. Finally, in Section discuss the societal impact of our method.
we
B Algorithm Formulation
Algorithm 1: Overview of the SUBP method.
Input: An L-layer CNN model with weights W = [ S < L); block binary mask
matrices M {M), {0, 1il1 Si sis L, L.Osis Can N 1,0 S C 1]; indices of activate blocks with the top scores T; indices of regrow blocks based importance sampling G; target prune rate p; initial regrowing factor dg; importance balance coefficient X; sampling
attention balance factor 7: training epochs Timati start and end epoch in the pruning-regrowing
stage ls. le training set ID;
Output: A sub-model satisfying the target prune rate p. its optimal weight values W and binary
mask M :
Randomly initialize the model weights W;
Initialize {Mj,r Vi, Vj, Vk} 1:
Reformat W to according to Section for each training epoch E [Trnat] do
Sample mini-batch from D and update the model weights W t, < S. te then
Reset [Mi,. Vi, Vkj to
10
Compute the importance score S of block by Eq. 4:
11
Get the indices of activate blocks with the top scores by Eq
12
Prune the bottom bottom-ranking block by set &M) $ TJ 0;
13
Compute the importance sampling probabilities by
P),, = exp
/EmTT exp
14
Compute the regrowing factor by Eq. 6
15
Get the indices of regrow blocks based on importance sampling by
Gj Multinomial (p),,.l & T, 8,C,) without replacement;
16
Regrow the blocks by resetting IM E 9j} to 1
C Ablation Analysis
Table 6: Compare different design choices in the regrowing stages of the SUBP method. All the experiments are based the TinylmageNet with ResNet18(1x33) x32). The random baseline is 57.0%.
Regrowing factor
Design choices
8n 0.1
8n = 0.2
dg = 0.3
8n 0.4
Full 58.5%
Top-1
57.6%
58.4%
57.9%
58.0%
Decay scheduler for block regrowing Design choices
Default Constant
Linear decay 58.4%
Cosine decay 58.3%
Top-1
58.3%
57.5%
In Table we investigate the effectiveness of different design choices our block regrowing stage. All the experiments are based on the TinylmageNet with ResNet18(1x32)) Compared to the random
16

---

baseline with 57.0% top-1 accuracy, our SUBP achieves consistent improvement under the different settings.
We find that regrowing factor 6p significantly impacts the final quality of the model. Intuitively, a larger regrowing factor provide more extensive sampling space during training and retain the model's capacity to greater extent. However, a sizeable regrowing factor may also cause drastic sub-model structure changes, affecting stability during training. As shown, the accuracy is improved by 0.8% as the óg increases from 0.1 to 0.2. When do increases again, the model's 's accuracy drops until óg is the full model size. This suggests that the relationship between the regrowing factor and the final quality of the model is varied, and selecting an appropriate regrowing factor in specific circumstances can improve the final quality.
We also investigate the decay scheduler for the block regrowing stage. We compare several decay schedulers, including default (Eq. constant, linear, and cosine. The experiments show SUBP has good robustness to different decay schedulers, as default, linear, and cosine decay schedulers all show similar performance. With decay scheduler, the sampling space can be gradually decreased, and the sub-model under training can converge stably.
Table 7: Ablation study of block size and epochs. All the experiments are based on the ImageNet with ResNet18.
Method
FLOPs
Accuracy
Epochs
DMCP
1.04G
69.0%
150
CHEX
1.03G
69.6%
250
SUBP (1x4)
1.03G
70.4%
250
SUBP (1x8)
1.03G
70.2%
250
SUBP (1x16)
1.03G
68.7%
100
SUBP (1x16)
1.03G
69.2%
150
To achieve a fairer and fuller comparison with previous methods, we conduct ablation study with the same epoch and different block sizes. It can be seen in Tabtt both longger training epochs and finer pruning granularity can improve the final results. We can weight trade offs in practical applications.
D Deployment on x86 Platform
100
Lesabisfen fet
LeTMts fame
125
Notwork Latency (ms)
Network Latency (ms)
1.30
0.125
0.25
0.375
0.5
82625
0.75
0.575
0.125
0.25
0.375
0.5
0.625
0.25
0.375
Pruning Rate P
Pruning Rate p
Figure 5: Network latency comparison between uniform xxN sparse against non-uniform and dense model with varying N and prune rates. The experiment is conducted using ResNet18 8 and set the input shape as (4, 3, 224, 224) on the x86 platform of Intel(R) Xeon(R) Platinum 8260L CPU 2.30GHz with single thread (left) and two threads (right). Best viewd in colors.
In order to further explore the performance of xN sparse DNNs on different platforms, as shown in Fig. 5 we also conducted corresponding experiments the x86 platform of Intel(R) Xeon(R) Platinum 8260I. CPU @ 2.30GHz and obtain the similar results general: 1) The gain vanilla convolution in multithreading scenarios is much greater than that of 1xN sparse convolution. 2) The
17

---

inference speed of uniform x is slightly faster than that of non-untie mon-unfform in the case of multithread- ing, indicating the importance of workload balance again. However, unlike the performance on the arm platform of Apple M1 Pro CPU @ 3.20GHz, the xxN sparse DNNs are significantly accelerated when N is set 16 and on the Platinum 82601. CPU @ 2.30GHz. We can also notice that in most cases, N=32 achieves fast inference speed.
E Societal Impact
Our method can reduce the computational overhead of training and inferencing stages while achieving satisfactory accuracy modern CNN models. This facilitate the application of CNN models on edge devices and is of high value for the community and society to realize Green AI. At the same time, our xN sparse DNNs are based new sparse operators, which promote the progress of related hardware and algorithms to a certain extent.
18


Reviews:
[{"review_id": "9wyFaHYzk3", "reviewer": "Reviewer_Njjh", "summary": "Structural weight pruning methods have gain popularity due to the hardware friendly pattern. This work improves 1xN pruning with prune-and-grow and uniform block pruning. Evaluation shows outperforming accuracy results at similar levels of FLOPs than prior work.", "questions": "- Regarding the key design idea of uniform block pruning, what is being done in Section 3.1 and how different compared with prior work of block pruning? Figure 2 could be modified to better illustrate and/or an algorithm description of the steps. Is it every output channel has the same number of pruned 1xN blocks? \n- In Section 3.2, block pruning vis angular redundancy could be better positioned in the overall pruning.\n- What are the sparsity ratios used in Table 2? While showing FLOPs is informative, it is not the end product, and information on the sparsity ratio could help evaluation the proposed method. ", "limitations": "Not adequate. From the evaluation, the proposed method, among many other pruning work, significantly increase the training epochs. The negative impacts from more training costs need to be discussed.  ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- This paper presents a new contribution on block-wise pruning with soft (prune-and-grow) in channel-wise uniformly.\n- The evaluation shows potential significance of the proposed block pruning method in achieve accuracy with reduced FLOPs.", "weaknesses": "- The paper could be improve in clarity and sufficient implemantion details to help readability. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your kind comments. We sincerely wish our response can well address your concerns so as to raise the rating score. \n\n**Q1:** The author explain how it differs from previous block pruning. The author should add an algorithm to describe of the steps better.\n\n**A1:** More specifically. According to [1] and [2], 1xN block pruning implies that **N consecutive output channels have the same sparse pattern.** In Section 3.1, we first introduced the basic concept of 1xN block pruning. However, we find that previous work has only been constrained by Eq.(1), which may result in different pruning rates among different otuput channels and thus unbalanced computational loads across threads. Thus we optimize Eq.(1) for the multi-threads scenario to Eq.(2). \n**For example**, if we assume a Conv2D with a weight (out_channel, in_channel, kernel_size, kernel_size) if (6, 4, 1, 1)，N=2，prune_rate=0.5. We claim 1 means preserved and 0 means pruned. Eq(1) may result in \n\n$$\\begin{bmatrix}\n 1 & 0 & 1 & 1 \\\\\\\\\n 1 & 0 & 1 & 1 \\\\\\\\\n 0 & 1 & 0 & 0 \\\\\\\\\n 0 & 1 & 0 & 0 \\\\\\\\\n 1 & 1 & 0 & 0 \\\\\\\\\n 1 & 1 & 0 & 0\n\\end{bmatrix}$$\n\nAccording to the shortest plank effect, the speed at which the network runs is limited by the first and second output channels. Eq. (2) constraints the sparsity of each output channel to keep the same, we may get \n\n$$\\begin{bmatrix}\n 1 & 0 & 0 & 1 \\\\\\\\\n 1 & 0 & 0 & 1 \\\\\\\\\n 0 & 1 & 1 & 0 \\\\\\\\\n 0 & 1 & 1 & 0 \\\\\\\\\n 1 & 1 & 0 & 0 \\\\\\\\\n 1 & 1 & 0 & 0\n\\end{bmatrix}$$\n\nIn this case, each output channel has the same computational load, favouring multi-threaded scenarios running. \nTo better illustrate the pruning process, we show the relevant **algorithm** description in the **Appendix.B.**\n\n**Q2:** The author could positione the block pruning vis angular redundancy better in the overall pruning.\n\n**A2:** Thank you for your careful reading and we will revise this section appropriately to better present the block pruning vis angular redundancy if our paper is accepted.\n\n**Q3:** The authors only report the training FLOPs, the authors are encouraged to report the sparsity ratio to help evaluation the proposed method.\n\n**A3:**  We appreciate this valuable comment, which helps us to further improve the quality of our paper. The **sparsity ratios (SR)** for Table 2 will be added in the final version if the paper is accepted (the table can not be shown fully for characters limitation). We note that our method can achieve better results in terms of sparsity ratio and accuracy. An intutive explanation is that previous methods with **different sparsity ratios (SR)** between layers tend to pruning the layers that are shallow and these parameters tend to have more computations.  For example, when the Conv2D's weight shape (in_channel, out_channel, kernel_size, kernel_size) is (32, 64, 3, 3), it will cause more FLOPs when input size is [1, 32, 56, 56] than [1, 32, 28, 28]. **In our SUBP, we prune all layers with the same sparsity ratio.**\n\n| Method        | PT | SR    | FLOPs | Top-1 | Epochs |\n|---------------|----|-------|-------|-------|--------|\n| ResNet-18     |    |       |       |       |        |\n|    PFP        | √  | 43.8% | 1.27G | 67.4% | 270    |\n|    SCOP       | √  | 39.3% | 1.10G | 69.2% | 230    |\n|    SFP        | √  | 47.6% | 1.04G | 67.1% | 200    |\n|    FPGM       | √  | 47.6% | 1.04G | 68.4% | 200    |\n|    DMCP       | ×  | 17.6% | 1.04G | 69.0% | 150    |\n|    CHEX       | ×  | 38.7% | 1.03G | 69.6% | 250    |\n|    SUBP(1x16) | ×  | 44.1% | 1.03G | 69.9% | 250    |\n|    SUBP(1x32) | ×  | 44.1% | 1.03G | 69.7% | 250    |\n| ResNet-34     |    |       |       |       |        |\n|    Taylor     | √  | 21.1% | 2.8G  | 72.8% | -      |\n|    SFP        | √  | 49.1% | 2.2G  | 71.8% | 200    |\n|    FPGM       | √  | 49.1% | 2.2G  | 72.5% | 200    |\n|    GFS        | √  | 32.5% | 2.1G  | 72.9% | 240    |\n|    DMC        | √  | -     | 2.1G  | 72.6% | 490    |\n|    NPPM       | √  | -     | 2.1G  | 73.0% | 390    |\n|    SCOP       | √  | 45.6% | 2.0G  | 72.6% | 230    |\n|    CafeNet    | ×  | 21.1% | 1.8G  | 73.1% | 300    |\n|    CHEX       | ×  | 29.2% | 2.0G  | 73.5% | 250    |\n|    SUBP(1x16) | ×  | 43.8% | 2.0G  | 73.7% | 250    |\n|    SUBP(1x32) | ×  | 43.8% | 2.0G  | 73.6% | 250    |\n\n**Q4:** The author should discuss the negative impact from the increasing training cost.\n\n**A4:**  To be honest, our method does increase the training overhead compared to other methods, a process that increases energy consumption and greenhouse gas emissions. Therefore, it is also our future duty to investigate how to improve the convergence speed of the model with limited training resources. If we look at it from a different perspective, in terms of the entire model lifecycle, the gains from these additional training overheads are acceptable to some extent if they can be covered at the time of model deployment.\n\n[1] Elsen, Erich, et al. Fast sparse convnets. *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*. 2020.\n\n[2] Mingbao Lin, et al. 1xN Pattern for Pruning Convolutional Neural Networks, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.\n\n\n\n"}, {"review_id": "CEu3NgE97q", "reviewer": "Reviewer_cAVo", "summary": "This paper proposes a new fine-grained structured pruning method, which can train from scratch to reduce the training cost and prunes the weights in a uniform manner to handle the load imbalance problem. During pruning, the method uses block angular redundancy as the criterion to discard the redundant blocks and allows pruned blocks to regrow to the network based on the importance sampling to search for a better sparse pattern. Experiments show that the SUBP method can achieve a significant accuracy improvement compared with other methods under the same FLOPs constraint.", "questions": "- If use the same data augmentation and hyper-parameters to train the dense model with the same 250 epochs, what’s the dense model accuracy? Because hyper-parameters and data augmentation are different from other methods, the comparison is unfair. If you can compare the accuracy loss of different methods compared with their dense baselines, it may better demonstrate the effectiveness of SUBP.\n- Is this method effective on vision transformers? Transformer models have been widely used in various computer vision tasks.\n- Does the regrow method apply to the experiments in Table 1, including the l1 norm and BPAR?", "limitations": "The author discussed the limitation of the method and the societal impacts. Other limitations of my concern can refer to the weaknesses.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written.\n- The proposed SUBP method can greatly improve the model accuracy than kernel pruning methods under a similar pruning ratio.\n- The authors propose uniform pruning to handle the load imbalance problem and show the real speedup on the CPU to evaluate the effectiveness of this method.\n", "weaknesses": "- Experiments are insufficient. The authors only evaluate the proposed method on the classification task. It is better to discuss the generalization of the method and evaluate on different tasks.\n- The paper lacks a comparison of accuracy with the non-uniform block pruning method. Figure 4 shows that the speedup of the uniform 1xN method is limited. If there is an accuracy drop, I wonder if it is worth adding the uniform constraint at the cost of such a model accuracy loss.\n- Experiments only compare with the kernel pruning methods and lack comparisons with other block pruning methods, e.g. PatDNN [1] and 1xN pruning [2].\n- This paper only focuses on CPU, but GPU on mobile is also important. For example, the implementation of the algorithms related to autonomous driving currently heavily relies on Orin GPUs. It is better to analyze the efficiency of this method on GPU, and the uniform pruning method should achieve better speedups.\n\n[1] Wei Niu, et al. PatDNN: Achieving Real-Time DNN Execution on Mobile Devices with Pattern-based Weight Pruning, ASPLOS, 2020.  \n[2] Mingbao Lin, et al. 1xN Pattern for Pruning Convolutional Neural Networks, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your responsible reviews that will help us improve the manuscript. Please see the following answers to your questions.\n\n**Q1:** Author should use the same hyperparameters for fair comparisons, as well as dense model baselines for other methods to better demonstrate the effectiveness of the proposed methods.\n\n**A1:** Your suggestion is valuable and we are aware of it. We trained the dense model of **ResNet18 for 250 epochs**, the top-1 and top-5 accuracy are **71.9% and 90.3%** respectively, which brings 2.1% and 1.2% improvement to the torchvision baseline (90 epochs). \n\nAdmittedly, increasing the epoch does promote better model fitting and thus better results. Methods based on pre-trained pruning usually require a large number of epochs to fine-tune the pruning model, thus achieving \"lossless\" pruning of the model. From a certain point of view, it is also a way to compare the fully trained pruned model with the insufficiently trained dense model, which is also unfair to a certain extent. \n\nTherefore, in this paper our main aim is to propose a method that can be trained from scratch to obtain a pruned model. We have chosen epochs that are similar to those of recent methods, in the hope of achieving some degree of fair comparison.\n\n**A2:** Does the method the authors work on Vision Transformer？\n\n**Q2:** Yes, our proposed method can be applied to Vision Transformer. On the one hand, we can equate the densely computed Dense layer in Transformer to a 1x1 convolutional layer, and understand the acceleration based on the acceleration principle of 1x1 convolution. On the other hand, the sparse model of 1xN block pruning is proposed for matrix multiplication, and according to [1], the computation of the Dense layer should have higher computational density than the convolutional layer, so we believe that 1xN block pruning can achieve better results in ViT.\n\n**Q3:** Whether the regrow method  is applied to results in Table1.\n\n**A3:** No. The main purpose of our Table 1 is to show that BPAR-based 1xN block pruning is superior to L1-norm . Thus our results in Table 1 are based on the pre-training-pruning-fine-tuning pruning model, using the same hyper-parameters in [2].\n\n**Q4:** The authors are encouraged to conduct experiments on other tasks.\n\n**A4:** Following your advice, we further conduct experiments on two other tasks including object detection and instance segmentation. Below displays the experimental results in which the proposed SUBP performs well on various vision tasks.\n\n**Object detection results on COCO:**\n    \n| **Model**         | **Block Size** | **mAP** |\n| ----------------- | -------------- | ------- |\n| F-RCNN-R50 (4.1G) | -              | 37.4    |\n| F-RCNN-R50 (2G)   | 1x32           | 38.4    |\n| F-RCNN-R50 (2G)   | 1x16           | 38.5    |\n| F-RCNN-R50 (1G)   | 1x32           | 37.1    |\n| F-RCNN-R50 (1G)   | 1x16           | 37.3    |\n\n**Instance segmentation results on COCO:**\n\n| **Model**         | **Block Size** | **Box mAP** | **Mask mAP** |\n| ----------------- | -------------- | ----------- | ------------ |\n| F-RCNN-R50 (4.1G) | -              | 38.2        | 34.7         |\n| F-RCNN-R50 (2G)   | 1x32           | 39.2        | 35.4         |\n| F-RCNN-R50 (2G)   | 1x16           | 39.4        | 35.5         |\n| F-RCNN-R50 (1G)   | 1x32           | 37.4        | 33.8         |\n| F-RCNN-R50 (1G)   | 1x16           | 37.5        | 33.8         |\n\n**Q5:** Authors should make fuller comparisons in their papers.\n\n**A5:** This is our problem and we should place some of the 1xN results from Table 1 in Table 2.  The results are shown as following:\n\n| Block size | Method | Network     | FLOPs | Top-1 accuracy | Epochs  |\n| ---------- | ------ | ----------- | ----- | -------------- | ------- |\n| 32         | 1xN    | ResNet50    | 2.3G  | 76.0%          | 100+180 |\n| 32         | SUBP   | ResNet50    | 2.0G  | 77.4%          | 250     |\n| 32         | 1xN    | MobileNetV1 | 300M  | 69.6%          | 100+180 |\n| 32         | SUBP   | MobileNetV1 | 279M  | 71.1%          | 250     |\n| 16         | 1xN    | ResNet50    | 2.3G  | 76.3%          | 100+180 |\n| 16         | SUBP   | ResNet50    | 2.0G  | 77.6%          | 250     |\n| 16         | 1xN    | MobileNetV1 | 300M  | 69.6%          | 100+180 |\n| 16         | SUBP   | MobileNetV1 | 279M  | 70.8%          | 250     |\n\n**Q6:** 1xN block pruning on GPU.\n\n**A6:**  Recently we find that there have been related papers for accelerated experiments on GPUs for 1xN block pruning. According to [3], 1xN block pruning can achieve significant speedups at lower sparsity ratios on general purpose GPUs. Therefore, we believe that 1xN has the same potential for GPUs.\n\n[1] Ding, Xiaohan, et al. Repmlpnet: Hierarchical vision mlp with re-parameterized locality. *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2022.\n\n[2] Mingbao Lin, et al. 1xN Pattern for Pruning Convolutional Neural Networks, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.\n\n[3] Tan, Yijun, et al. Accelerating Sparse Convolution with Column Vector-Wise Sparsity. *Advances in Neural Information Processing Systems* 35 (2022).\n\n\n\n\n"}, {"review_id": "PLrXAqqXoU", "reviewer": "Reviewer_nWst", "summary": "SUBP trains a uniform 1×N sparse CNNs from scratch, with three characteristics: (1) a periodic block pruning and regrowing technique via importance sampling, (2) a pruning criterion based on angular redundancy across blocks,  and (3) a uniform 1×N sparse pattern for multithreading acceleration. The authors also provide latency results on hardware platforms.", "questions": "Please refer to “Weaknesses\".", "limitations": "Please refer to “Weaknesses\" for limitations. The authors adequately addressed the potential negative societal impact of their work.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 1, "strengths": "1. The method achieves competitive performance against many existing methods. \n\n2. SUBP trains a uniform 1×N sparse CNNs from scratch.", "weaknesses": "1. Novelty concern. This paper uses a prune-and-grow idea to train a sparse model from scratch, which is been well-studied by many papers in sparse training [1-2]. DMCP and CHEX also don’t require pre-training. Work [3] studied how to prune the network from scratch with a grow-and-prune methodology. Besides, the 1xN pruning technique needs compiler optimization (like TVM) to achieve real inference acceleration. Therefore, the acceleration part of this work highly relies on TVM.\n\n2. The performance improvements are marginal. In Table 1, it’s hard to tell if SUBP is better than CHEX. More important, CHEX uses channel pruning, which could achieve a faster acceleration rate when the pruning ratio is the same. From my point of view, CHEX is basically faster and has the same performance in terms of accuracy. DMCP (channel pruning) is only trained 150 epochs, which is also hard to tell if SUBP has better performance (69.0%,150 epochs vs 69.7%, 250 epochs). Overall, when compared with many SOTA methods, SUBP could not achieve a faster acceleration rate or has an obviously better performance in terms of accuracy. Table 2 has the same problem with many outdated baselines. The authors should provide error bars when the accuracy improvements are marginal.\n \n3. The paper is written poorly. Sections 1 and 3 should be organized better, and section 3 is not clearly written. Table 1 first appeared on Page 6 (section 3) and was first mentioned at the end of Page 7 (section 4). Table 2 has an error with vertical lines exceeding the table. Also, the font size of the figures in this paper is very small, especially in Figure 2. All these things make this paper hard to read.\n\n4. There is no ablation study to show the effectiveness of each components of SUBP.\n\n[1] Rigging the Lottery: Making All Tickets Winners, ICML 2020.\n\n[2] MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edge, Neurips 2021.\n\n[3] Effective Model Sparsification by Scheduled Grow-and-Prune Methods, ICLR 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your in-depth review that will help us strengthen the manuscript. We hope our response can address your concern here.\n\n**Q1:** Prune-and-regrow idea has been well-studied and acceleration of TVM is highly relies on TVM.\n\n**A1:** With all due respect, though the concept of grow-and-prune exists in other methods, we would like to highlight that our SUBP is different from the previous method on several aspects. **In terms of pruning strategy**, our approach uses importance sampling based approach for the weights that have been removed, unlike the deterministic strategy of selecting topk importance weights for resurrection in [1-3]; **in terms of weight/channel importance measure**, [1-4] use the magnitude of the weights, norm-based criterion or matrix decomposition-based importance criterion, our method takes into account the angular redundancy of the weight blocks, which is different from the previous methods; **in terms of granularity of pruning**, we study 1xN block pruning, while CHEX [4], DMCP [5] focussing on channel pruning and [1] focussing on weight pruning.\n\n**For the acceleration of 1xN block pruning,** on CPU devices, in addition to TVM[6], MNN[7] and XNNPACK[8] provide corresponding acceleration operators. On GPU devices, CVW[9] has also experimentally demonstrated that 1xN block pruning on GPUs has equally great potential. \n\n**Q2:** The authors' method does not improve significantly enough compared to CHEX and is not as efficient as CHEX or other SOTA methods with channel pruning. Also, compared to DMCP, the authors' method trains more epochs, making it difficult to state whether the method has better performance.\n\n**A2:** **In response to the issue that the accuracy of our method compared CHEX is marginal,** we believe there are two reasons: **firstly**, since different layers of the neural network have different degrees of redundancy, methods like CHEX and DMCP that apply different sparsity ratios to different layers tend to achieve higher results; **secondly**, since our methods all apply a coarser-grained block structure to the model, when we apply our method When we apply our method to 1x4, 1x8 block pruning, our method achieves better results, while referring to the inference delay in Fig.4, the appropriate fine-grainedness can also achieve efficient inference latency.\n\n**For inference latency**, papers [8-10] have shown that this semi-structured sparse pruning model achieves a better trade-off between accuracy and inference latency compared to channel pruning. It is also important to note that the current channel pruning approach, although it can achieve better efficiency on GPUs, can not be deployed on edge devices like CHEX with single channel pruning because the computational chips on edge devices and deploy framework often require channels to be aligned in multiples of 4 or 8, such as Ascend310 and Bolt, complementary channels incurs additional overheads. Our approach can be deployed on general-purpose hardware with specific sparse operators, and channel patching is not required. In addition, there has been a large amount of research focused on how to improve the efficiency of sparse computing. We believe that in the future, this semi-structured sparse model can achieve better trade-off.\n\n**Regarding the comparison with DMCP**, we train 100 and 150 epochs on ResNet18 (1x16), and we can see that our method can also achieve better performance with similar epochs.\nAll the results are shown in the following table.\n\n| Method | FLOPs | Accuracy | Epochs |\n| -| -- | - | - |\n| DMCP | 1.04G | 69.0% | 150 |\n| CHEX | 1.03G | 69.6% | 250 |\n| SUBP (1x4)  | 1.03G | 70.4% | 250 |\n| SUBP (1x8)  | 1.03G | 70.2% | 250 |\n| SUBP (1x16) | 1.03G | 68.7% | 100 |\n| SUBP (1x16) | 1.03G | 69.2% | 150 |\n\n**Q3:** The authors should have organised the chapters more clearly and revised the problematic areas in the figures and tables.\n\n**A3:** First of all, I am very sorry for the problems in the fonts and tables in the figures in this paper that cause you problems in reading, if the paper is accepted we will revise the problems that exist. \n\n**Section 1** focuses on the organization from introducing the background, to the granularity of pruning and the problems that exist at the hardware and deployment level. All of them lead to the soundness of our study of 1xN pruning in this paper. Then we point out the shortcomings of the previous studies, and finally present our approach.\n\n**In Section 3** we organize along the definition of 1xN block pruning, block pruning based on angular redundant and prune-and-grow method. We place the relevant algorithm the appendix for better readability.\nFinally, we hope that our explanations will solve your doubts.\n\n**Q4:** Authors should add ablation experiments to demonstrate the effectiveness of each component.\n\n**A4:** We conducted experiments in Table.1 to validate the effectiveness of our pruning criterion compared to norm-based. Also, combining the data in Tables 1 and Tables 2, we verified the effectiveness of the different parts of the methodology. In the Appendix section C, we also added the  ablation experiments of hyper-parameter. \n\n[1] Rigging the Lottery: Making All Tickets Winners, ICML 2020.\n\n[2] MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edge, Neurips 2021.\n\n[3] Effective Model Sparsification by Scheduled Grow-and-Prune Methods, ICLR 2022.\n\n[4] CHEX: Channel exploration for CNN model compression. CVPR 2022.\n\n[5] DMCP: Differentiable markov channel pruning for neural networks. CVPR 2020.\n\n[6] TVM: An Automated End-to-End Optimizing Compiler for Deep Learning. OSDI 2018.\n\n[7] MNN: A universal and efficient inference engine. MLSYS 2020.\n\n[8] Fast sparse convnets. CVPR. 2020.\n\n[9] Accelerating Sparse Convolution with Column Vector-Wise Sparsity. NeurIPS 2022.\n\n[10] 1xN Pattern for Pruning Convolutional Neural Networks. TPAMI, 2022.\n"}, {"review_id": "bmbTCMqvGg", "reviewer": "Reviewer_Ah3e", "summary": "This paper proposes a new pruning approach to train a uniform 1×N sparse structured network from scratch, dubbed as SUBP. SUBP pruned blocks according to block angular redundancy. The blocks are pruned in a uniform manner to make full use of multithread parallelism computation. During the training process, the pruned blocks can be regrown through importance sampling. They conduct experiments across various CNN architectures to show the superiority of  SUBP.", "questions": "- Please see the weaknesses part.\n- Why the inference latency of the sparse model is lower than the inference latency of the dense model in low pruning rates?", "limitations": "As discussed in Section 5, there are still missing experiments, including applying  1×N sparsity on other types of DNNs like RNN, transformer, and other tasks including object detection, natural language processing, etc.. The generalization on downstream tasks still needs to be verified.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper is well-written and its motivation is reasonable.\n- Code seems to be available, and it'll be helpful if the authors could open-source the work if the paper is accepted.\n- They deploy the pruned model in both the arm platform and x86 platform to get actual inference speed. These results well-validate that uniform pruning methods are faster than non-uniform pruning methods.", "weaknesses": "-  The authors are encouraged to conduct experiments on other tasks, not just classification.\n-  The authors only report the FLOPs in Table 2, the authors are encouraged to report the training time.\n-  Compared with method CHEX, the improvements in accuracy are limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive and supportive comments.\n\n**Q1:** The authors are encouraged to conduct experiments on other tasks.\n\n**A1:** Following your advice, we further conduct experiments on two other tasks including object detection and instance segmentation. Below displays the experimental results in which the proposed SUBP performs well on various vision tasks.\n\n**Object detection results on COCO:**\n\n|**Model**|**Block Size**|**mAP**|\n|-|-|-|\n|F-RCNN-R50 (4.1G)|-|37.4|\n|F-RCNN-R50 (2G)|1x32|38.4|\n|F-RCNN-R50 (2G)|1x16|38.5|\n|F-RCNN-R50 (1G)|1x32|37.1|\n|F-RCNN-R50 (1G)|1x16|37.3|\n\n**Instance segmentation results on COCO:**\n\n|**Model**|**Block Size**|**Box mAP**|**Mask mAP**|\n|-|-|-|-|\n|F-RCNN-R50 (4.1G)|-|38.2|34.7|\n|F-RCNN-R50 (2G)|1x32|39.2|35.4|\n|F-RCNN-R50 (2G)|1x16|39.4|35.5|\n|F-RCNN-R50 (1G)|1x32|37.4|33.8|\n|F-RCNN-R50 (1G)|1x16|37.5|33.8|\n\n**Q2**: The authors only report the training FLOPs, the authors are encouraged to report the training time.\n\n**A2**: The training time for sparsifying ResNet-18/34/50 on ImageNet is provided in the following.\n\n|Model|Block size|Training Time (NVIDIA RTX 3090 GPU days)|\n|-|-|-|\n|ResNet18|1x16|1.84|\n|ResNet18|1x32|1.83|\n|ResNet34|1x16|2.88|\n|ResNet34|1x32|2.87|\n|ResNet50|1x16|3.61|\n|ResNet50|1x32|3.59|\n\n**Q3:** The authors' method improves accuracy to a limited extent compared to CHEX.\n\n**A3:** Your question is very valuable and we have realised this point in the course of our experiments. We think this reason may come from **two points:** 1. CHEX applies different sparsity ratios to different layers, using the FLOPs of the whole network as pruning targets. Different layers of a neural network have different redundancies, so applying different sparsity ratios to different layers tends to yield better results. Our approach applies the same sparse ratio to each layer of the network; 2. We apply coarser-grained block pruning (1x32, 1x16) to 1xN, and we conducted 1x8 and 1x4 experiments on ResNet18, which yielded 70.2% and 70.4% top-1 accuracy on ImageNet, respectively, compared to CHEX's 69.6%, and, as can be seen in Fig. 4, the 1x8 pruning granularity also achieves better inference latency on hardware.\n\n**Q4:** Why the inference latency of the sparse model is lower than the inference latency of the dense model in low pruning rates.\n\n**A4:** The pattern of sparse computation is very different from that of dense computation. For dense matrix multiplication, GEMM is currently the most dominant method. Sparse matrix multiplication can not make full use of cache like GEMM, so its computational efficiency is very much related to the sparse algorithm and the pattern of sparse matrix, and researchers have carried out a lot of studies, such as spmm, spmv and so on. Current sparse matrix multiplication methods have a sparsity threshold for speedup, above which the computational delay of sparse matrices outperforms that of dense matrix multiplication only. The more efficient sparse matrix multiplication corresponds to a lower sparsity threshold. \n\n"}]

Author rebuttal:

Meta-review: This work improves 1xN pruning with prune-and-grow and uniform block pruning, which can be used for training from scratch to reduce the training cost and prunes the weights in a uniform manner to handle the load imbalance problem. During pruning, the method uses block angular redundancy as the criterion to discard the redundant blocks and allows pruned blocks to regrow to the network based on the importance sampling to search for a better sparse pattern. Experiments show that the SUBP method can achieve accuracy improvements compared with state-of-the-art methods with similar FLOPs. The method adopts uniform pruning to handle the load imbalance problem and shows the real speedup on the CPU platform to demonstrate the actual acceleration.


The paper received scores 3, 5, 6, 6 after the rebuttal. Additional experimental results were added during the rebuttal, which cover more model architectures and downstream tasks for better demonstrating the its effectivness and solved most reviewers' concerns. The main criticism of the score-3 comment focuses on the "marginal improvement over CHEX" and the “limited applications in today's GPU”. Considering the improvement from DMCP to CHEX (60.90% to 69.6%), the improvement of the proposed method (69.6%->70.2%/70.4%) is reasonable. The deployment to actual GPUs might be a challenge for the current compilers. But the GPU makers might contribute to this direction if they find the 1xN pruning methods promising. In addition, CPU makers are also pushing forward deep learning models' deployment on CPUs.

The authors are recommended to include the additional results during the rebuttal in the final version.
Decision: Accept (poster)