SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: Performance comparison of our BPAR prune criterion against  $\ell_1$  norm. The experiment is conducted using MobileNetV1 and ResNet50 with the pruning rate  $p = 50\%$  on ImageNet dataset. We test our BPAR on three cases, which N is set to 8, 16 and 32 respectively.
MobileNetV1
(p = 50%)	ResNet50
(p = 50%)
Accuracy	Top-1	Top-5	Top-1	Top-5	Top-1	Top-5	Top-1	Top-5
Origin	71.15	89.83	71.15	89.83	77.00	93.65	77.01	93.65
Weight Pruning	70.76	89.59	70.76	89.59	77.09	93.61	77.09	93.61
Filter Pruning	65.35	86.26	65.35	86.26	75.38	92.52	75.38	92.52
Prune Criterion	l1norm	BPAR	l1norm	BPAR
Accuracy	Top-1	Top-5	Top-1	Top-5	Top-1	Top-5	Top-1	Top-5
1×2 Pattern	70.28	89.37	-	-	76.65	93.47	-	-
1×4 Pattern	70.05	89.06	-	-	76.51	93.24	-	-
1×8 Pattern	69.91	89.08	70.55	89.42	76.15	93.13	76.70	93.51
1×16 Pattern	69.56	88.93	70.30	89.35	76.25	93.08	76.52	93.20
1×32 Pattern	69.54	88.80	70.03	89.23	75.96	92.95	76.31	93.18
Table 2: Results of ResNet18, ResNet34, ResNet50 and MobileNetV1 on ImageNet dataset. "PT": require pre-training. "SR": sparse ratio.
Method	PT FLOPs	SR	Top-1	Epochs	Method	PT FLOPs	SR	Top-1	Epochs
ResNet-18	ResNet-50
PFP [39]	✓	1.27G	43.8%	67.4%	270	SSS [31]	✗	2.3G	38.8%
SCOP [59]	✓	1.10G	39.3%	69.2%	230	TAS [11]	✗	2.3G	43.5%
SFP [24]	✓	1.04G	47.6%	67.1%	200	GAL [43]	✓	2.3G	16.8%
FPGM [25]	✓	1.04G	47.6%	68.4%	200	Hrank [40]	✓	2.3G	36.7%
DMCP [18]	✗	1.04G	17.6%	69.0%	150	Taylor [51]	✓	2.2G	44.3%
CHEX [29]	✗	1.03G	38.7%	69.6%	250	C-SGD [6]	✓	2.2G	42.8%
SUBP(1×16)	✗	1.03G	44.1%	69.9%	250	SCOP [59]	✓	2.2G	42.8%
SUBP(1×32)	✗	1.03G	44.1%	69.7%	250	DSA [52]	✗	2.0G	-
ResNet-34	CafeNet [57]	✗	2.0G	27.8%	76.9%
Taylor[51]	✓	2.8G	21.1%	72.8%	-	CHEX [29]	✗	2.0G	35.8%
SFP [24]	✓	2.2G	49.1%	71.8%	200	SUBP(1×16)	✗	2.0G	45.5%
FPGM [25]	✓	2.2G	49.1%	72.5%	200	SUBP(1×32)	✗	2.0G	45.5%
GFS [62]	✓	2.1G	32.5%	72.9%	240	SCP [32]	✗	1.9G	-
DMC [15]	✓	2.1G	-	72.6%	490	Hinge [38]	✓	1.9G	-
NPPM [14]	✓	2.1G	-	73.0%	390	AdaptDCP [69]	✓	1.9G	51.5%
SCOP [59]	✓	2.0G	45.6%	72.6%	230	LFPC [23]	✓	1.6G	-
CafeNet [57]	✗	1.8G	21.1%	73.1%	300	ResRep [8]	✓	1.5G	-
CHEX [29]	✗	2.0G	29.2%	73.5%	250	Polarize [68]	✓	1.2G	-
SUBP(1×16)	✗	2.0G	43.8%	73.7%	250	DSNet [36]	✓	1.2G	-
SUBP(1×32)	✗	2.0G	43.8%	73.6%	250	CURL [47]	✓	1.1G	73.8%
MobileNetV1	DMCP [18]	✗	1.1G	43.6%	74.1%
0.75x[30]	✗	325M	38.1%	68.4%	-	MetaPrune [45]	✗	1.0G	53.5%
NetAdapt [61]	✓	284M	-	69.1%	-	EagleEye [35]	✓	1.0G	69.4%
AMC [26]	✓	285M	-	70.5%	-	CafeNet [57]	✗	1.0G	52.9%
MetaPruning [45]	✓	281M	50.9%	70.6%	320	CHEX [29]	✗	1.0G	67.1%
SUBP(1×16)	✗	279M	40.0%	70.8%	250	SUBP(1×16)	✗	1.0G	68.3%
SUBP(1×32)	✗	279M	40.0%	71.1%	250	SUBP(1×32)	✗	1.0G	68.3%
parameters for different models and pruning rates, better results should be obtained in general. In this paper, we calculate FLOPs by counting multiplication and addition as one operation as He [22] did.
4.2 Influence of Pruning Criterion
In this subsection, we will investigate the influence of  $1 \times \mathrm{N}$  pruning criterion. Table 1 shows the compared results with respect to  $\ell_1$  norm [42] and BPAR. As suggested by [37, 48, 51, 28], the pruning criterion selects the appropriate sub-model on the basis of its importance or redundancy, which plays an important role in the traditional PPF approach. Table 1 tells us that our BPAR
outperforms the norm-based method on the ImageNet dataset. For MobileNetV1 and ResNet50, our BPAR can achieve consistent accuracy improvements with the same sparsity and inference speedup. On MobileNetV1(1 × 16), it obtains 0.74% and 0.38% top1-accuracy and top5-accuracy improvement when BPAR is applied. In particular, for pruning a pre-trained MobileNetV1, BPAR achieves better results for N=8 than  $\ell_1$  norm for N=2, which indicates  $1 \times N$  pruning based on BPAR can obtain a better trade-off between accuracy and inference latencies. Similar results can also be observed on the ResNet50 in Table 1. In essence, our BPAR explicitly utilizes the relationship and identifies the mutual angular redundancy between blocks, giving rise to its superior performance.
4.3 Results on ImageNet
To verify the effectiveness of our SUBP, we apply it to the heavyweight CNN model (i.e. ResNet [22]) with different depths and the lightweight CNN model (i.e. MobileNet) initialized with random weights on ImageNet [5] dataset. Here, ResNet-18/34/50 and MobileNetV1 are used as the baseline models and have 1.8/3.7/4.1 GFLOPs and 572 MFLOPs.
The results in Table 2 show that SUBP achieves noticeably higher accuracy than the state-of-the-art pruning methods under the same FLOPs constraints. For example, our SUBP on ResNet50(1 × 16) with  $2 \times$  FLOPs reduction achieves  $77.6\%$  top-1 accuracy, which is  $3.1\%$ ,  $1.6\%$ ,  $0.7\%$  and  $0.2\%$  higher than Taylor[51], SCOP [59], CafeNet [57] and CHEX [29] respectively. The results on  $1 \times 32$  sparsity also demonstrate the superiority against the other methods. On the other hand, at the same target accuracy, our SUBP also achieves higher FLOPs reduction. For example, our SUBP achieves  $4 \times$  FLOPs reduction and  $76.0\%$  top-1 accuracy on ResNet50(1 × 32) compared to the SCOP, which only yields  $1.9 \times$  FLOPs reduction. We further observe that SUBP also achieves higher accuracy at a fraction of the training cost for MobileNetV1. For instance, SUBP achieves  $71.1\%$  top-1 accuracy on MobileNetV1(1 × 32, 250 epoch) under the 279M FLOPs constraint, which is  $0.5\%$  higher than MetaPruning [45](320 epoch). This is because SUBP can dynamically explore the optimal sub-model in one training pass from scratch, circumventing the expensive PPF cycles.
4.4 Results on Object Detection and Instance Segmentation
To further explore the performance of SUBP in downstream tasks, we conduct experiments on object detection and instance segmentation on the challenging COCO dataset [44]. The results are shown in Table 3 and Table 4. We adopt the classical method Faster RCNN [54] for object detection and Mask RCNN [21] for instance segmentation. We use ResNet50 with different sparse ratios and block sizes as backbone. All the experiments are conducted based on MMDetection [1]. Compared to Dense ResNet50, the SUBP can achieve competitive results, which further demonstrates its robustness and superiority on downstream computer vision tasks.
Table 3: Object detection results on COCO.
Model	Block Size	mAP
F-RCNN-R50(4.1G)	-	37.4
F-RCNN-R50(2.0G)	1 × 32	38.4
F-RCNN-R50(2.0G)	1 × 16	38.5
F-RCNN-R50(1.0G)	1 × 32	37.1
F-RCNN-R50(1.0G)	1 × 16	37.3
Table 4: Instance segmentation results on COCO.
Model	Block Size	Box mAP	Mask mAP
M-RCNN-R50(4.1G)	-	38.2	34.7
M-RCNN-R50(2.0G)	1 × 32	39.2	35.4
M-RCNN-R50(2.0G)	1 × 16	39.4	35.5
M-RCNN-R50(1.0G)	1 × 32	37.4	33.8
M-RCNN-R50(1.0G)	1 × 16	37.5	33.8
4.5 SUBP from Pre-trained Models
In order to further investigate the generality property of our approach, we apply SUBP to the model initialized with pre-trained weights. For an equitable comparison with other PPF methods, we adopt the pre-trained ResNet18 provided by the torchvision $^2$  and run SUBP for 120 epochs as the previous one did. The results in Table 5 show that our SUBP still achieves competitive top-1 and top-5 accuracy under the same FLOPs compared to the previous state-of-the-art PPF methods. Furthermore, when ResNet18 is trained for  $90 + 120$  epochs, it achieves  $69.5\%$  top-1 accuracy, which is only  $0.4\%$  lower than training it from scratch for 250 epochs in Table 2.
$^{2}$ https://pytorch.org/vision/stable/models.html
Table 5: ResNet18 starting from the pre-trained models on ImageNet dataset. "Epochs" are reported as: pre-training epochs plus all subsequent training epochs needed to obtain the final pruned model.
Model	Method	Params	FLOPs	Top-1	Top-5	Epochs
ResNet-18	Baseline	11.7M	1.81G	69.8%	89.1%	90
PFP [39]	6.6M	1.27G	67.4%	87.9%	90+180
SCOP [59]	7.1M	1.10G	69.2%	88.9%	90+140
SFP [24]	7.1M	1.04G	67.1%	87.8%	100+100
FPGM [25]	7.1M	1.04G	68.4%	88.5%	100+100
CHEX [29]	-	1.04G	69.2%	-	90+120
SUBP(1×16)	7.1M	1.04G	69.5%	89.0%	90+120
SUBP(1×32)	7.1M	1.04G	69.3%	88.9%	90+120
4.6 Deployment Efficiency Analysis
Figure 4: Network latency comparison between uniform  $1 \times N$  sparse against non-uniform and dense model with varying N and prune rates. The experiment is conducted using ResNet18 and set the input shape as (4, 3, 224, 224) on the arm platform of Apple M1 Pro CPU @ 3.20GHz with single thread (left) and two threads (right). Best view in colors.
To further explore the acceleration capacity with respect to different N and prune rates on CPUs-based platforms, we adopt TVM [2] to compile the uniform and non-uniform  $1\times \mathrm{N}$  pruned models. For a fair comparison, we also consider TVM for dense model to obtain baseline latency. We deploy the  $1\times \mathrm{N}$  pruned models to obtain network latencies on the arm platform of Apple M1 Pro CPU @ 3.20GHz. From Fig. 4, we observe  $1\times \mathrm{N}$  pruned model achieves noticeable latency reductions across various pruning rates and N. For example, when the pruning rate is greater than 0.25, the inference latencies of different N are all the better than their dense baseline under the single thread scenario. When the thread num is set to 2, the inference speed of different configurations has been improved. Due to the workload imbalance between threads in non-uniform  $1\times \mathrm{N}$ , it always lags behind its uniform counterpart. From Fig. 4, there are also two points worth noting: 1) the improvement of  $1\times \mathrm{N}$  is not as significant as a vanilla convolution in a multithreading context; 2)  $\mathrm{N} = 16$  achieves a faster inference speed than other N on M1 Pro. Therefore, optimizing multithreading inference with  $1\times \mathrm{N}$  pruning and selecting appropriate N based on suitable platforms are directions that can be further explored.
Firstly, our SUBP masks weights throughout the training and still needs dense computations. Therefore, while SUBP has advantages over methods that depend on PPF pipeline, there is still room for improving its training efficiency. Secondly, there are still missing experiments, including applying  $1 \times N$  sparsity on other types of DNNs like RNN, transformer and other tasks including object detection, natural language processing, etc.. We will design corresponding high-performance operators to improve the training efficiency of SUBP, and explore the performance of  $1 \times N$  sparsity in other types of DNNs and tasks to verify its broader applicability in our future work.
Uniform  $1 \times N$  sparsity is an important technique that allows fast inference on multithreading scenarios under multi-core architecture. This paper proposes soft uniform block pruning, SUBP, to efficiently train a uniform  $1 \times N$  sparsity network from scratch. SUBP dynamically adjusts the pruning blocks based on a periodic pruning and regrowing process, which prevents the important blocks from being prematurely pruned and keeps the model's capacity. We also present a new block pruning criterion named BPAR, which explicitly considers the mutual angular redundancy between blocks rather than "relatively less" importance only. By proposing a BPAR based block subset selection approach for pruning and an importance sampling based strategy for regrowing, we can obtain a sub-model with high accuracy by end-to-end implementation without pre-training a large model or requiring extra fine-tuning. Extensive experiments have exhibited our method can effectively reduce the FLOPs and inference latencies while achieving superior accuracy over several SOTAs.
This work is funded in part by the Key Research and Development Project of Zhejiang Province under Grant 2021C01035.
[1] Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:1906.07155, 2019.
[2] Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, et al. Tvm: An automated end-to-end optimizing compiler for deep learning. In Symposium on Operating Systems Design and Implementation (OSDI), pages 578-594, 2018.
[3] Zhuangzhi Chen, Jingyang Xiang, Yao Lu, Qi Xuan, Zhen Wang, Guanrong Chen, and Xiaoniu Yang. Rgp: Neural network pruning through regular graph with edges swapping. IEEE Transactions on Neural Networks and Learning Systems, pages 1-13, 2023.
[4] Ioana Croitoru, Simion-Vlad Bogolin, and Marius Leordeanu. Unsupervised learning of foreground object segmentation. International Journal of Computer Vision (IJCV), 127(9):1279-1302, 2019.
[5] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 248–255, 2009.
[6] Xiaohan Ding, Guiguang Ding, Yuchen Guo, and Jungong Han. Centripetal sgd for pruning very deep convolutional networks with complicated structure. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4943-4953, 2019.
[7] Xiaohan Ding, Guiguang Ding, Yuchen Guo, Jungong Han, and Chenggang Yan. Approximated oracle filter pruning for destructive cnn width optimization. In International Conference on Machine Learning (ICML), pages 1607-1616. PMLR, 2019.
[8] Xiaohan Ding, Tianxiang Hao, Jianchao Tan, Ji Liu, Jungong Han, Yuchen Guo, and Guiguang Ding. Lossless cnn channel pruning via decoupling remembering and forgetting. Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
[9] Xiaohan Ding, Tianxiang Hao, Jianchao Tan, Ji Liu, Jungong Han, Yuchen Guo, and Guiguang Ding. Resrep: Lossless cnn pruning via decoupling remembering and forgetting. In IEEE International Conference on Computer Vision (ICCV), pages 4510-4520, 2021.
[10] Xiaohan Ding, Xiangxin Zhou, Yuchen Guo, Jungong Han, Ji Liu, et al. Global sparse momentum sgd for pruning very deep neural networks. In Advances in Neural Information Processing Systems (NeurIPS), pages 6382-6394, 2019.
[11] Xuanyi Dong and Yi Yang. Network pruning via transformable architecture search. Proceedings of Advances in Neural Information Processing Systems, pages 759-770, 2019.
[12] Erich Olsen, Marat Dukhan, Trevor Gale, and Karen Simonyan. Fast sparse convnets. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14629-14638, 2020.
[13] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations (ICLR), 2019.
[14] Shangqian Gao, Feihu Huang, Weidong Cai, and Heng Huang. Network pruning via performance maximization. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9270-9280, 2021.
[63] Zihao Ye, Ruihang Lai, Junru Shao, Tianqi Chen, and Luis Ceze. Sparsetir: Composable abstractions for sparse compilation in deep learning. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, ASPLOS 2023, page 660-678, New York, NY, USA, 2023. Association for Computing Machinery.
[64] Yuxin Zhang, Mingbao Lin, Chia-Wen Lin, Jie Chen, Yongjian Wu, Yonghong Tian, and Rongrong Ji. Carrying out cnn channel pruning in a white box. IEEE Transactions on Neural Networks and Learning Systems (TNNLS), 2022.
[65] Yuxin Zhang, Mingbao Lin, ZhiHang Lin, Yiting Luo, Ke Li, Fei Chao, YONGJIAN WU, and Rongrong Ji. Learning best combination for efficient n: M sparsity. In Advances in Neural Information Processing Systems, 2022.
[66] Aojun Zhou, Yukun Ma, Junnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hongsheng Li. Learning n: M fine-grained structured sparse neural networks from scratch. In International Conference on Learning Representations (ICLR), 2021.
[67] Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. In International Conference on Learning Representations Workshop (ICLRW), 2017.
[68] Tao Zhuang, Zhixuan Zhang, Yuheng Huang, Xiaoyi Zeng, Kai Shuang, and Xiang Li. Neuron-level structured pruning using polarization regularizer. Proceedings of Advances in Neural Information Processing Systems, 2020.
[69] Zhuangwei Zhuang, Mingkui Tan, Bohan Zhuang, Jing Liu, Yong Guo, Qingyao Wu, Junzhou Huang, and Jinhui Zhu. Discrimination-aware channel pruning for deep neural networks. Proceedings of Advances in Neural Information Processing Systems, pages 883-894, 2018.
This Supplementary Material is structured as follows. We provide a formulation of our algorithm in Section B. To investigate the effectiveness of different components of our SUBP, we conduct ablation studies and provide additional experimental results in Section C. In Section D, we provide deployment results on the x86 platform of Intel(R) Xeon(R) Platinum 8260L CPU @ 2.30GHz to further explore the performance of  $1\times \mathrm{N}$  sparse on different platforms. Finally, in Section E, we discuss the societal impact of our method.
B Algorithm Formulation
Algorithm 1: Overview of the SUBP method.
1 Input: An  $L$  -layer CNN model with weights  $\mathbf{W} = \{W^i |1\leq i\leq L\}$  ; block binary mask matrices  $\mathbf{M} = \{M_{j,k}^{i}\in \{0,1\} |1\leq i\leq L,0\leq j\leq \frac{C_{i + 1}}{N} -1,0\leq k\leq C_i - 1\}$  indices of activate blocks with the top scores T; indices of regrow blocks based on importance sampling G; target prune rate p; initial regrowing factor  $\delta_0$  ; importance balance coefficient  $\lambda$  ; sampling attention balance factor  $\tau$  ; training epochs  $T_{\mathrm{total}}$  ; start and end epoch in the pruning-regrowing stage  $t_s,t_e$  ; training set  $\mathcal{D}$  .   
2 Output: A sub-model satisfying the target prune rate  $p$  , its optimal weight values  $\mathbf{W}^*$  and binary mask  $\mathbf{M}^*$  .   
3 Randomly initialize the model weights W;   
4 Initialize  $\{M_{j,k}^{i}\mid \forall i,\forall j,\forall k\}$  to 1 ;   
5 Reformat W to B according to Section 3 ;   
6 for each training epoch  $t\in [T_{\mathrm{total}}]$  do   
7 Sample a mini-batch from  $\mathcal{D}$  and update the model weights W;   
if  $t_s <   t\leq t_e$  then   
9 Reset  $\{M_{j,k}^{i}\mid \forall i,\forall j,\forall k\}$  to 1 ;   
10 Compute the importance score  $S$  of block by Eq. 4 ;   
11 Get the indices of activate blocks with the top scores by Eq. 5 ;   
12 Prune the bottom-ranking block by set  $\{M_{j,k}^{i}|k\notin \mathcal{T}_{j}^{i}\}$  to 0;   
13 Compute the importance sampling probabilities by   
 $p_{j,k}^{i} = \exp \left(\frac{S_{j,k}^{i}}{\tau}\right) / \sum_{m\notin \mathcal{T}_{j}^{i}}\exp \left(\frac{S_{j,m}^{i}}{\tau}\right)$    
14 Compute the regrowing factor by Eq. 6 ;   
15 Get the indices of regrow blocks based on importance sampling by   
 $\mathcal{G}_j^i =$  Multinomial(  $\{p_{j,k}^{i}|k\notin \mathcal{T}_{j}^{i}\}$  ,  $\delta_tC_i)$  without replacement ;   
16 Regrow the blocks by resetting  $\{M_{j,k}^{i}|k\in \mathcal{G}_{j}^{i}\}$  to 1 ;
C Ablation Analysis
Table 6: Compare different design choices in the regrowing stages of the SUBP method. All the experiments are based on the TinyImageNet with ResNet18(1×32). The random baseline is 57.0%.
Regrowing factor
Design choices Top-1	δ0= 0.1 57.6%	δ0= 0.2 58.4%	δ0= 0.3 57.9%	δ0= 0.4 58.0%	Full 58.5%
Decay scheduler for block regowing
Design choices Top-1	Default 58.3%	Constant 57.5%	Linear decay 58.4%	Cosine decay 58.3%
In Table 6, we investigate the effectiveness of different design choices in our block regrowing stage. All the experiments are based on the TinyImageNet with ResNet18(1×32). Compared to the random
baseline with  $57.0\%$  top-1 accuracy, our SUBP achieves consistent improvement under the different settings.
We find that regrowing factor  $\delta_0$  significantly impacts the final quality of the model. Intuitively, a larger regrowing factor can provide a more extensive sampling space during training and retain the model's capacity to a greater extent. However, a sizeable regrowing factor may also cause drastic sub-model structure changes, affecting stability during training. As shown, the accuracy is improved by  $0.8\%$  as the  $\delta_0$  increases from 0.1 to 0.2. When  $\delta_0$  increases again, the model's accuracy drops until  $\delta_0$  is the full model size. This suggests that the relationship between the regrowing factor and the final quality of the model is varied, and selecting an appropriate regrowing factor in specific circumstances can improve the final quality.
We also investigate the decay scheduler for the block regrowing stage. We compare several decay schedulers, including default (Eq. 6), constant, linear, and cosine. The experiments show SUBP has good robustness to different decay schedulers, as default, linear, and cosine decay schedulers all show similar performance. With a decay scheduler, the sampling space can be gradually decreased, and the sub-model under training can converge stably.
Table 7: Ablation study of block size and epochs. All the experiments are based on the ImageNet with ResNet18.
Method	FLOPs	Accuracy	Epochs
DMCP	1.04G	69.0%	150
CHEX	1.03G	69.6%	250
SUBP (1×4)	1.03G	70.4%	250
SUBP (1×8)	1.03G	70.2%	250
SUBP (1×16)	1.03G	68.7%	100
SUBP (1×16)	1.03G	69.2%	150
To achieve a fairer and fuller comparison with previous methods, we conduct ablation study with the same epoch and different block sizes. It can be seen in Table 7 that both longger training epochs and finer pruning granularity can improve the final results. We can weight trade offs in practical applications.
D Deployment on x86 Platform
Figure 5: Network latency comparison between uniform  $1 \times N$  sparse against non-uniform and dense model with varying N and prune rates. The experiment is conducted using ResNet18 and set the input shape as (4, 3, 224, 224) on the x86 platform of Intel(R) Xeon(R) Platinum 8260L CPU @ 2.30GHz with single thread (left) and two threads (right). Best view in colors.
In order to further explore the performance of  $1\times \mathrm{N}$  sparse DNNs on different platforms, as shown in Fig. 5, we also conducted corresponding experiments on the x86 platform of Intel(R) Xeon(R) Platinum 8260L CPU @ 2.30GHz and obtain the similar results in general: 1) The gain of vanilla convolution in multithreading scenarios is much greater than that of  $1\times \mathrm{N}$  sparse convolution. 2) The
inference speed of uniform  $1 \times \mathrm{N}$  is slightly faster than that of non-uniform in the case of multithreading, indicating the importance of workload balance again. However, unlike the performance on the arm platform of Apple M1 Pro CPU @ 3.20GHz, the  $1 \times \mathrm{N}$  sparse DNNs are significantly accelerated when  $\mathrm{N}$  is set to 16 and 32 on the Platinum 8260L CPU @ 2.30GHz. We can also notice that in most cases,  $\mathrm{N} = 32$  achieves a fast inference speed.
E Societal Impact
Our method can reduce the computational overhead of training and inferencing stages while achieving satisfactory accuracy on modern CNN models. This can facilitate the application of CNN models on edge devices and is of high value for the community and society to realize Green AI. At the same time, our  $1\times N$  sparse DNNs are based on new sparse operators, which can promote the progress of related hardware and algorithms to a certain extent.