SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 3: Test accuracy (%) of the sparse VGG-19 and ResNet-20/50 on CIFAR-10/100.
Dataset	CIFAR-10	CIFAR-100
Sparsity	90%	95%	98%	90%	95%	98%
VGG-19 (Dense)	93.85±0.05	93.85±0.05	93.85±0.05	73.43±0.08	73.43±0.08	73.43±0.08
SynFlow [53]	93.35	93.45	92.24	71.77	71.72	70.94
GraSP [56]	93.30	93.04	92.19	71.95	71.23	68.90
SNIP [28]	93.63	93.43	-	72.84	71.83	-
Chase+GraSP (Sc=0.5)	94.06±0.22	93.88±0.06	93.89±0.20	73.17±0.09	72.81±0.11	71.66±0.15
Chase+SNIP (Sc=0.5)	94.83±0.06	95.08±0.14	-	78.26±0.26	77.16±0.04	-
Deep-R [1]	90.81	89.59	86.77	66.83	63.46	59.58
SET [39]	93.61±0.13	93.09±0.25	91.81±0.04	72.58±0.12	71.48±0.12	69.04±0.15
RigL [10]	93.60±0.09	93.05±0.06	91.95±0.15	72.92±0.31	71.85±0.53	69.57±0.24
MEST [59]	93.61±0.36	93.46±0.41	92.30±0.44	72.52±0.37	71.21±0.41	69.02±0.34
Chase (Sc=0)	94.02±0.13	93.89±0.12	93.60±0.05	73.54±0.12	73.05±0.25	72.19±0.33
Chase (Sc=0.5)	94.03±0.11	93.84±0.08	93.69±0.03	73.43±0.12	73.04±0.22	71.85±0.18
ResNet-50 (Dense)	94.75±0.01	94.75±0.01	94.75±0.01	78.23±0.18	78.23±0.18	78.23±0.18
SynFlow [53]	92.49	91.22	88.82	73.37	70.37	62.17
SNIP [28]	92.65	90.86	-	73.14	69.25	-
GraSP [56]	92.47	91.32	88.77	73.28	70.29	62.12
Chase+SNIP (Sc=0.5)	93.99±0.09	93.89±0.10	-	73.44±0.02	72.80±0.05	-
Chase+GraSP (Sc=0.5)	94.78±0.35	94.71±0.07	94.36±0.15	77.70±0.24	77.65±0.22	75.74±0.24
Deep-R [1]	91.62	89.84	86.45	66.78	63.90	58.47
SET [39]	94.65±0.01	94.05±0.06	92.98±0.18	76.14±0.54	75.90±0.19	73.21±0.06
RigL [10]	94.42±0.17	94.22±0.23	93.20±0.08	77.18±0.42	76.50±0.26	74.84±0.13
Chase (Sc=0)	94.95±0.02	94.87±0.02	94.15±0.17	78.11±0.11	78.14±0.28	76.88±0.31
Chase (Sc=0.5)	94.88±0.03	94.85±0.18	94.20±0.18	77.52±0.30	77.48±0.62	77.03±0.29
ResNet-20 (Dense)	92.55±0.02	92.55±0.02	92.55±0.02	68.65±0.19	68.65±0.19	68.65±0.19
SNIP [28]	88.06±0.07	84.21±0.33	74.61±0.40	54.40±0.09	42.45±0.65	24.55±0.56
GraSP [56]	88.35±0.12	84.95±0.30	78.25±0.22	55.49±0.08	45.96±0.15	30.67±0.94
SET [39]	90.16±0.09	87.70±0.09	83.41±0.04	62.08±0.16	54.77±0.74	43.70±0.92
RigL [10]	89.82±0.10	87.44±0.33	79.16±0.96	60.49±0.17	52.97±0.23	31.94±1.52
Chase (Sc=0)	90.43±0.16	88.65±0.29	85.26±0.29	62.18±0.05	57.38±0.41	47.06±0.65
Chase (Sc=0.5)	89.98±0.45	88.65±0.02	85.24±0.18	60.88±0.19	55.78±0.37	46.91±0.41
implementation details for Chase in Appendix B. To show the superior performance of Chase on unstructured and structured sparsity, we report two variants of Chase: Chase  $(S_{c} = 0)$  represents the unstructured version without channel pruning and Chase  $(S_{c} = 0.5)$  stands for the structured version with  $50\%$  channel-level sparsity.
4.1 Comparison with off-the-shelf SOTA DST
CIFAR-10/100. Our method is naturally versatile and can be applied to both static sparse training and dynamic sparse training regimes. Therefore, for each model, we categorize the results into two groups and report the results in Table 3.
1 Chase dramatically improves the performance of SST. As shown in the upper panel of each group, Chase significantly boosts the accuracy (up to  $13.62\%$  for GraSP with ResNet-50 on CIFAR-100) of SST methods like SNIP and GraSP. 2 Chase  $(S_{c} = 0)$  establishes a new state-of-the-art performance bar for unstructured sparse training. Compared with the SOTA DST methods such as RigL and MEST, we clearly see that Chase  $(S_{c} = 0)$  universally outperforms all the presented DST methods by a large margin. Specifically, Chase  $(S_{c} = 0)$  achieves  $3.67\%$  and  $3.15\%$  performance gains compared with SET on ResNet-50 and VGG-19. We also notice that the performance gain on CIFAR-100 is larger than the ones on CIFAR-10, which is as expected since CIFAR-100 has a larger improvement space than CIFAR-10. 3 Chase  $(S_{c} = 0.5)$ , with only  $50\%$  channels remaining, matches the performance of its unstructured variant, demonstrating the promise of the unstructured DST can be favorably transferred to the structured regime.
ImageNet. We reported the results on ImageNet in Table 4. Again, Chase  $(S_{c} = 0)$  dominates the performance in the unstructured sparsity regime, achieving  $1.7\%$  and  $2.12\%$  accuracy improvements over RigL and MEST, respectively. While the accuracy of Chase  $(S_{c} = 0.4)$  slightly decreases by  $0.67\%$  compared to Chase  $(S_{c} = 0)$ , it still outperforms RigL by a good margin  $(1.03\%)$ , while enjoying a  $1.5\times$  real inference speedup on common GPU.
Table 4: Test accuracy (\%) of sparse ResNet-50 on ImageNet trained with 100 epochs and 150 epochs  $(1.5\times)$ . Training FLOPs of sparse training methods are normalized with the FLOPs used to train a dense model. "GPU-supported FLOPs" refers to the real FLOPs that are required to calculate on a common GPU which usually does not support irregular sparsity patterns.
Method	Top-1 Accuracy	Theoretical FLOPs (Train)	Theoretical FLOPs (Test)	GPU-Supported FLOPs (Test)	TOP-1 Accuracy	Theoretical FLOPs (Train)	Theoretical FLOP (Test)	GPU-Supported FLOPs (Test)
ResNet-50 (Dense)	76.8±0.09	1x (3.2e18)	1x (8.2e9)	1x (8.2e9)	76.8±0.09	1x (3.2e18)	1x (8.2e9)	1x (8.2e9)
Sparsity	80%	90%
SET [39]	72.9±0.39	0.23×	0.23×	1.00×	69.6±0.23	0.10×	0.10×	1.00×
DSR [40]	73.3	0.40×	0.40×	1.00×	71.6	0.30×	0.30×	1.00×
SNFS [5]	75.2±0.11	0.61×	0.42×	1.00×	72.9±0.06	0.50×	0.24×	1.00×
RigL [10]	75.1±0.05	0.42×	0.42×	1.00×	73.0±0.04	0.25×	0.24×	1.00×
MEST [59]	75.39	0.23×	0.21×	1.00×	72.58	0.12×	0.11×	1.00×
RigL-ITOP [35]	75.84±0.05	0.42×	0.42×	1.00×	73.82±0.08	0.25×	0.24×	1.00×
Chase (Sc=0)	75.87	0.37×	0.34×	1.00×	74.70	0.24×	0.21×	1.00×
Chase (Sc=0.3)	75.62	0.39×	0.36×	0.75×	74.35	0.25×	0.22×	0.74×
Chase (Sc=0.4)	75.27	0.39×	0.37×	0.68×	74.03	0.26×	0.23×	0.67×
MEST1.5×	75.73	0.40×	0.21×	1.00×	75.00	0.20×	0.11×	1.00×
Chase1.5× (Sc=0)	76.67	0.55×	0.34×	1.00×	75.77	0.36×	0.21×	1.00×
Chase1.5× (Sc=0.3)	76.23	0.57×	0.36×	0.75×	75.20	0.37×	0.22×	0.74×
Chase1.5× (Sc=0.4)	76.00	0.59×	0.37×	0.68×	74.87	0.38×	0.23×	0.67×
When increasing the training time to  $1.5 \times$  (150 epochs), Chase also demonstrates a promising scaling trend. Chase ( $S_{c} = 0$ ) matches the dense performance with only  $0.55 \times$  training FLOPs and consistently outperforms the off-the-shelf best DST method, MEST. Again, the promising results of Chase ( $S_{c} = 0$ ) can be effectively translated to channel-level sparsity. Chase ( $S_{c} = 0.4$ ) is  $1.5 \times$  faster than MEST, while still performing on par or even better than MEST.
Real Inference Speedups. We further compare the actual inference throughput and latency of our model against RigL in Table 5. All results are averaged from 100 individual runs with one NVIDIA 2080TI GPU in float32 on PyTorch. We set the batch size to 128 for CIFAR-100 and 2 for ImageNet, when evaluating the latency. We empirically find that pruning the skip connection leads to a significant accuracy drop while providing benefits on speedups. Therefore, the standard Chase keeps the skip connection layers untouched for optimal accuracy. To fully unleash Chase's potential on real inference speedups, we also provide another variant of Chase that prunes skip connection layers, dubbed Chase (prune skip). Compared with SoTA RigL, Chase (prune skip) is able to prune  $50\%$  channels with ResNet-50 on ImageNet, leading to a notable  $68\%$  of throughput gain, while only losing  $0.29\%$  accuracy. Even without pruning skip connections, our model is about to provide  $31\%$  throughput speedups, while outperforming RigL by  $0.39\%$ .
Table 5: Inference throughput and latency. The best results are marked in bold.
Method	Dataset	Model	\(s_p\)	\(S_c\)	Accuracy (%) (↑)	Throughput (↑)	Latency (ms) (↓)
RigL [10]	CIFAR-100	VGG-19	0.9	0.0	72.92±0.31	15274.31	8.42
Chase	CIFAR-100	VGG-19	0.9	0.5	73.43±0.12	24981.77 (64%↑)	5.37 (36%↓)
RigL [10]	CIFAR-100	ResNet-50	0.9	0.0	77.18±0.42	3095.13	44.23
Chase	CIFAR-100	ResNet-50	0.9	0.5	77.52±0.30	3958.67 (28%↑)	34.76 (21%↓)
RigL [10]	ImageNet	ResNet-50	0.9	0.0	73.00±0.04	59.50	35.79
Chase	ImageNet	ResNet-50	0.9	0.5	73.39±0.04	78.19 (31%↑)	27.55 (23%↓)
Chase (prune skip)	ImageNet	ResNet-50	0.9	0.5	72.71±0.03	99.97 (68%↑)	21.55 (40%↓)
Chase	ImageNet	ResNet-50	0.9	0.4	74.03±0.03	71.54 (20%↑)	30.13 (16%↓)
Chase (prune skip)	ImageNet	ResNet-50	0.9	0.4	73.24±0.03	87.38 (47%↑)	24.53 (31%↓)
4.2 Extensive Analysis
Performance under different channel sparsity. To investigate the performance of different channel sparsity with the same parameter count, we maintain  $2\%$ ,  $5\%$  parameters and prune the model to different channel sparsity ranging from  $20\%$  to  $70\%$ . The results are reported in Figure 3. The same training scheme is adopted as Section 4.1. Two DST baselines, RigL and SET are adopted for comparison. Not surprisingly, the more channels remain the better performance of the model archives. Notably, in all settings, Chase archives better performance than the baselines with just  $50\%$  channels, and Chase outperforms SET and RigL with just  $30\%$  channels on ResNet-50 at  $98\%$  sparsity.
Figure 3: Performance of Chase under different channel sparsity. For Rigl and SET, we keep the channels un-pruned as baselines.
Figure 4: Ablation Study of Chase. GACP denotes gradual amenable channel pruning (50% channel sparsity), SM indicates soft memory bound, GE represents global parameter exploration.
Effect of the channel pruning frequency. We also study how the channel pruning frequency  $\Delta T$  affects Chase's performance. For all experiments, we fixed the ending time  $\tau_{stop}$  for gradual amenable channel pruning as 130 epochs, the total training epochs  $\tau_{total}$  as 160 epochs and the minimum channel ratio factor as  $\beta$  as 0.5, while altering  $\Delta T$  to 1000, 4000, 8000, and 16000 iterations. We report the results in Appendix A.4. Overall, the largest  $\Delta T$  16000 leads to worse performance. This observation is as expected, as we aim to achieve the same channel sparsity and larger  $\Delta T$  results in more removed channels in each punning operation. Consequently, larger performance degradation will be introduced during each pruning which could degrade the training stability.
Ablation study. In Figure 4, we study the effectiveness of different components in Chase, namely, the soft memory constraint (SM) and global parameter exploration (GE) on CIFAR-10/100 with ResNet-50 and VGG-19. We denote the RigL as our baseline, as RigL applies magnitude-based pruning and gradients-based growth like Chase. We apply the same training recipe as described in Section 4.1. Gradually amenable channel pruning safely removes  $50\%$  channels from RigL, while only suffering from minor or even no performance degradation. As for SM and GE, we found these techniques all bring universal performance improvements. Surprisingly, adding SM results in a  $1.26\%$  accuracy increase on CIFAR-100 with ResNet-50 at  $98\%$  sparsity. With GE, we can obtain a more optimal layer-wise ratio, which also consistently improves the accuracy from SM.
Recently, as the financial and environmental costs of model training grow exponentially [50; 43], endeavors start to pursue training efficiency by investigating training sparse neural networks from scratch. Most Sparse training works can be divided into two categories, static sparse training, and dynamic sparse training. Static sparse training determines the structure of the sparse network at the initial stage of training by using certain pre-defined layer-wise sparsity ratios [38; 39; 10; 33].
Dynamic sparse training is designed to reduce the computation as well as memory footprint during the whole training phase. It trains a sparse neural network from scratch while allowing the sparse mask to be updated during training. SET [39] update sparse mask at the end of each training epoch by magnitude-based pruning and random growing. DSR [40] develops a dynamic reparameterization method that allows parameter reallocation during dynamic mask updating. DeepR [1] combines dynamic sparse parameterization with stochastic parameter updates for training. RigL [10] and
SNFS [5] propose to uses gradient information to grow weights. ITOP [35] studies the underlying mechanism of DST and discovers that the benefits of DST come from searching across time all possible parameters. GraNet [31] introduces the concept of "pruning plasticity" and quantitatively studies the effect of pruning throughout training. MEST [59] proposes a memory-friendly training framework that could perform fast execution on edge devices. AC/DC [44] co-trains the sparse and dense models to return both accurate sparse and dense models. [23] dynamically activates top-K parameters during forward-pass while keeping a larger number of parameters updated in backward-pass to get rid of dense calculation of gradient. Top-KAST [23] preserves constant sparsity throughout training in both the forward and backward passes. Built upon Top-KAST, Powerpropagation [47] leaves the low-magnitude parameters largely unaffected by learning, achieving strong results. CHEX [19] applied dynamic prune and regrow channels strategies to avoid pruning important channels prematurely. Very recently, SLaK [32] leverages dynamic sparse training to successfully train intrinsically sparse  $51 \times 51$  kernels, which performs on par with or better than advanced Transformers. A concurrent work [21] discovers that a tiny fraction of channels (up to  $4.3\%$ ) of RigL become totally sparse after training.
To enable acceleration of sparse training in practice, [34] build a truly sparse framework based on SciPy sparse matrices [55] that enables efficient sparse evolutionary training [39] in CPU. [6] fulfill group-wise DST on Graphcore IPU [24] and demonstrate its efficacy on pre-training BERT. Moreover, some previous work develops sparse kernels [12; 9] to directly support unstructured sparsity in GPU. DeepSparse [27] deploys large-scale BERT-level and YOLO-level sparse models on CPU.
In this paper, we have presented Chase, a new sparse training approach that seamlessly translates the promise of unstructured sparsity into channel-level sparsity, while performing on par or even often better than state-of-the-art DST approaches. Extensive experiments across various network architectures including VGG-19 and ResNet-50 on CIFAR-10/100 and ImageNet demonstrated Chase can achieve better performance with  $1.2 \times \sim 1.7 \times$  real inference speedup on common GPU devices while performing on par or even better than unstructured SoTA. The results in this paper strongly challenge the common belief that sparse training typically suffers from limited acceleration support in common hardware, opening doors for future work to build more efficient sparse neural networks.
S. Liu and Z. Wang are in part supported by the NSF AI Institute for Foundations of Machine Learning (IFML). Part of this work used the Dutch national e-infrastructure with the support of the SURF Cooperative using grant no. NWO2021.060, EINF-2694 and EINF-2943/L1. It is also supported by the NSF CCF-2312616. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of NSF.
[1] G. Bellec, D. Kappel, W. Maass, and R. Legenstein. Deep rewiring: Training very sparse deep networks. In International Conference on Learning Representations, 2018. 7, 9
[2] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020. 1
[3] T. Chen, X. Chen, X. Ma, Y. Wang, and Z. Wang. Coarsening the granularity: Towards structurally sparse lottery tickets. arXiv preprint arXiv:2202.04736, 2022. 2
[4] T.-W. Chin, R. Ding, C. Zhang, and D. Marculescu. Legr: Filter pruning via learned global ranking. 2019. 17
[56] C. Wang, G. Zhang, and R. Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations, 2020. 2, 7
[57] Z. You, K. Yan, J. Ye, M. Ma, and P. Wang. Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks. Advances in neural information processing systems, 32, 2019. 17
[58] J. Yu and T. Huang. Autoslim: Towards one-shot architecture search for channel numbers. arXiv preprint arXiv:1903.11728, 2019. 17
[59] G. Yuan, X. Ma, W. Niu, Z. Li, Z. Kong, N. Liu, Y. Gong, Z. Zhan, C. He, Q. Jin, et al. Mest: Accurate and fast memory-economic sparse training framework on the edge. Advances in Neural Information Processing Systems, 34:20838-20850, 2021. 6, 7, 8, 10
[60] A. Zhou, Y. Ma, J. Zhu, J. Liu, Z. Zhang, K. Yuan, W. Sun, and H. Li. Learning n:m fine-grained structured sparse neural networks from scratch. In International Conference on Learning Representations, 2021. 2
[61] M. H. Zhu and S. Gupta. To prune, or not to prune: Exploring the efficacy of pruning for model compression, 2018. 6
A Remaining Experimental Analysis
A.1 Effect of the Initial Sparsity
Chase starts from a subnetwork with unstructured sparsity to produce channel-level sparsity during one end-to-end training process. Here we fix the target channel-level sparsity  $S_{c}$  as 0.4 and study how the initialized unstructured sparsity impacts the model performance. The results are reported in Figure 5. It could be seen that, in general, initialization with more parameters leads to better performance. Counter-intuitively, the best accuracy is achieved using 0.6 unstructured sparsity, which is  $0.26\%$  higher than initialized as a dense model.
Figure 5: Performance under different initial unstructured sparsity on ResNet-18, CIFAR-100.
A.2 Effect of the Minimum Channel Ratio Limitation  $\beta$
We alter the ratio of the minimum channel ratio  $\beta$  to 0.2, 0.5, 0.8 and show the performance in Table 6. The channel pruning frequency  $\Delta T$  is fixed at 4000 iterations. Apparently, large ratio  $\beta = 0.8$  archives the worst performance while  $\beta = 0.2$  outperforms other settings in 4 out of 6 cases. In view of the fact that smaller  $\beta$  provide more channel exploration space.
Table 6: Test accuracy (%) on CIFAR-100 of Chase at  $50\%$  channel-wise sparsity using different minimum layer limitation factor  $\beta$ . The best results are marked in bold.
Minimum layer ratio	Sparity
90%	95%	98%
VGG-19
0.20	73.45±0.27	72.98±0.32	71.69±0.21
0.50	73.16±0.06	72.39±0.17	71.74±0.06
0.80	72.26±0.24	72.20±0.27	71.28±0.26'
ResNet-50
0.20	77.47±0.40	77.43±0.36	76.68±0.27
0.50	77.54±0.40	77.31±0.34	76.64±0.26
0.80	76.76±0.66	77.26±0.73	76.67±0.29
A.3 Ablation of Gradual Amenable Channel Pruning
Here, we perform a more university ablation study of Gradual Amenable Channel Pruning (GACP) on CIFAR10/100, ResNet-50 and VGG-19, with RigL and SET. The results are reported in Table 7. Surprisingly, GACP brings performance increases in most cases. To be specific, we found that GACP could boost the performance of RigL in 9 out of 12 cases and output SET in 10 out of 12 cases with just  $50\%$  remaining channels.
A.4 Effect of the Channel Pruning Frequency
In this Appendix, we study how the channel pruning frequency  $\Delta T$  affects Chase's performance. For all experiments, we fixed the ending time  $\tau_{stop}$  for gradual amenable channel pruning as 130 epochs, the total training epochs  $\tau_{total}$  as 160 epochs and the minimum channel ratio factor as  $\beta$  as 0.5, while altering  $\Delta T$  to 1000, 4000, 8000, and 16000 iterations. We report the results in Table 8. Overall, the largest  $\Delta T$  16000 leads to worse performance. This observation is as expected, as we aim to achieve the same channel sparsity and larger  $\Delta T$  results in more removed channels in each
Table 7: Ablation of gradual amenable channel pruning (GACP). The best results are marked in bold.
Dataset	CIFAR-10	CIFAR-100
Sparsity	90%	95%	98%	90%	95%	98%
VGG-19	SET [39]	93.61±0.13	93.09±0.25	91.81±0.04	72.58±0.12	71.48±0.12	69.04±0.15
SET+GACP (Sc=0.5)	93.78±0.16	93.56±0.05	92.66±0.10	72.93±0.18	71.89±0.20	70.03±0.13
RigL [10]	93.60±0.09	93.05±0.06	91.95±0.15	72.92±0.31	71.85±0.53	69.57±0.24
RigL+GACP (Sc=0.5)	93.91±0.14	93.70±0.06	92.80±0.06	72.97±0.09	71.80±0.17	69.83±0.20
ResNet-50	SET [39]	94.65±0.01	94.05±0.06	92.98±0.18	76.14±0.54	75.90±0.19	73.21±0.06
SET+GACP (Sc=0.5)	94.52±0.17	94.51±0.19	93.23±0.13	76.75±0.47	75.67±0.64	73.86±0.12
RigL [10]	94.42±0.17	94.22±0.23	93.20±0.08	77.18±0.42	76.50±0.26	74.84±0.13
RigL+GACP (Sc=0.5)	94.45±0.10	94.41±0.13	93.50±0.16	76.86±0.56	76.33±0.63	74.92±0.27
punning operation. Consequently, larger performance degradation will be introduced during each pruning which could degrade the training stability.
Table 8: Test accuracy  $(\%)$  on CIFAR-100 of Chase at  $50\%$  channel-wise sparsity using different channel pruning frequency  $\Delta T$ . The best results are marked in bold.
ΔT(Iterations)	Sparsity
90%	95%	98%
VGG-19
1000	73.07±0.26	72.72±0.1	71.69±0.35
4000	73.16±0.06	72.39±0.17	71.74±0.06
8000	73.15±0.23	72.81±0.07	71.94±0.13
16000	72.88±0.24	72.66±0.04	71.79±0.43
ResNet-50
1000	77.52±0.30	77.48±0.62	77.03±0.29
4000	77.54±0.40	77.31±0.34	76.64±0.26
8000	77.46±0.47	77.49±0.47	76.74±0.27
16000	77.02±0.25	76.96±0.37	76.77±0.35
B Implementation Details of Chase
In this appendix, we report the implementation details for Chase, including total training time ( $\tau_{total}$ ), exploration stop time ( $\tau_{stop}$ ), gradual channel pruning frequency ( $\Delta T$ ), parameter update frequency  $\Delta T_p$ , minimum layer limitation factor ( $\beta$ ), learning rate (LR), batch size (BS), learning rate drop (LR Drop), weight decay (WD), SGD momentum (Momentum), sparse initialization (Sparse Init), target sparsity ( $s_p$ ), target channel-wise sparsity ( $S_c$ ), etc.
Table 9: Implementation hyperparameters of Chase in Table 3, on CIFAR-10/100.
Model	τtotal(epochoes)	τstop(epochoes)	ΔT(iterations)	ΔTp(iterations)	β	BS	LR	LR Drop, Epochos	Optimizer	Momentum	WD	Sparse Init
VGG-19	160	130	8000	1000	0.2	128	0.1	10x, [80, 120]	SGD	0.9	5e-4	ERK
ResNet-50	160	130	1000	1000	0.5	128	0.1	10x, [80, 120]	SGD	0.9	5e-4	ERK
Table 10: Implementation hyperparameters of Chase in Table 4, on ImageNet.
Model	τtotal(epoche)	τstop(epoche)	ΔT(iterations)	ΔTp(iterations)	β	BS	LR	LR Drop	Optimizer	Momentum	WD	Sparse Init
ResNet-50	100	80	1000	1000	0.2	512	0.512	Cosine Decay	SGD	0.9	1e-4	ERK
ResNet-50	150	120	1000	1500	0.2	512	0.512	Cosine Decay	SGD	0.9	1e-4	ERK
Table 11: Implementation hyperparameters of Chase in Table 13, on ImageNet.
Name	\( {s}_{p} \)	\( {S}_{c} \)	Model	\( {\tau }_{\text{total }} \) (epochs)	\( {\tau }_{\text{stop }} \) (epochs)	\( {\Delta T} \) (iterations)	\( \Delta {T}_{p} \) (iterations)	\( \beta \)	BS	LR	LR Drop	Optimizer	Momentum	WD	Sparse Init
Chase-1	80%	40%	ResNet-50	250	170	1000	2500	0.2	512	0.512	Cosine Decay	SGD	0.9	1e-4	ERK
Chase-2	90%	40%	ResNet-50	250	170	1000	2500	0.2	512	0.512	Cosine Decay	SGD	0.9	1e-4	ERK
C Real Inference Speedups
We report the real inference latency and throughput of Chase on various sparsity in Table 12.