SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 12: Real inference latency and throughput of Chase on the ResNet-50/ImageNet benchmark. The best results are marked in bold.
Method	sp	Sc	Accuracy (%) (↑)	Throughput (↑)	Latency (ms) (↓)
Chase	0.9	0.2	74.40	69.30	30.89
Chase	0.9	0.3	74.35	70.07	30.39
Chase	0.9	0.4	74.03	71.54	30.13
Chase	0.9	0.5	73.39	78.19	27.55
Chase	0.9	0.6	72.85	82.98	26.09
Chase	0.9	0.7	71.98	86.26	25.10
Chase (prune skip)	0.9	0.2	74.15	72.57	29.38
Chase (prune skip)	0.9	0.3	74.06	78.30	27.33
Chase (prune skip)	0.9	0.4	73.24	87.38	24.53
Chase (prune skip)	0.9	0.5	72.71	99.97	21.55
Chase (prune skip)	0.9	0.6	71.62	107.99	20.03
Chase (prune skip)	0.9	0.7	67.15	123.30	17.56
Chase	0.8	0.2	75.82	66.39	32.05
Chase	0.8	0.3	75.62	69.30	30.81
Chase	0.8	0.4	75.27	73.32	29.22
Chase	0.8	0.5	74.76	76.68	28.05
Chase	0.8	0.6	73.77	80.51	26.81
Chase	0.8	0.7	72.88	86.12	25.15
Chase (prune skip)	0.8	0.2	75.27	72.47	29.38
Chase (prune skip)	0.8	0.3	74.96	78.60	27.20
Chase (prune skip)	0.8	0.4	74.58	87.91	24.38
Chase (prune skip)	0.8	0.5	73.53	98.43	21.86
Chase (prune skip)	0.8	0.6	71.70	104.82	20.58
Chase (prune skip)	0.8	0.7	67.53	123.46	17.57
D Comparisons with SOTA Channel Pruning Methods
We further compare Chase with various state-of-the-art channel pruning approaches in Table 13. It is encouraging to see that Chase performs on par with state-of-the-art SOTA channel pruning approaches, such as Group Fisher [30], CafeNet-R [51], and CHIP [52], without the need for the costly dense pretraining step. The implementation details are reported in Table 13.
Table 13: Comparison with state-of-the-art channel pruning methods on popular benchmark: ResNet-50 on ImageNet.
Methods	FLOPs	Top-1	Epochs
GBN [57]	2.4G	76.2%	350
LEGR [4]	2.4G	75.7%	-
FPGM [16]	2.4G	75.6%	200
TAS [7]	2.3G	76.2%	240
Hrank [29]	2.3G	75.0%	570
SCOP [54]	2.2G	76.0%	230
CHIP [52]	2.2G	76.3%	-
Group Fisher [30]	2.0G	76.4%	-
AutoSlim [58]	2.0G	75.6%	-
Uniform	2.0G	75.1%	300
Random	2.0G	74.6%	300
CafeNet-R [51]	2.0G	76.5%	300
Chase-1	1.5G	76.6%	250
Uniform	1.0G	73.1%	300
Random	1.0G	72.2%	300
Group Fisher [30]	1.0G	73.9%	-
CafeNet-R [51]	1.0G	74.9%	300
CafeNet-E [51]	1.0G	75.3%	300
Chase-2	0.9G	75.7%	250
E Existence of Sparse Amenable Channel in Various Settings
To demonstrate the broad prevalence of sparse amenable channels across diverse architectures and datasets, we have evaluated their ratio in multiple scenarios, including ResNet-32/VGG-16 on CIFAR-100, ResNet-50/ViT-small on ImageNet, an MLP model on CIFAR-10.
The MLP model consists of two hidden layers, each with 512 neurons. For the ViT small model, we focus our attention on the neurons within the MLP layers that exhibit suitability for pruning. In all cases, the sparse amenable channels are identified by Unmasked Mean Magnitude (UMM), with the threshold,  $v$ , set to  $20\%$ .
The results, shown in the corresponding tables, underline the consistent presence of sparse amenable channels across various architectures and datasets, reinforcing the argument that the phenomenon is both significant and widespread.
Table 14: Sparse amenable channel portion during training on various settings
Settings	Layer	10 Epoch	20 Epoch	40 Epoch	100 Epoch
ResNet-32/CIFAR-100	blocks.3 conv1	0.23	0.38	0.38	0.63
blocks.6 conv1	0.16	0.19	0.28	0.32
VGG-16/CIFAR-100	features.0	0.11	0.22	0.30	0.39
features.7	0.19	0.20	0.17	0.65
Settings	Layer	5 Epoch	10 Epoch	20 Epoch	50 Epoch
ResNet-50/ImageNet	layer3.1.conv2	0.14	0.18	0.21	0.33
layer4.1.conv1	0.44	0.49	0.49	0.56
ViT-Small/ImageNet	blocks.0.mlp.fc1	0.11	0.14	0.15	0.31
blocks.8.mlp.fc1	0.20	0.22	0.21	0.33
Settings	Layer	5 Epoch	10 Epoch	20 Epoch	50 Epoch
MLP Model/Cifar10	fc1	0.21	0.21	0.30	0.37
fc2	0.36	0.42	0.49	0.63
F Addressing the Memory Limitation of Global Parameter Exploration
During global parameter exploration, directly loading all the parameters for gradients/magnitude sorting is memory-consuming. Inspired by [40], we layer-wisely select parameters with the largest gradients for growth and the lowest magnitude for pruning by an adaptive global threshold  $H$ , until reaching the target sparsity.  $H$  is determined by a set point negative feedback loop to maintain an approximate parameter amount during each reallocation step, as reported below.
Algorithm 2: Overview of Global Parameter Exploration
Input: Network with sparse weight  $\theta_{\mathrm{s}}$  , target sparsity  $s_p$  , current sparsity  $s_t$  , prune magnitude threshold  $H_{p}$  , grow gradient threshold  $H_{g}$  , threshold incremental value  $H_{i}$  , sparsity tolerance  $s_\delta$    
Output: A sparse model  $\theta_{\mathrm{s}}$  satisfying the target sparsity  $s_p$  . Initialize a grow threshold  $H_{g}$  Begin global parameter growing while not  $s_p + s_\delta >s_t > s_p - s_\delta$  do for each sparse tensor  $\theta_{\mathrm{s}}^l$  of layer  $l$  do  $\begin{array}{rlr}{\big(\theta_{\mathrm{s}}^{l},gl\big)\leftarrow\mathrm{grow\_by\_threshold}(\theta_{\mathrm{s}}^{l},H_{g})}&{\triangleright g_{l}}&{\mathrm{is~the~number~of~pruned~weights~in~layer~l}}\\ {G\leftarrow\sum_{i}g_{i},s_{t}\leftarrow\mathrm{calculate\_current\_sparsity}(G)}&{\triangleright G\mathrm{~is~total~number~of~grown~weights}}\\ {\mathrm{in~all~layers}}&{}&{}\\ {\mathrm{if~}s_t <   s_p - s_\delta \mathrm{then}} & {} & {}\\ {\mathrm{H_g}\leftarrow (H_g + H_i)} & {} & {}\\ {\mathrm{if~}s_t > s_p + s_\delta \mathrm{then}} & {} & {}\\ {\mathrm{H_g}\leftarrow (H_g - H_i)} & {} & {\triangleright\text{Update the grow threshold}}\end{array}$  Initialize a prune threshold  $H_{p}$  Begin global parameter pruning while not  $s_p + s_\delta >s_t > s_p - s_\delta$  do for each sparse tensor  $\theta_{\mathrm{s}}^l$  of layer  $l$  do  $\begin{array}{rlr}{\big(\theta_{\mathrm{s}}^{l},pl\big)\leftarrow\mathrm{prune\_by\_threshold}(\theta_{\mathrm{s}}^{l},H_{p})}&{\triangleright p_l}&{\mathrm{is~the~number~of~pruned~weights~in~layer~l}}\\ {P\leftarrow\sum_{i}p_i,s_t\leftarrow\mathrm{calculate\_current\_sparsity}(P)}&{\triangleright P\mathrm{~is~the~total~number~of~pruned}}\\ {\mathrm{weights~in~all~layers}}&{}&{}\\ {\mathrm{if~}s_t <   s_p - s_\delta \mathrm{then}} & {} & {}\\ {\mathrm{H_p}\leftarrow (H_p - H_i)} & {} & {}\\ {\mathrm{if~}s_t > s_p + s_\delta \mathrm{then}} & {} & {}\\ {\mathrm{H_p}\leftarrow (H_p + H_i)} & {} & {\triangleright\text{Update the prune threshold}}\end{array}$
G Learned Layerwise Sparsity
Table 15 summarize the final learned layer-wise sparsity on ResNet-50 under  $80\%$  sparsity and  $40\%$  channel-wise sparsity. The model is obtained by training 100 epochs on ImageNet-1K. The parameterwise sparsity represents the sparsity budgets for all the CNN layers without the last fully-connected layer. The channel-wise sparsity denotes the sparsity of all the CNN internal layers (the first two convolution layers in the bottleneck blocks).