SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The point of studying adversarially robust structured pruning methods is to progress ML efficiency while being aware of its safety. We note that structured pruning is not the sole method that can achieve efficiency benefits, yet adversarial robustness is nowhere near a comprehensive metric for AI
safety. Approaches such as quantization, network architecture search, knowledge distillation, lossy approximation, efficiency-aware architecture tweak, and transformation to hardware-friendly patterns also provide efficiency gains [Frantar et al., 2023, Lin et al., 2023, Wang et al., 2020a, Zhang et al., 2022b, Liu et al., 2023, Xiao et al., 2023, Dettmers et al., 2023, Dao et al., 2022b,a]; yet aspects like explainability, domain generalization, fairness/bias-mitigation, model/data security, and robustness under a noisy environment are also considered important metrics for building reliable, safe, and trustworthy AI solutions [Wang et al., 2022, Wan et al., 2023, Chuang et al., 2023a,b, Chang et al., 2023c,a, Jiang et al., 2022, Chang et al., 2023b, Tang et al., Zha et al., 2023].
C Additional Details on Proposed Method
C.1 Naive Mix-Up Attempts of Kernel Smoothness
As per Section 3.2, there are generally two stages in a GKP procedure: Filter Grouping and Grouped Kernel Pruning. The following experiments shall attempt to mix-up kernel smoothness criteria into each of such stages, and we can therefore find out which stage is "friendly" to such mix-up operations and how such operations should look like.
C.1.1 During GKP Filter Grouping (Stage 1)
In TMI-GKP [Zhong et al., 2022], the filter grouping stage is driven by the tickets magnitude increase score, known as TMI-driven Clustering. We utilize it as a baseline to investigate whether our smoothness-aware filter grouping operation — Smoothness Snaking (Section 3.2.1 and Figure 3) — can maintain the baseline performance and provide improvements. Note, we denote TMI-GKP's grouped kernel pruning scheme (GK Pruning) as Greedy in the following Table 5.
Table 5: Comparison of different Filter Grouping methods in GKP. All compared models are pruned to identical MACs/Params for fairness.
Model	Filter Grouping	GK Pruning	Criterion	Benign	(a) FGSMε=0.01	(b) FGSMε=0.1
ResNet-32	Unpruned	-	92.80	71.93	31.35
TMI-driven Clustering	Greedy	Best Benign	92.99	69.15	19.59
Best (a)	92.03	70.61	15.85
Best (b)	92.77	69.90	30.51
Smoothness Snaking	Greedy	Best Benign	92.77	71.47	30.64
Best (a)	92.65	70.82	30.65
ResNet-56	Unpruned	-	93.24	75.15	39.64
TMI-driven Clustering	Greedy	Best Benign	93.95	75.18	42.18
Best (a)	93.37	75.88	42.55
Best (b)	93.66	75.74	44.09
Smoothness Snaking	Greedy	Best Benign	93.62	75.68	41.21
Best (a)	93.40	76.05	43.03
Best (b)	93.44	75.69	44.62
It can be observed that though Smoothness Snaking may yield a slightly lower benign performance than its TMI-driven baseline, it may improve the adversarial robustness when used with the same grouped kernel pruning procedure. We would also note Smoothness Snaking is significantly faster (up to  $1,600\mathrm{x}$ ) than TMI-driven clustering due to the absence of dimensionality reduction and clustering procedure (see Table 8 for details).
C.1.2 During GKP Pruning Strategies Obtaining/Pruning (Stage 2)
Following the section above, here we investigate the performance of different Grouped Kernel Pruning methods. Our proposed method is denoted as Smooth Beam Greedy (see Figure 4 for details), TMI-GKP's grouped kernel pruning method is denoted as Greedy as above, and Least Smooth represents a vanilla adaptation of smoothness-driven pruning, where the grouped kernels that are least smooth are pruned.
Table 6: Comparison of different Grouped Kernel Pruning methods in GKP. All compared models are pruned to identical MACs/Params for fairness.
Model	Filter Grouping	GK Pruning	Criterion	Benign	(a) FGSMε=0.01	(b) FGSMε=0.1
ResNet-32	Unpruned	-	92.80	71.93	31.35
Smoothness Snaking	Greedy	Best Benign	92.77	71.47	30.64
Best (a)	92.65	70.82	30.65
Smoothness Snaking	Least Smooth	Best Benign	90.09	63.91	17.22
Best (a)	88.90	70.41	15.33
Best (b)	48.37	47.80	28.55
Smoothness Snaking	Smooth Beam Greedy	Best Benign	92.97	70.57	29.31
Best (a)	92.88	71.52	30.39
Best (b)	92.86	70.79	31.32
ResNet-56	Unpruned	-	93.24	75.15	39.64
Smoothness Snaking	Greedy	Best Benign	93.62	75.68	41.21
Best (a)	93.40	76.05	43.03
Best (b)	93.44	75.69	44.62
Smoothness Snaking	Least Smooth	Best Benign	90.97	73.32	24.95
Best (a)	90.01	74.63	14.10
Best (b)	90.59	73.60	26.40
Smoothness Snaking	Smooth Beam Greedy	Best Benign	94.08	75.89	42.60
Best (a)	93.83	76.40	45.17
Best (b)
From Table 6, we may tell that Smooth Beam Greedy may significantly improve the adversarial robustness of the pruned networks, yet, it also provides remedies to the decrease in benign performance due to the smoothness snaking operation. It may also be worth noting that the Least Smooth operation is extremely detrimental to almost all tracking metrics, suggesting a vanilla mix-up is inappropriate.
C.1.3 Bonus Investigation: Is Smoothness-aware Filter Pruning Possible?
One main purpose of our paper is to endorse the potential of GKP under adversarial tasks, after [Zhong et al., 2022, Zhang et al., 2022a, Park et al., 2023] showcased the power of GKP in a benign context, thus "one less reason for filter pruning." But to make such a claim proper, we will need to investigate whether it is possible to do the same smoothness-aware mix-up with a filter pruning procedure.
Here in Table 7, we use SFP [He et al., 2018] as the baseline, which is considered one of the strongest filter pruning methods on CIFAR-10 [Krizhevsky, 2009]. Then, we try to apply the same ranked-based kernel-smoothness mix-up algorithm as in Equation 2 to find out if such a strong filter pruning baseline can withstand the same mix-up under a filter level.
Table 7: Filter pruning method SFP [He et al., 2018] applied with the same mix-up algorithm in Equation 2. Note "CSB" represents "cost smoothness balancer", which is also  $\alpha$  in Equation 2 — so a higher CSB means more biased towards the distance-based Cost metrics. All compared models are pruned to identical MACs/Params for fairness.
Model	Method	Benign	FGSMε=0.01	FGSMε=0.1	PGDε=8/255, εstep=2/255 max_iter=3
ResNet-32	SFP	91.94	69.25	30.34	30.18
CSB = 0.5	83.49	40.86	21.12	24.28
CSB = 0.75	91.03	65.94	22.92	25.40
ResNet-56	SFP	93.15	75.63	43.83	44.10
CSB = 0.5	81.83	60.46	19.59	28.45
CSB = 0.75	92.26	72.11	37.42	36.64
CSB = 0.9	93.09	74.78	40.64	41.41
It can be observed that SFP under such mix-up is completely unusable due to the degradation of benign performance; which suggests the same mix-up strategy, though effective on GKP, is not transferable to filter pruning.
C.2 Speed-Up Analysis
C.2.1 Pruning Procedure Speed Comparison: TMI-Driven Clustering v.s. Smoothness Snaking
Table 8: Wall-clock runtime comparison between SR-GKP (Ours) and TMI-GKP [Zhong et al., 2022].
Method	ResNet-32 Group	ResNet-32 Total	ResNet-56 Group	ResNet-56 Total	ResNet-110 Group	ResNet-110 Total
TMI-GKP	47m 6s	1h 20m 10s	2h 32m 12s	2h 36m 22s	4h 53m 33s	5h 30m 18s
SR-GKP	3s	11m 36s	5s	20m 43s	11s	1h 10m 16s
In Table 8, we showcased the significant runtime advantage of SR-GKP to TMI-GKP due to the absence of clustering and dimensionality reduction procedures.
C.2.2 Inference Speed Discussion: Standard Convolution v.s. Grouped Convolution
In the realm of structured pruning, there are two types of popular implementations in practice, which may result in different readings on efficiency metrics, such as inference cost reduction (FLOPs/MACs), inference speed-up, and model size compression (number of parameters):
Masking (soft pruning) To deploy a structured binary mask upon the original unpruned model, but clear the gradients of some (or all) pruned components before the weight update step during fine-tuning, therefore theoretically "structural pruning" the model. In this implementation, the pruned model will not reduce in dimension; thus, no acceleration and compression benefits can be directly observed.
This is a popular implementation — as surveyed in Table 10 “Zero-Masked?” column — because some iterative pruning methods would like the pruned/zeroed components to be reactive during fine-tuning, then pruning another set of components instead. If the pruning granularity is structured (e.g., filter-level), the fine-tuned model can be converted to a hard pruned model by the end of the update as defined below.
Component removal (hard pruning) To remove the pruned components entirely, where the pruned model will reduce in size, providing immediate compression benefits.
Our method, SR-GKP, is implemented as hard pruning. This means that if you inspect the weights of a GKP-pruned network, they are in regular shapes, and there is no zeroed weight to be found — a.k.a. "densely structured." Which may lead to a direct reduction in FLOPs/MACs and model size.
In terms of inference acceleration, previous grouped convolution implementations cannot provide speed-up benefits against a standard convolution (even with much fewer MACs/Params) because the standard convolution operator has been extensively optimized, while the grouped one hasn't (e.g., pytorch issue #10229 and #18631). But after torch 2.0, this is not the case at large. For simplicity, here we compare the forward wall-clock between a standard Conv2d of  $(\mathbb{C}_{-}\mathrm{in} = 512$ $C_{-}\mathrm{out} = 512$  , kernel_size  $= (3,3)$  ) with the exact same Conv2d but with groups  $= 2$  (meaning half of its kernels are structurally removed).
Table 9: Inference wall-clock comparison between a standard convolution operator and a GKP convolution operator (w/ groups=2 and pruning rate = 50%). Input size set as (64, 3, 224, 224).
Operator	Forward	Macs	Params
Unpruned Standard Conv	129.56 ms	550528	359296
GKP-pruned Grouped Conv	72.96 ms (56.31%)	275264 (50%)	179648 (50%)
As showcased in Table 9, an organic inference time speed-up of grouped convolution over standard convolution can be observed.
(Note, we emphasized at large above because we indeed can find shapes that are slower with grouped convolution in torch 2.0. We believe this is very much a framework optimization issue that is beyond the scope of our paper and shall be on the road map of the torch community, as well as other more inference-specific frameworks, granted their already observed improvements.)
C.3 SR-GKP Procedure
C.3.1 Simplifying the Cost Formula from TMI-GKP
For better readability, we hereby follow the notation of TMI-GKP [Zhong et al., 2022] (Equation 4), where we assume  $V^{*}$  represents a set of kept grouped kernels provided by a pruning strategy, where  $g^{\ell}$  the convolutional layer in question. The quality  $V^{*}$ , under our design, is deemed by:
$$
\underset {\text {g r u n i n g}} {\operatorname {S c o r e}} \left(V ^ {*}, \boldsymbol {g} ^ {\ell}\right) = \sum_ {s _ {u}, s _ {v} \in \binom {V ^ {*}} {2}} w \left(s _ {u}, s _ {v}\right) - \beta \left(\sum_ {s _ {i = 1} \in V ^ {*}} w \left(p _ {i}, s _ {i}\right)\right). \tag {3}
$$
Where  $w(s_{u}, s_{v})$  represents the Euclidean distance between grouped kernels  $s_{u}$  and  $s_{v}$ , but  $s_{i}$  represents the kept grouped kernel that has the least  $w(p_{i}, s_{i})$  to a pruned grouped kernel  $p_{i}$ . Thus, the former term of this equation calculates the inner distance sum of grouped kernels within strategy  $V^{*}$ , and the latter term represents the outer distance between a pruned grouped kernel and its closest kept grouped kernel. Intuitively, we would like the former term to be large, as we would prefer our ideal  $V^{*}$  to have great diversity. Following the same idea, we would like to have the latter term small, as we want the kept kernels to cover the representation power of a certain pruned kernel. By using a  $-\beta$  to connect two terms, we have  $V_{\mathrm{best}}^{*} = \arg \max_{V^{*}}(\mathrm{Score}(V^{*}, g^{\ell}))$  for all  $V^{*}$ 's obtained in Stage 2 (Figure 4).
Though similar, we differ from TMI-GKP [Zhong et al., 2022] (Equation 4) in two spots: first, we set  $\beta = \binom{V^*}{2} / p_{\mathrm{num}}$ , where  $p_{\mathrm{num}}$  represent the number of pruned grouped kernels in layer  $g^\ell$ , balancing the two terms automatically. Secondly, for the latter term of Equation 3, we only match one pruned kernel  $p_i$  with one kept kernel  $s_i$ , instead of several of them. We made these modifications for the main purpose of removing hyperparameters. We kept the grouped kernel strategy selection stage similar to TMI-GKP, but only expanded its search scope using the Smooth Beam Greedy search (Figure 4), because prior investigation suggests this stage is sensitive to smoothness-aware mix-up operations.
D Extended Experiments and Results
D.1 Preliminary
D.1.1 Details of Experiment Setups
For all experiments on VGG-16 and ResNet-32/56/110, we aim to provide all pruning methods an around 300 epochs fine-tuning/retraining budget. Experiments conducted on ResNet-50/101 are budgeted with 100 epochs. We allow compared methods to utilize their own training schedule and vanilla SGD optimizer setup.
SR-GKP utilizes an initial 1r of 0.01, with a step size of 100 or 30, depending on if it is on the 300 epoch or 100 epoch schedule. BasicBlock ResNets with VGG use a weight decay of 5e-4 but BottleNeck ResNets use a weight decay of 1e-4. SR-GKP strictly employs a batch size of 64 for all CIFAR-10 experiments, 128 for Tiny/ImageNet experiments, and 256 for ImageNet-1k experiments.
D.1.2 Details of Evaluation Criteria
We manually ensure all pruned models have a similar MACs/Params reduction from the identical baseline models. Then, following Li et al. [2017], we report all model checkpoints — yielded during the pruning procedure — that may reach the target MACs/Params reduction. For performance (accuracy) evaluation, we test each model against FGSM [Goodfellow et al., 2014] and PGD [Madry et al., 2018] under various settings and intensities.
D.1.3 Details of Compared Methods
We provide a method overview in Table 10 to provide our readers with a more comprehensive understanding of such methods.
Table 10: Details of Compared Methods. Note "C/F/GK" under "Type" represents Channel/Filter/Grouped Kernel Pruning. Whether a method requires a "Special Setup" is determined by whether it follows the most vanilla train - prune - fine-tune procedure. "Fully Pruned Epoch" reflects if given a 300 fine-tune/retrain budget, what would be the first epoch that meets the target MACs/Params reduction? (a.k.a. checkpoints after this epoch are reportable as "superscore" readings defined in Section 4.2.) "Zero-Masked?" reflects whether the pruning method can easily yield a compressed model without masking (a.k.a. hard pruning).
Method	Venue	Type	Procedure	Special Setup?	Zero-Masked?	Fully Pruned Epoch
CC [Li et al., 2021]	CVPR	C	One-shot	Y(requires data)	N	1
DHP [Li et al., 2020]	ECCV	F	Iterative (from-scratch)	Y (hypernet)	Y	100
FPGM [He et al., 2019]	CVPR	F	Iterative	N	Y	1
GAL [Lin et al., 2019]	CVPR	F	Iterative	Y (GAN)	Y	close to 300
GReg [Wang et al., 2021]	ICLR	F	One-shot	No	N	1
HARP [Zhao and Wressnegger, 2023]	ICLR	C	One-shot	Y (adv. training)	N	1
HRank [Lin et al., 2020]	CVPR	F	Iterative	N	Y	325 or 327
L1Norm [Li et al., 2017]	ICLR	F	One-shot	Y (dynamic pruning rate)	N	1
LRF [Joo et al., 2021]	AAAI	C	One-shot	Y (requires data, adding 1x1, dark knowledge)	N	1
NPPM [Gao et al., 2021]	CVPR	C	One-shot	Y (hypernet)	N	1
OToV2 [Chen et al., 2023]	ICLR	F	Iterative (from-scratch)	N	N	300
RAP-ADMM [Ye et al., 2019]	ICCV	F	Iterative	Y (adv. training)	Y	151
SFP [He et al., 2018]	IJCAI	F	Iterative	Y (soft pruning)	Y	300
TMI-GKP [Zhong et al., 2022]	ICLR	GK	One-shot	N	N	1
SR-GKP (Ours, 2023)	NeurlPS	GK	One-shot	N	N	1
Further, following the background introduction of pruning implementation in Appendix C.2.2, we hereby justify why a post-train one-shot hard pruning method — such as SR-GKP — may “require no extra (in fact, often less) cost than a standard pruning procedure.”
Under the classic train - prune - fine-tune/retrain paradigm (where our method lives), the training cost is determined at two stages: training the unpruned baseline model and fine-tuning the pruned model.
SR-GKP carries no additional training cost as it is a post-train pruning method, so it won't affect the training cost of the first stage. As for fine-tuning, the SR-GKP one-shot pruned model is already compressed and will go through a vanilla SGD update with no extra operation; we also specifically ensured that all compared methods are using an identical (or similar to the best of the method's ability) epoch budget for fine-tuning as stipulated in Section 4.2 — thus, it requires no additional (in fact, often less) cost on the fine-tuning stage as well.
For illustration convenience, here we walk through two exemplary comparisons:
Compared to TMI-GKP [Zhong et al., 2022] — the founding method of Grouped Kernel Pruning — a SR-GKP pruned model shares the exact same format with a TMI-GKP pruned model. Both pruned models will go through an identical fine-tuning procedure. In this case, SR-GKP yields no additional training/fine-tuning cost. Additionally, we'd like to direct your attention to Table 8, which indicates SR-GKP has a much faster execution time than TMI-GKP in terms of pruning procedure, thus being a more "executionally efficient" method overall.
Compared to SFP [He et al., 2018] — a popular filter pruning method that exhibits strong adversarial performance — SFP produces a zero-masked model while SR-GKP's pruned model is hard-pruned. Such zeroed weights will reactivate during its retraining/fine-tuning, where the algorithm decides which filter to zero out between epochs. In this case, the one-shot pruned SR-GKP induces significantly less cost as a) it does not require weight updates to the full model, but only on a compressed one; and b) it does not perform extra pruning operations during the fine-tuning/retraining stage.
D.2 Additional Experiments
D.2.1 ResNet-32/56/110 on CIFAR-10 with pruning rate  $\approx 43.75\%$  (in addition to Table 2)
Please refer to Table 11 with Figure 6, Table 12 with Figure 7, and Table 13 with Figure 8 for details. We also provide Table 3 in a ranked chart format for an easy digest of the three aforementioned tables.
Figure 6: "Superscore" visualization of ResNet-32 on CIFAR-10 with pruning rate  $\approx 43.75\%$
Table 11: Full experiments of ResNet-32 on CIFAR-10 with pruning rate  $\approx 43.75\%$ . All pruning methods are performed on the same baseline model. "Best (a)" represents the performance of a model checkpoint that meets the showcased MACs/Params reduction and performs best against criterion (a).
Method	Criterion	Benign	(a) FGSMε=0.01	(b) FGSMε=0.1	(c) PGDmax,iter=3	MACs (M)	Params (M)
Unpruned	-	92.80	71.93	31.35	35.40	69.479	0.464
CC [Li et al., 2021]	Best Benign	93.01	70.06	25.26	30.40
Best (a)	92.69	70.92	25.84	30.75	39.261	0.312
Best (b)	92.82	70.50	26.79	31.20
DHP [Li et al., 2020]	Best Benign	92.26	67.25	21.35	26.12
Best (a)	91.88	68.14	26.64	26.80	40.091	0.283
Best (b)	91.80	67.62	27.23	26.65
FPGM [He et al., 2019]	Best Benign	92.41	69.75	29.61	32.32
Best (a)	92.06	70.84	29.74	32.57	39.352	0.262
Best (b)	92.23	69.96	30.99	32.43
GAL [Lin et al., 2019]	-	90.31	72.21	31.68	38.32	38.80	0.233
L1Norm-A [Li et al., 2017]	Best Benign	91.45	68.04	24.03	28.71
Best (a)	91.03	70.00	25.92	30.70	39.861	0.252
Best (b)	90.38	67.61	32.50	29.67
L1Norm-B [Li et al., 2017]	Best Benign	91.58	67.04	25.24	28.91
Best (a)	91.03	71.59	29.11	35.20	39.630	0.313
Best (b)	90.59	69.67	34.62	34.02
LRF [Joo et al., 2021]	Best Benign	93.04	69.38	25.82	30.07
Best (a)	92.63	70.22	25.13	29.74	38.791	0.260
Best (b)	92.79	69.21	27.54	28.64
NPPM [Gao et al., 2021]	Best Benign	93.13	69.88	26.03	29.75
Best (a)	92.88	70.45	27.27	30.41	39.605	0.326
Best (b)	92.89	70.08	27.97	30.51
RAP-ADMM [Ye et al., 2019]	-	74.88	72.27	25.80	57.66	39.370	0.271
OTOv2 (from scratch) [Chen et al., 2023]	-	90.97	66.36	17.28	27.15	-	-
OTOv2 (post train) [Chen et al., 2023]	-	92.14	70.63	28.01	31.96	-	-
SFP [He et al., 2018]	-	91.94	69.25	30.34	30.18	40.375	0.266
TMI-GKP [Zhong et al., 2022]	Best Benign	92.99	69.15	19.59	28.09
Best (a)	92.03	70.61	15.85	29.44	39.545	0.263
Best (b)	92.77	69.90	30.51	32.05
SR-GKP (Ours)	Best Benign	92.97	70.57	29.31	32.01
Best (a)	92.88	71.52	30.39	33.36	39.545	0.263
Best (b)	92.86	70.79	31.32	32.89
Figure 7: "Superscore" visualization of ResNet-56 on CIFAR-10 with pruning rate  $\approx 43.75\%$
Table 12: Full experiments of ResNet-56 on CIFAR-10 with pruning rate  $\approx 43.75\%$ . All pruning methods are performed on the same baseline model. "Best (a)" represents the performance of a model checkpoint that meets the showcased MACs/Params reduction and performs best against criterion (a).
Method	Criterion	Benign	(a) FGSMε=0.01	(b) FGSMε=0.1	(c) PGDmaxiter=3	MACs (M)	Params (M)
Unpruned	-	93.24	75.15	39.64	42.58	126.561	0.853
CC [Li et al., 2021]	Best Benign	94.04	74.78	29.25	37.85	69.837	0.616
Best (a)	93.73	75.75	29.26	38.86
Best (b)	93.70	75.56	29.89	38.90
DHP [Li et al., 2020]	Best Benign	92.42	69.50	29.93	32.56	73.289	0.480
Best (a)	92.34	70.66	30.17	33.64
Best (b)	92.07	31.54	70.42	32.83
FPGM [He et al., 2019]	Best Benign	93.60	75.31	43.20	43.55	71.661	0.482
Best (a)	93.37	76.28	44.96	44.66
Best (b)
GAL [Lin et al., 2019]	-	91.27	76.38	47.32	47.36	98.24	0.700
HRank [Lin et al., 2020]	Best Benign	92.27	72.32	19.11	32.51	79.237	0.584
Best (b)	92.07	72.59	18.94	32.21
Best (a)
L1Norm-A [Li et al., 2017]	Best Benign	92.44	73.00	41.00	40.76	67.995	0.487
Best (a)	91.65	74.30	43.59	42.74
Best (b)	91.34	71.94	48.19	42.10
L1Norm-B [Li et al., 2017]	Best Benign	92.62	72.97	41.30	41.79	72.115	0.586
Best (a)	91.94	75.16	42.49	43.71
Best (b)	91.70	74.40	45.16	43.41
LRF [Joo et al., 2021]	Best Benign	93.93	73.47	25.59	34.86	71.009	0.490
Best (a)	93.68	74.80	27.39	36.78
Best (b)	93.63	74.06	28.20	35.98
NPPM [Gao et al., 2021]	Best Benign	93.55	74.82	29.07	37.12	70.843	0.601
Best (a)	93.35	75.50	30.29	38.09
Best (b)	93.43	75.27	31.18	38.13
RAP-ADMM [Ye et al., 2019]	-	78.37	75.19	27.84	61.15	71.661	0.482
OTOv2 (from scratch) [Chen et al., 2023]	-	91.57	65.91	23.63	25.52	79.780	0.480
OTOv2 (post train) [Chen et al., 2023]	-	93.02	74.29	40.46	41.58	66.196	0.554
SFP [He et al., 2018]	-	93.15	75.63	43.83	44.10	71.462	0.481
TMI-GKP [Zhong et al., 2022]	Best Benign	93.95	75.18	42.18	43.46	71.855	0.482
Best (a)	93.37	75.88	42.55	43.74
Best (b)	93.66	75.74	44.09	44.51
SR-GKP (Ours)	Best Benign	94.08	75.89	42.60	43.85	71.855	0.482
Best (a)	93.83	76.40	45.17	44.85
Best (b)
Figure 8: "Superscore" visualization of ResNet-110 on CIFAR-10 with pruning rate  $\approx 43.75\%$