SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 13: Full experiments of ResNet-110 on CIFAR-10 with pruning rate  $\approx 43.75\%$ . All pruning methods are performed on the same baseline model. "Best (a)" represents the performance of a model checkpoint that meets the showcased MACs/Params reduction and performs best against criterion (a).
Method	Criterion	Benign	(a) FGSMz=0.01	(b) FGSMz=0.1	(c) PGDz=8/255, εsup=2/255	MACs (M)	Params (M)
Unpruned	-	94.26	78.56	48.83	50.85	254.995	1.728
CC [Li et al., 2021]	Best Benign	94.31	75.12	27.31	38.24	144.414	1.046
Best (a)	94.17	75.88	28.14	39.78
Best (b)	94.23	75.52	29.00	39.64
DHP [Li et al., 2020]	Best Benign	92.53	67.94	25.08	26.83	101.350	0.612
Best (a)	92.21	68.86	25.51	27.19
Best (b)	92.25	67.82	26.64	26.67
FPGM [He et al., 2019]	Best Benign	94.18	79.32	52.16	52.46	114.357	0.976
Best (a)	94.02	79.78	53.33	52.73
Best (b)	94.10	79.70	54.15	52.82
GAL [Lin et al., 2019]	-	93.42	82.34	52.79	57.55	180.677	1.186
HRank [Lin et al., 2020]	Best Benign	92.96	73.70	16.87	35.72	158.992	1.060
L1Norm-A [Li et al., 2017]	Best Benign	92.75	74.84	38.03	41.54	143.454	0.958
Best (a)	91.93	75.83	39.71	44.04
Best (b)	92.21	75.26	45.48	45.49
L1Norm-B [Li et al., 2017]	Best Benign	92.96	75.26	41.19	44.86	144.909	1.094
Best (a)	92.40	76.10	41.36	47.05
Best (b)	91.71	73.58	41.92	42.69
LRF [Joo et al., 2021]	Best Benign	94.49	76.60	29.50	42.15	144.405	0.997
Best (a)	94.22	76.93	29.27	42.59
Best (b)	94.38	76.71	30.67	42.67
NPPM [Gao et al., 2021]	Best Benign	94.16	76.16	32.84	41.54	146.722	1.120
Best (a)	94.01	76.56	33.71	42.14
Best (b)
RAP-ADMM [Ye et al., 2019]	-	81.43	78.36	29.26	63.11	144.357	0.976
OTOv2 (from scratch) [Chen et al., 2023]	-	91.58	71.43	23.94	34.74	-	-
OTOv2 (post train) [Chen et al., 2023]	-	93.99	77.25	47.96	48.46	-	-
SFP [He et al., 2018]	-	94.44	78.75	53.62	52.57	144.274	0.976
TMI-GKP [Zhong et al., 2022]	Best Benign	94.90	77.50	47.15	49.00	144.551	0.976
Best (a)	94.63	78.02	45.95	48.81
Best (b)	94.90	77.50	47.15	49.00
SR-GKP (Ours)	Best Benign	95.00	78.01	46.53	48.86	144.551	0.976
Best (a)	94.69	79.14	47.49	50.01
Best (b)	94.60	78.92	49.09	49.83
Please refer to Table 15, 16, and 17. We also provide Table 14 in a ranked chart format for an easy digest of the three aforementioned tables.
Table 14: Methods ranked against each other on each model with pruning rate  $\approx 62.5\%$ , lower is better. "ResNet-XX Mean Rank" means a method's average ranks across four metrics on ResNet-XX; e.g., SR-GKP is ranked #1/#1/#3/#3 for its best performance across benign/FGSM 0.01/FGSM 0.1/PGD metrics on ResNet-110 against other methods, so it'd have a ResNet-110 Mean Rank of  $(1 + 1 + 3 + 3) / 4 = \# 2$ . Methods with incomplete presence across the three models are excluded. This table is provided to facilitate the digestion of Table 15, 16, and 17, which consist of raw results.
Method	ResNet-32 Mean Rank	ResNet-56 Mean Rank	ResNet-110 Mean Rank	All Models Mean Rank
CC [Li et al., 2021]	#1	#3.5	#5.75	#3.42
DHP [Li et al., 2020]	#3.75	#7.5	#7.75	#6.33
FPGM [He et al., 2019]	#4.25	#3.75	#2	#3.33
L1Norm-A [Li et al., 2017]	#5.5	#3.75	#5.75	#5.00
L1Norm-B [Li et al., 2017]	#6.25	#6.25	#4.25	#5.58
NPPM [Gao et al., 2021]	#5	#4	#5.5	#4.83
SFP [He et al., 2018]	#7.5	#5	#3	#5.17
SR-GKP (Ours)	#2.75 (2nd-best)	#2.25	#2	#2.33
Table 15: Full experiments of ResNet-32 on CIFAR-10 with pruning rate  $\approx 62.5\%$ . All pruning are performed on the same baseline model. "Best (a)" represents the performance of a model checkpoint that meets the showcased MACs/Params reduction and performs best against criterion (a).
Method	Criterion	Benign	(a) FGSMε=0.01	(b) FGSMε=0.1	(c) PGDmax_iter=3	MACs (M)	Params (M)
Unpruned	-	92.80	71.93	31.35	35.40	69.479	0.464
CC [Li et al., 2021]	Best Benign	92.39	70.71	15.45	30.87	26.904	0.210
Best (a)	91.83	69.99	28.84	30.91
Best (b)	92.01	67.97	29.07	29.29
DHP [Li et al., 2020]	Best Benign	91.73	66.87	28.00	26.71	-	-
Best (a)	91.36	67.71	26.83	26.89
Best (b)	91.31	66.84	28.75	26.39
FPGM [He et al., 2019]	Best Benign	91.32	65.41	20.91	24.97	-	-
Best (a)	90.47	67.77	15.95	26.61
Best (b)	91.04	56.51	24.47	25.68
L1Norm-A [Li et al., 2017]	Best Benign	89.96	66.06	20.44	27.29	26.511	0.163
Best (a)	89.52	67.65	18.07	27.93
Best (b)	89.23	66.75	23.21	27.88
L1Norm-B [Li et al., 2017]	Best Benign	90.01	64.89	19.39	24.52	26.157	0.146
Best (a)	89.78	67.14	17.75	26.91
Best (b)	89.42	66.66	21.63	27.55
LRF [Joo et al., 2021]	Best Benign	92.79	68.97	22.02	27.95	29.915	0.196
Best (a)	92.46	70.56	20.91	28.90
Best (b)	92.43	69.50	25.40	29.22
NPPM [Gao et al., 2021]	Best Benign	91.92	66.83	22.23	25.63	26.998	0.198
Best (a)	91.79	67.56	22.39	25.91
Best (b)	91.67	67.16	23.97	25.60
SFP [He et al., 2018]	-	90.28	66.71	20.35	25.47	-	-
SR-GKP (Ours)	Best Benign	92.21	66.38	21.91	25.98	26.717	0.176
Best (a)	91.52	69.33	14.83	27.94
Best (b)	92.04	66.37	23.55	25.80
D.2.2 ResNet-32/56/110 on CIFAR-10 with pruning rate  $\approx 62.5\%$
Table 16: Full experiments of ResNet-56 on CIFAR-10 with pruning rate  $\approx 62.5\%$ . All pruning methods are performed on the same baseline model. "Best (a)" represents the performance of a model checkpoint that meets the showcased MACs/Params reduction and performs best against criterion (a).
Method	Criterion	Benign	(a) FGSMε=0.01	(b) FGSMε=0.1	(c) PGDmax_iter=3	MACs (M)	Params (M)
Unpruned	-	93.24	75.15	39.64	42.58	126.561	0.853
CC [Li et al., 2021]	Best Benign	93.57	73.63	25.30	35.40	48.692	0.421
Best (a)	93.33	74.29	24.96	35.54
Best (b)	93.37	62.56	26.12	35.25
DHP [Li et al., 2020]	Best Benign	91.66	70.66	29.75	31.40	-	-
Best (a)	91.36	71.22	26.36	31.05
Best (b)	91.48	70.41	30.15	31.27
FPGM [He et al., 2019]	Best Benign	92.64	71.80	35.17	35.17	-	-
Best (a)	92.31	72.58	35.98	35.94
Best (b)	92.62	71.86	35.99	35.61
HRank [Lin et al., 2020]	-	90.63	69.49	17.14	29.51	-	-
L1Norm-A [Li et al., 2017]	Best Benign	91.79	68.95	24.68	34.90	47.562	0.355
Best (a)	91.12	71.76	37.01	36.97
Best (b)	91.47	70.10	39.83	37.03
L1Norm-B [Li et al., 2017]	Best Benign	91.56	69.56	32.80	33.61	47.794	0.322
Best (a)	90.66	71.24	33.22	35.09
Best (b)	91.07	69.25	26.19	33.66
NPPM [Gao et al., 2021]	Best Benign	93.07	73.23	29.66	35.24	52.550	0.446
Best (a)	92.84	74.21	28.27	35.38
Best (b)	93.02	72.91	30.39	34.03
SFP [He et al., 2018]	-	92.24	72.21	33.65	35.39	-	-
SR-GKP (Ours)	Best Benign	92.93	70.94	21.15	32.01	48.409	0.323
Best (a)	92.69	73.54	35.42	38.39
Best (b)	92.69	73.38	36.53	38.95
Table 17: Full experiments of ResNet-110 on CIFAR-10 with pruning rate  $\approx 62.5\%$ . All pruning methods are performed on the same baseline model. "Best (a)" represents the performance of a model checkpoint that meets the showcased MACs/Params reduction and performs best against criterion (a).
Method	Criterion	Benign	(a) FGSMε=0.01	(b) FGSMε=0.1	(c) PGDε=8/255, εstep=2/255 max,iter=3	MACs (M)	Params (M)
Unpruned	-	94.26	78.55	48.84	50.85	254.995	1.728
CC [Li et al., 2021]	Best Benign	94.29	73.77	24.50	36.32	98.582	0.727
Best (a)	94.05	74.23	24.87	36.71
Best (b)	94.03	73.70	26.03	36.44
DHP [Li et al., 2020]	Best Benign	92.73	71.39	23.19	35.51	-	-
Best (a)	92.35	72.41	23.70	36.22
Best (b)	92.50	71.27	25.13	34.86
FPGM [He et al., 2019]	Best Benign	94.11	76.11	47.62	47.54	-	-
Best (a)	94.00	76.52	48.39	47.75
Best (b)	93.93	76.41	49.33	47.62
HRank [Lin et al., 2020]	-	91.94	70.13	15.04	30.19	-	-
L1Norm-A [Li et al., 2017]	Best Benign	92.50	73.06	40.19	41.77	97.952	0.622
Best (a) & (b)	91.51	74.99	42.62	43.45
L1Norm-B [Li et al., 2017]	Best Benign	94.04	74.81	41.82	41.28	101.256	0.484
Best (a)	93.79	75.55	42.36	41.99
Best (b)	93.86	74.96	43.43	41.99
LRF [Joo et al., 2021]	Best Benign	94.10	75.47	20.66	39.87	94.479	0.638
Best (a)	93.88	76.96	33.44	42.52
Best (b)	93.98	76.21	34.55	41.77
NPPM [Gao et al., 2021]	Best Benign	93.93	74.71	31.37	38.81	99.915	0.746
Best (a)	93.76	75.32	31.12	39.26
Best (b)	93.79	75.02	32.62	39.26
SFP [He et al., 2018]	-	92.98	76.08	52.15	47.26	-	-
SR-GKP (Ours)	Best Benign	94.31	76.31	43.88	45.44	97.217	0.654
Best (a)	94.17	76.52	43.98	46.01
Best (b)	94.27	76.17	44.56	45.74
D.2.3 VGG-16 on CIFAR-10
Please refer to Table 18 with Figure 9 for details.
Figure 9: "Superscore" visualization of VGG-16 on CIFAR-10.
Table 18: Full experiments of VGG-16 on CIFAR-10. All pruning methods are performed on the same baseline model. "Best (a)" represents the performance of a model checkpoint that meets the showcased MACs/Params reduction and performs best against criterion (a).
Method	Criterion	Benign	(a) FGSMε=0.01	(b) FGSMε=0.1	(c) PGDε=8/255, εstep=2/255	MACs (M)	Params (M)
Unpruned	-	93.94	83.73	50.51	64.99	313.433	14.728
CC [Li et al., 2021]	Best Benign	94.14	83.09	47.83	62.81
Best (a)	93.88	83.38	47.70	62.74	178.107	-
Best (b)	93.97	83.14	48.97	62.85
GAL [Lin et al., 2019]	-	91.29	81.69	61.96	67.29	203.224	7.732
HRank [Lin et al., 2020]	Best Benign	93.57	81.45	31.80	56.28
Best (a)	93.53	81.58	34.00	56.50	212.264	8.700
Best (b)	93.54	81.52	34.76	56.34
L1Norm [Li et al., 2017]	Best Benign	92.88	80.85	37.21	56.43
Best (a)	92.41	81.53	31.10	56.17	179.561	9.135
Best (b)	92.06	80.48	41.81	57.03
TMI-GKP [Zhong et al., 2022]	Best Benign	94.07	83.48	55.30	66.06	178.184	8.293
SR-GKP (Ours)	Best Benign	93.95	83.49	56.64	65.57
Best (a)	93.77	83.85	57.48	66.51	178.184	8.293
Best (b)	93.86	83.61	59.03	66.62
D.2.4 ResNet-56/101 on Tiny-ImageNet
Please refer to Table 19 for details.
Table 19: Full experiments of ResNet-56/101 on Tiny-Imagenet. All pruning methods are performed on the same baseline model. "Best (a)" represents the performance of a model checkpoint that meets the showcased MACs/Params reduction and performs best against criterion (a).
Model	Method	Criterion	Benign	(a) FGSMε=0.001	(b) FGSMε=0.01	(c) FGSMε=0.1	(d) PGMDmax_iiter=3	MACs (M)	Params (M)
ResNet-56	Unpruned	-	55.59	53.55	28.29	8.00	15.80	506.254	0.865
SR-GKP (Ours)	Best Benign	54.83	54.14	29.22	7.71	17.08	318.690	0.547
TMI-GKP [Zhong et al., 2022]	Best Benign	51.48	50.07	27.23	7.62	15.42	318.690	0.547
ResNet-101	Unpruned	-	65.51	65.12	48.10	10.13	37.66	10081.092	42.902
SR-GKP (Ours)	Best Benign	67.21	66.52	46.98	8.95	37.34	5721.113	24.226
Best (a)	65.69	64.91	37.95	10.95	25.94
Best (b)	65.61	64.71	38.27	10.85	25.97
Best (c)	65.69	64.91	37.95	10.96	25.95
ResNet-101	TMI-GKP [Zhong et al., 2022]	Best Benign	64.69	64.03	42.57	8.40	32.52	5721.113	24.226
Best (a)	63.53	62.23	33.47	10.46	20.61	5721.113	24.226
Best (b)	63.60	61.94	33.71	10.71	20.47	5721.113	24.226
Best (c)	63.50	61.93	33.67	10.77	20.63	5721.113	24.226
D.2.5 Comparison with Channel Pruned HARP
HARP by Zhao and Wressnegger [2023] is one of a few adversarily robust structured pruning methods available for comparison, as it has a channel pruning version offered in its appendix (HARP-CP). However, HARP utilizes several different training schedules than other methods, and we therefore cannot compare it with other methods using a unified baseline (as in Table 18). To fulfill the comparison, we opt to align SR-GKP with HARP under different potential training schedule settings offered in HARP literature or our experiment setup. Please refer to Table 20 for details.
Table 20: Comparison with Channel-Pruned HARP (HARP-CP) by Zhao and Wressnegger [2023] on VGG-16 with CIFAR-10 dataset. (AT = Adversarial Training; NT = Natural Training; AS = epoch with best adversarial accuracy is saved; NS = epoch with best benign accuracy is saved; LS = last epoch is saved; SS = "Superscore" readings according to Section 4.2 and Table 10.
Method	Baseline Budget	Pruning Budget	Fine-tuning Budget	Best Benign	FGSMε=0.01	FGSMε=0.1	PGD
HARP-CP	NT, 300 (AS)	AT, 20 (LS)	NT, 300 (NS)	91.66	55.11	23.13	3.30
HARP-CP	NT, 300 (NS)	AT, 20 (LS)	NT, 300 (NS)	87.04	23.49	10.71	0.05
HARP-CP	NT, 300 (AS)	AT, 20 (LS)	AT, 100 (AS)	72.06	64.08	17.96	50.93
SR-GKP (Ours)	NT, 300, (LS)	0	NT, 300 (SS)	93.95	83.85	59.03	66.62
SR-GKP (Ours)	NT, 300, (LS)	0	NT, 100 (SS)	93.48	82.62	52.35	62.18
We believe it is safe to conclude that SR-GKP showcased clear dominance over the HARP-CP.
Although one network-dataset combination is not a comprehensive evaluation, Figure 13 of HARP suggests HARP-CP is extremely close to RAP-ADMM [Ye et al., 2019] — a method we already compared — in terms of both benign and adversarial acc; and from Figure 6, 7, 8 as well as Table 3, we may conclude that RAP-ADMM performs vastly below SR-GKP on various metrics. We find it interesting that HARP and RAP-ADMM — two methods utilizing adversarial training — still perform worse than SR-GKP on most adversarial tasks, though SR-GKP only sees benign inputs; indicating the effectiveness of SR-GKP. We suspect this is mainly due to the fact HARP and RAP-ADMM are not structured pruning-focused methods.
D.3 Ablation Studies
Please refer to Table 21, Table 22, and Table 23 for ablation studies on hyperparameter  $\alpha$  in Equation 2. We denote it as CSB as it is in essence a "cost-smoothness balancer." It can be observed that a relatively high CSB — meaning giving more bias to the distance-based cost metrics — may yield better performance.
Table 21: Ablation study of cost-smoothness balancer "CSB" (α in Equation 2) on ResNet-32 with CIFAR-10.
Method	Criterion	Benign	(a) FGSMε=0.01	(b) FGSMε=0.1	MACs (M)	Params (M)
Unpruned	-	92.80	71.93	31.35	69.479	0.464
0.1 CSB	Best Benign	92.79	70.39	28.09
Best (a)	92.59	71.50	29.03	39.545	0.263
Best (b)	92.47	70.94	29.69
0.25 CSB	Best Benign	93.01	70.45	27.91
Best (a)	92.70	71.32	30.46	39.545	0.263
Best (b)
0.5 CSB	Best Benign	93.07	68.74	28.07
Best (a)	92.54	70.63	21.42	39.545	0.263
Best (b)	92.78	70.29	30.47
0.75 CSB	Best Benign	92.93	70.55	30.02
Best (a)	92.70	71.09	30.89	39.545	0.263
Best (b)
0.9 CSB	Best Benign	92.97	70.57	29.31
Best (a)	92.88	71.52	30.39	39.545	0.263
Best (b)	92.86	70.79	31.32
Table 22: Ablation study of cost-smoothness balancer "CSB" (α in Equation 2) on ResNet-56 with CIFAR-10.
Method	Criterion	Benign	(a) FGSMε=0.01	(b) FGSMε=0.1	MACs (M)	Params (M)
Unpruned	-	93.24	75.15	39.64	126.561	0.853
0.1 CSB	Best Benign	93.72	75.17	39.90
Best (a)	93.38	75.79	42.53	71.855	0.482
Best (b)
0.25 CSB	Best Benign	93.83	75.68	40.40
Best (a)	93.70	76.49	42.35	71.855	0.482
Best (b)	93.57	75.72	43.92
0.5 CSB	Best Benign	93.76	75.61	41.20
Best (a)	93.41	76.69	42.58	71.855	0.482
Best (b)	93.45	75.97	42.69
0.75/0.9 CSB	Best Benign	94.08	75.89	42.60
Best (a)	93.83	76.40	45.17	71.855	0.482
Best (b)
Table 23: Ablation study of cost-smoothness balancer "CSB" ( $\alpha$  in Equation 2) on ResNet-110 with CIFAR-10.
Method	Criterion	Benign	(a) FGSMε=0.01	(b) FGSMε=0.1	MACs (M)	Params (M)
Unpruned	-	94.26	78.56	48.83	254.995	1.728
0.1 CSB	Best Benign	94.60	78.44	44.20
Best (a)	94.47	79.62	46.03	144.551	0.976
Best (b)	94.35	79.41	47.5
0.25 CSB	Best Benign	94.50	77.86	47.07
Best (a)	94.35	78.65	46.04	144.551	0.976
Best (b)	94.40	78.52	48.10
0.5 CSB	Best Benign	95.00	78.01	46.53
Best (a)	94.69	79.14	47.49	144.551	0.976
Best (b)	94.60	78.92	49.09
0.75 CSB	Best Benign	94.49	77.97	45.93
Best (a)	94.26	78.91	47.84	144.551	0.976
Best (b)	94.32	78.44	48.89
0.9 CSB	Best Benign	94.54	78.21	47.16
Best (a)	94.29	78.81	46.71	144.551	0.976
Best (b)	94.31	78.25	48.55