SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Grouped-based pruning will always seek out some kind of balance among groups to avoid a skewed distribution where all important components are distributed to certain groups [Zhong et al., 2022]. Following this principle, we want our filter groups to have balanced smoothness — so that we do not end up having any filter group that is "over" or "under-smoothed" to start with for pruning. However, this is equivalent to the partition problem [Korf, 1998], which is known to be NP-hard. Given the filter grouping stage is often robust to adjustments<sup>2</sup>, we proposed to sort filters according to their smoothness, then assign them iteratively in an S-shaped "snaking" manner across a predefined number of filter groups, as shown in Figure 3, namely Smoothness Snaking. Empirical results suggest although smoothness snaking may not be as optimal as the dynamic clustering scheme in Zhong et al. [2022] in terms of benign performance, it is able to provide better adversarial robustness under adversarial attacks and is much faster to execute (Table 8) due to the absence of dimensionality reduction & clustering procedures (Appendix C). We consider this to be a successful mix-up.
3.2.2 Smooth Beam Greedy GKP Search
Knowing that the pruning strategies obtaining/decision stage (Stage 2 per Section 3.2) of GKP is sensitive to tampering, a direct application of smoothness criteria would not work (Appendix C). This indicates the distance-based cost formula (Equation 5) proposed in Zhong et al. [2022] carries a significant influence on the performance of a pruned network, which is unsurprising given the vast popularity of distance-based pruning arts. Since we can't directly replace this cost formula, we propose to widen the capacity of pruning strategies obtaining stage with smoothness in mind while keeping the decision stage akin to the cost formula. By doing this, the chances of more "smoothness-aware" pruning strategies being employed are increased.
Figure 4: Smoothness-aware Beam Greedy Search for Grouped Kernel Pruning (Stage 2).
Again, to keep our approach simple, we implemented a custom beam search element to consider more grouped kernels during each advancement (Figure 4). Per each iteration, all gathered candidate grouped kernel pruning strategies will be evaluated against a mixture of smoothness and cost criteria, where only a  $\text{Beam}_{\text{width}}$  amount of grouped kernels will be kept for further advancement (until the desired pruning ratio is reached). The scoring formula to determine which subset of strategies may be kept in the beam is defined as (larger scores are better/preferred):
$$
\underset {\text {m i x - u p}} {\operatorname {S c o r e}} \left(\boldsymbol {G} \boldsymbol {K} _ {\text {b r a n c h}}, \alpha\right) = \alpha \cdot \phi \left(\operatorname {C o s t} \left(\boldsymbol {G} \boldsymbol {K} _ {\text {b r a n c h}}\right)\right) + (1 - \alpha) \cdot \phi \left(\operatorname {S m o o t h n e s s} \left(\boldsymbol {G} \boldsymbol {K} _ {\text {b r a n c h}}\right)\right), \tag {2}
$$
where  $\phi\big(\text{Criterion}(\pmb{G}\pmb{K}_{\text{branch}})\big)$  for  $\text{Criterion} \in \{\text{Cost}, \text{Smoothness}\}$  represents the rank of such  $\pmb{G}\pmb{K}_{\text{branch}}$  when all collected  $\pmb{G}\pmb{K}_{\text{branch}}$  candidates are sorted according to the given criterion in a descending order. So  $\phi\big(\text{Smoothness}(\pmb{G}\pmb{K}_{\text{branch}})\big) = 0$  would suggest this particular  $\pmb{G}\pmb{K}_{\text{branch}}$  has a greater smoothness ranking (a.k.a. "less smooth") then all other  $\pmb{G}\pmb{K}_{\text{branch}}$  candidates in considerations.  $\alpha$  is a tunable balancing parameter that adjusts the importance of one criterion over
$^{2}$ In Zhong et al. [2022], various filter grouping strategies — including random assignment — were proposed. Many of them tend to perform reasonably well, albeit less performant nor stable than the proposed TMI-driven solution.
the other. The Smoothness  $(GK_{\mathrm{branch}})$  equation is simply the sum of all kernels within all  $GK_{\mathrm{kept}}$  over Equation 1; the  $\mathrm{Cost}(GK_{\mathrm{branch}})$  equation is a modified version of Equation 5 in Zhong et al. [2022], where we removed some hyperparameters for simplicity and to reduce tuning workload. Please refer to Figure 4 and Appendix C.3 for more details.
4.1 Experiment Setups
We evaluate the efficacy of our method on ResNet-32/56/110 with the BasicBlock implementation, ResNet-50/101 with the BottleNeck implementation, and VGG-16 [He et al., 2016, Simonyan and Zisserman, 2015]. For datasets, we choose CIFAR-10 [Krizhevsky, 2009], Tiny-ImageNet [Wu et al., 2017], and ImageNet-1k [Deng et al., 2009] for a wide range of coverage. For all compared methods and under most model-dataset combinations, we tried our best to replicate them with a  $\approx 300$  epochs (except for ImageNet, where we only employ 100 epochs) of fine-tuning/retraining budget while maintaining all other settings either identical or proportional to their original publications.
4.2 Compared Methods and Evaluating Criteria
We evaluate our proposed method against up to 13 popular densely structured pruning methods and variants shown in Appendix D. Whenever possible, we produce pruned-and-fine-tuned (or retrained) models upon identical unpruned baseline models with similar post-prune MACs and Params. Then, we compare their inference accuracy on benign inputs as well as adversarially-perturbed inputs powered by FGSM and PGD attacks in various perturbation budgets and intensities [Goodfellow et al., 2014, Madry et al., 2018].
One reporting mechanism that is probably unique to our paper — in comparison to standard pruning art under the benign space — is for some of our methods, results of multiple epochs are reported, each showcasing a method's peak performance against different evaluation metrics (a.k.a. "superscore"). This is because for benign tasks, following Li et al. [2017] and He et al. [2019], only the epoch checkpoint with the best benign accuracy needs to be reported. But under an adversarial context, if a pruning method is capable of producing more than one fully pruned model during the fine-tuning/retraining stage, oftentimes, the best performer per each evaluation metric does not overlap. We believe it is responsible to report them all as there are no dominant evaluation metrics in our experiments; however, this is an important advantage for methods that can generate multiple usable pruned models (e.g., one-shot pruning) over methods that can only generate a few or just one fully pruned model (e.g., layer-wise iterative pruning). We survey the availability of such checkpoints and their epoch cutoff in Table 10: "Fully Pruned Epoch" column.
We also comprehensively investigate each pruning method against a checklist of questions as presented in Table 10 of Appendix D.1.3, including pruning granularity, procedure, when is the first fully pruned epoch, zero-masked or hard-pruning, and many other important questions. This investigation, along with the epoch budget constraint and baseline control, should provide our audience with a more leveled playing field for fair and informed methods comparison.
Further, to facilitate digesting mass results reported across many methods/variants and against various metrics, we present our large-scale empirical evaluation in three different ways. Take BasicBlock ResNets on CIFAR-10 as an example (with pruning rate  $\approx 43.75\%$ ), we present the raw results as Table 11, 12, and 13; where we visualize them accordingly as Figure 6, 7, and 8. Last, we make a rank chart, Table 3, to provide a straightforward gauge of the competitiveness among different pruning methods.
Our abbreviated results — Table 2 and Table 4 — showcased the performance of various modern SOTA methods as well as our proposed methods on the two most popular model-dataset combinations [Blalock et al., 2020]. For ResNet-56 on CIFAR-10, our method outperformed every other method on all evaluating criteria, except for PGD; as RAP-ADMM [Ye et al., 2019] outperformed SR-GKP significantly with  $61.16\%$  v.  $44.85\%$ . However, it is worth noting that RAP-ADMM does adversarial training on PGD-perturbed data, so it is not surprising that it performs well against the seen type of attack. Unfortunately, it seems like the adversarially trained RAP-ADMM cannot generalize its
Figure 5: Visualization of ResNet-56 on CIFAR-10 with pruning rate  $\approx 43.75\%$  — note this plot is done in a "superscore" manner for a concise presentation; the four bars of a method may not belong to the same model checkpoint.
Table 2: ResNet-56 on CIFAR-10. All pruning methods are performed on the same baseline model. "Best (a)" represents the performance of a model checkpoint that meats the showcased MACs/Params reduction that performs the best against criterion (a).
Method	Criterion	Benign	(a) FGSMε=0.01	(b) FGSMε=0.1	(c) PGDε=8/255, εstep=2/255	MACs (M)	Params (M)
Unpruned	-	93.24	75.15	39.64	42.58	126.561	0.853
CC [Li et al., 2021]	Best Benign	94.04	74.78	29.25	37.85	69.837	0.616
Best (a)	93.73	75.75	29.26	38.86
Best (b)	93.70	75.56	29.89	38.90
FPGM [He et al., 2019]	Best Benign	93.60	75.31	43.20	43.55	71.661	0.482
Best (a)	93.37	76.28	44.96	44.66
Best (b)
HRank [Lin et al., 2020]	Best Benign	92.27	72.32	19.11	32.51	79.237	0.584
Best (b)	92.07	72.59	18.94	32.21
L1Norm-B [Li et al., 2017]	Best Benign	92.62	72.97	41.30	41.79	72.115	0.586
Best (a)	91.94	75.16	42.49	43.71
Best (b)	91.70	74.40	45.16	43.41
LRF [Joo et al., 2021]	Best Benign	93.93	73.47	25.59	34.86	71.009	0.490
Best (a)	93.68	74.80	27.39	36.78
Best (b)	93.63	74.06	28.20	35.98
NPPM [Gao et al., 2021]	Best Benign	93.55	74.82	29.07	37.12	70.843	0.601
Best (a)	93.35	75.50	30.29	38.09
Best (b)	93.43	75.27	31.18	38.13
RAP-ADMM[Ye et al., 2019]	-	78.37	75.19	27.84	61.15	71.661	0.482
SFP[He et al., 2018]	-	93.15	75.63	43.83	44.10	71.462	0.481
TMI-GKP [Zhong et al., 2022]	Best Benign	93.95	75.18	42.18	43.46	71.855	0.482
Best (a)	93.37	75.88	42.55	43.74
Best (b)	93.66	75.74	44.09	44.51
SR-GKP (Ours)	Best Benign	94.08	75.89	42.60	43.85	71.855	0.482
Best (a)	93.83	76.40	45.17	44.85
Best (b)
Table 3: Methods ranked against each other on each model with pruning rate  $\approx 43.75\%$ , lower is better. "ResNet-XX Mean Rank" means a method's average ranks across four metrics on ResNet-XX; e.g., SR-GKP is ranked #1/#1/#4/#2 for its best performance across benign/FGSM 0.01/FGSM 0.1/PGD metrics on ResNet-56 against other methods, so it'd have a ResNet-56 Mean Rank of  $(1 + 1 + 4 + 2) / 4 = \# 2$ . Methods with incomplete presence across the three models are excluded. This table is provided to facilitate the digestion of Table 2, 11, 12, and 13, which consist of raw results.
Method	ResNet-32 Mean Rank	ResNet-56 Mean Rank	ResNet-110 Mean Rank	All Models Mean Rank
CC [Li et al., 2021]	#5.5	#6.5	#9.25	#7.08
DHP [Li et al., 2020]	#10.5	#8.75	#12.25	#10.5
FPGM [He et al., 2019]	#4.75	#4	#2.5	#3.75
L1Norm-A [Li et al., 2017]	#7.75	#7	#8.75	#7.83
L1Norm-B [Li et al., 2017]	#4	#6.75	#8	#6.25
LRF [Joo et al., 2021]	#7.75	#8.75	#7	#7.83
NPPM [Gao et al., 2021]	#6.5	#8	#8.25	#7.58
OToV2 (from-scratch) [Chen et al., 2023]	#12.5	#12.5	#12.25	#12.42
OToV2 (post-train) [Chen et al., 2023]	#6.75	#8.75	#6	#7.17
RAP-ADMM [Ye et al., 2019]	#7	#7.5	#7	#7.17
SFP [He et al., 2018]	#9	#6.25	#3	#6.08
TMI-GKP [Zhong et al., 2022]	#5.25	#4.25	#4.25	#4.58
SR-GKP (Ours)	#3.75	#2	#2.5	#2.75
Table 4: ResNet-50 on ImageNet-1k. Note this table includes two baselines: "self-trained" and "torchvision". This is because TMI-GKP requires training epoch snapshots, which is not supplied with the torchvision pretrained ResNet-50. Methods included in this table are either well-performing methods under the CIFAR-10 experiments or specifically recommended by the reviewers.
Method	Baseline	Benign	FGSMε=0.001	FGSMε=0.01	FGSMε=0.1	PGDε=4/255, εsep=1/255 maxiter=3	MACs (M)	Params (M)
Unpruned	Self-trained	75.70	67.57	25.82	16.16	6.66	4122.828	25.557
TMI-GKP [Zhong et al., 2022]	Self-trained	75.02	67.62	25.86	15.82	7.26	2725.954	17.069
SR-GKP	75.29	68.02	26.45	15.83	8.08	2725.954	17.069
Unpruned	torchvision	76.13	70.18	28.70	13.93	9.27	4122.828	25.557
DFPC [Narshana et al., 2022]	torchvision	73.80	67.45	25.15	12.24	7.43	-	-
FPGM [He et al., 2019]	torchvision	75.04	68.50	25.84	13.06	7.43	2641.670	18.310
OToV2 (post-train) [Chen et al., 2023]	torchvision	75.38	68.04	21.01	12.76	5.26	-	-
SFP [He et al., 2018]	torchvision	58.50	55.72	25.82	9.01	11.00	2635.129	17.302
SR-GKP	torchvision	75.34	68.04	26.65	15.94	7.85	2759.672	17.803
defense to other adversarial attacks, even though they are similar in nature. Also, RAP-ADMM has the worst benign performance across all showcased methods, yet its training time is significantly longer due to the need to perturb its training data on the fly constantly.
Table 2 (and similar experiments showcased in Appendix D) may answer one of our research questions: are carefully designed modern structured pruning methods also fragile under adversarial attacks? The answer is an unfortunate "Yes," as not only do recent structured pruning methods show serious performance drops under adversarial attacks, such drops are often more severe than their predecessors — which often rely on much naïve designs. Figure 5 as well as the ranked chart Table 3 provide a vivid illustration with LRF [Joo et al., 2021] — a 2021 method — showing one of the weakest adversially robustness across all evaluated methods.
Upon careful comparison, we noticed that SFP [He et al., 2018] and FPGM [He et al., 2019] — two pre-2020 methods — tend to be the best filter pruning methods under the double scrutiny of benign and adversarial tasks. However, this only holds true to smaller scale experiments, as SFP is significantly outperformed by SR-GKP on ResNet-50 on ImageNet for benign accuracy (58.50% v. 74.34%). Though FPGM tends to perform better on the same task, it is still behind SR-GKP in a general sense (Table 4). We further note ImageNet with adversarial metrics is rarely evaluated under the context of adversarially robust pruning [Chen et al., 2022, Ye et al., 2019]; this is mainly because many of ImageNet sub-classes are closely related in terms of semantics (e.g., man-eating shark v. tiger shark) [Ozbulak et al., 2021]. We provide the ImageNet results primarily to show SR-GKP can deliver competitive benign ImageNet performance, where most ImageNet results compared in Table 4 are from methods with strong adversarial performance on CIFAR-10.
Last, although TMI-GKP [Zhong et al., 2022] is optimized for benign tasks, and its procedure does not consider adversarial scenarios, it naturally comes with strong adversarial robustness. We believe this has a lot to do with the increased pruning freedom enabled by the GKP granularity, which further highlights the potential of GKP-based methods outside its achievements in the benign space. Due to the page limitation, we hereby only showcase a much-abbreviated version of our experiments. We strongly encourage our readers to check out our full experiments at Appendix D with a lot more comprehensive coverage on many more dataset-model combinations, as well as ablation studies.
Our work studies the area of adversarially robust structured pruning, a topic presenting severe problems but lacking proper recognition or exploration. On the investigation side, we reveal that just like their naïve predecessors — carefully designed modern structured pruning methods are also fragile under adversarial attacks, yet different pruning methods may yield drastically different adversarial performance while hiding behind similar benign reports. On the solution side, we propose SR-GKP: a simple one-shot GKP method that showcases competitive benign performance with a significant advantage under adversarial attacks against comparable SOTA methods while requiring no extra cost from a pruning procedure perspective.
We believe the overlooked nature of this field is mainly two-fold: the lack of pruning freedom to utilize findings of other fields, and the lack of user-friendly tools and resources for doing adversarial evaluations under a pruning context. We present our take and contribution to both issues by showcasing the capability of GKP-based methods and providing our community with an open-sourced tool and model checkpoints for future studies.
This research was supported, in part, by NSF Awards OAC-2117439, OAC-2112606, IIS-2224843, and IIS-1900990. This work made use of the High Performance Computing Resource in the Core Facility for Advanced Research Computing at Case Western Reserve University. We give our special thanks to the CWRU HPC team for their timely and professional help and maintenance.
D. Blalock, J. J. Gonzalez Ortiz, J. Frankle, and J. Guttag. What is the state of neural network pruning? Proceedings of machine learning and systems, 2:129-146, 2020.
C.-Y. Chang, Y.-N. Chuang, Z. Jiang, K.-H. Lai, A. Jiang, and N. Zou. Coda: Temporal domain generalization via concept drift simulator, 2023a.
C.-Y. Chang, Y.-N. Chuang, K.-H. Lai, X. Han, X. Hu, and N. Zou. Towards assumption-free bias mitigation, 2023b.
C.-Y. Chang, Y.-N. Chuang, G. Wang, M. Du, and N. Zou. Dispel: Domain generalization via domain-specific liberating, 2023c.
J. Chen, M. I. Jordan, and M. J. Wainwright. Hopskipjumpattack: A query-efficient decision-based attack. In 2020 IEEE Symposium on Security and Privacy, SP 2020, San Francisco, CA, USA, May 18-21, 2020, pages 1277-1294. IEEE, 2020. doi: 10.1109/SP40000.2020.00045. URL https://doi.org/10.1109/SP40000.2020.00045.
T. Chen, H. Zhang, Z. Zhang, S. Chang, S. Liu, P. Chen, and Z. Wang. Linearity grafting: Relaxed neuron pruning helps certifiable robustness. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, 2022.
T. Chen, L. Liang, D. Tianyu, Z. Zhu, and I. Zharkov. Ototv2: Automatic, generic, user-friendly. In International Conference on Learning Representations, 2023.
Y.-N. Chuang, G. Wang, F. Yang, Z. Liu, X. Cai, M. Du, and X. Hu. Efficient xai techniques: A taxonomic survey. arXiv preprint arXiv:2302.03225, 2023a.
Y.-N. Chuang, G. Wang, F. Yang, Q. Zhou, P. Tripathi, X. Cai, and X. Hu. Cortex: Contrastive framework for real-time explanation. arXiv preprint arXiv:2303.02794, 2023b.
T. Dao, B. Chen, K. Liang, J. Yang, Z. Song, A. Rudra, and C. Ré. Pixelated butterfly: Simple and efficient sparse training for neural network models, 2022a.
T. Dao, B. Chen, N. S. Sohoni, A. Desai, M. Poli, J. Grogan, A. Liu, A. Rao, A. Rudra, and C. Re. Monarch: Expressive structured matrices for efficient and accurate training. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 4690-4721. PMLR, 17-23 Jul 2022b.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. IEEE, 2009.
T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314, 2023.
J. Frankle and M. Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=rJ1-b3RcF7.
E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh. OPTQ: Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id= tcbBPnfwxS.
S. Zhang, Y. Liu, Y. Sun, and N. Shah. Graph-less neural networks: Teaching old MLPs new tricks via distillation. In International Conference on Learning Representations, 2022b. URL https://openreview.net/forum?id=4p6_5HBWPCw.
Q. Zhao and C. Wressnegger. Holistic adversarially robust pruning. In Proc. of the International Conference on Learning Representations (ICLR), May 2023.
S. Zhong, G. Zhang, N. Huang, and S. Xu. Revisit kernel pruning with lottery regulated grouped convolutions. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=LdEhiMG9WL0.
A Limitation and Broader Impact
Note although our investigation revealed serious performance issues under adversarial tasks in terms of structured pruning methods, where our proposed method provides sensible mitigation; our findings are still limited to the benign and artificially perturbed input. Though we expect grouped kernel pruning-based methods to deliver good performance under other reasonable evaluating metrics due to the improved pruning freedom, we will leave a more comprehensive investigation against other robustness metrics for future work. We also provide a brief discussion of such related metrics in Appendix B.4.
B Extended Related Work and Discussion
B.1 Structured v.s. Unstructured Pruning
Pruning methods can be roughly divided into structured pruning and unstructured pruning according to the pruning granularity. Specifically, unstructured pruning often means we prune each weight independently. In contrast, structured pruning bundles weights into groups, then prune the whole group instead of the individual weight (e.g., block-wise Lagunas et al. [2021], channel-wise He et al. [2017], group-wise pruning Zhong et al. [2022]).
Unstructured pruning can maintain the model performance better with the same number of parameters. However, unstructured pruned models yield marginal wall-clock time efficiency or even slower than the unpruned model at the low sparsity regime. This is because unstructured pruned matrices need to be stored in sparse matrix format, as the zeros are randomly distributed in these matrices Yang et al. [2018]. Operations executed on sparse matrices (e.g., sparse matrix multiplication, sparse embedding table look-up) are notoriously inefficient on commodity hardware, e.g., GPUs and CPUs, due to the limited data reuse and random memory access Yang et al. [2018], Han et al. [2016].
In contrast, although structured pruning has less flexibility compared to unstructured one, it is much more hardware/library-friendly since structurally pruned matrices can still be stored in the dense matrix format. Thus, operations executed on structured pruned matrices are the same as those in the unpruned model, which are highly optimized. Consequently, the compression provided by structured pruning can often translate into the real wall-clock time speedup upon proper implementation.
B.2 Unstructured Pruning for Adversarial Robustness
In Section 2, we introduced the few structured pruning methods which claim increased adversarial robustness, indicating the lack of presence of adversarially robust structured pruning methods. However, many unstructured pruning literature have explored the possibility of pruned models with improved adversarial robustness, such as Sehwag et al. [2020], Zhao and Wressnegger [2023], Ye et al. [2019], Jian et al. [2022], Li et al. [2022], Gui et al. [2019]. Given it is natural for scholars to adopt insights from unstructured pruning methods into a structured context, we hereby provide this list of unstructured pruning arts for inspiration purposes.
B.3 Learning of High-Frequency Components and its Adversarial Implications
As we consult adversarial-robustness-boosting kernel metrics and operations, we heavily rely on the findings from Wang et al. [2020b], a work that discusses how learning components with different frequencies may affect the adversarial robustness of a CNN, and how some kernel-level metrics like kernel smoothness influence such type of learning. We have elaborated more on this in the Section 3.1 above. Following work can also be referenced when it comes to how component frequency may affect (or be adopted to improve) adversarial robustness, efficiency, and general neural network learning [Wang et al., 2020a, 2023b,a].
B.4 More on ML efficiency and AI safety