One Less Reason for Filter Pruning: Gaining Free Adversarial Robustness with Structured Grouped Kernel Pruning
Authors: Shaochen Zhong
Zaichuan You
Jiamu Zhang
Sebastian Zhao
Zachary LeClaire
Zirui Liu
Daochen Zha
Vipin Chaudhary
Shuai Xu
Xia Hu
Venue: NeurIPS 2023 poster (2023)
Keywords: pruning
structured pruning
adversarial robustness
grouped kernel pruning
CNN
one-shot
TL;DR: Structurally pruned models can exhibit similar benign performance while having drastic (and different) drops on adversarial tasks. We mitigate the issue with a one-shot kernel smoothness-aware grouped kernel pruning procedure requiring no extra cost.

Abstract: Densely structured pruning methods utilizing simple pruning heuristics can deliver immediate compression and acceleration benefits with acceptable benign performances. However, empirical findings indicate such naively pruned networks are extremely fragile under simple adversarial attacks. Naturally, we would be interested in knowing if such a phenomenon also holds for carefully designed modern structured pruning methods. If so, then to what extent is the severity? And what kind of remedies are available? Unfortunately, both the questions and the solution remain largely unaddressed: no prior art is able to provide a thorough investigation on the adversarial performance of modern structured pruning methods (spoiler: it is not good), yet the few works that attempt to provide mitigation often do so at various extra costs with only to-be-desired performance.

In this work, we answer both questions by fairly and comprehensively investigating the adversarial performance of 10+ popular structured pruning methods. Solution-wise, we take advantage of *Grouped Kernel Pruning (GKP)*'s recent success in pushing densely structured pruning freedom to a more fine-grained level. By mixing up kernel smoothness — a classic robustness-related kernel-level metric — into a modified GKP procedure, we present a one-shot-post-train-weight-dependent GKP method capable of advancing SOTA performance on both the benign and adversarial scale, while requiring no extra (in fact, often less) cost than a standard pruning procedure. Please refer to our [GitHub repository](https://github.com/henryzhongsc/adv_robust_gkp) for code implementation, tool sharing, and model checkpoints.

One Less Reason for Filter-Pruning: Gaining Free Adversarial Robustness with Structured Grouped Kernel Pruning
Shaochen (Henry) Zhong' Zaichuan You 22 Jiamu Zhang", Sebastian Zhao 3 Zachary LeClaire", Zirui Liu', Daochen Zha', Vipin Chaudhary?. Shuai Xu², and Xia Hul
DDepartment of Computer Science, Rice University
baaaachen.zhongg zirui 1iu, daochen. zha, xia.hu)@rii wrwichurrree...eeeee edu 2 Department of Computer and Data Sciences, Case Western Reserve University
{zxy456, jxz1217, , 2j116, vipin, sxx214) @case edu Electrical Engineering and Computer Sciences, UC Berkeley sebbyzhao@berkeley edu
Abstract
Densely structured pruning methods utilizing simple pruning heuristics de- liver immediate compression and acceleration benefits with acceptable benign performances. However, empirical findings indicate such naively pruned networks are extremely fragile under simple adversarial attacks. Naturally, we would be interested in knowing if such a phenomenon also holds for carefully designed modern structured pruning methods. If so, then to what extent is the severity? And what kind of remedies are available? Unfortunately. both questions remain largely unaddressed: no prior art is able to provide a thorough investigation on the adversarial performance of modern structured pruning methods (spoiler: it is not good), yet the few works that attempt to provide mitigation often do so at various extra costs with only to-be-desired performance.
In this work, we answer both questions by fairly and comprehensively investigating the adversarial performance of 10+ popular structured pruning methods. Solution- wise, we take advantage Grouped Kernel Pruning (GKP)'s recent success pushing densely structured pruning freedom to a more fne-grained level. By mixing kernel smoothness - a classic robustness-related kernel-level metric into modified GKP procedure, we present GKP method capable of advancing SOTA performance on both the benign and adversarial scale, while requiring no extra (in fact, often less) cost than standard pruning procedure. Please refer to our GitHub repository for code implementation, tool sharing, and model checkpoints.
1 Introduction
Convolutional neural networks (CNNs) have demonstrated solid performance on tasks centered around computer vision. However, with modern CNNs growing in both widths and depths, the issue of over-parameterization has drawn increasing attention due to such networks often requiring large computational resources and memory capacity. To mitigate the burden, network pruning the study of removing redundant parameters from original networks without significant performance loss has become popular approach for its simplicity and directness [LeCun et al. 1989 Blalock et al. 2020
He and Xiao 2023].
Equal contribution. Order determined alphabetically by last names.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Despite the popularity of the pruning field in general, few prior arts have been available to provide improved adversarial robustness under the constraint of (densely) structured pruning: even though empirical findings show vanilla structured pruning methods implemented with naïve pruning strategies often experience huge performance drop on such adversarial tasks [Wang et al.) 20018] Sehwag et al., 2020) Vemparala et al.) 202T]. More concerning, no prior art has made an eftort to provide a comprchensive investigation on whether the same phenomenon also exists under carefully designed modern structured pruning methods, where such methods are often capable of delivering excellent benign accuracy retention after pruning (sometimes, even improvements).
Below, we provide a walk-througe of why densely structured method and having an adversarially robust pruned model are preferable and important, how we developed our solution by leveraging the power of increased structural pruning freedom (grouped kernel pruning) with kernel-level metrics (kernel smoothness). the later sections this paper, we replicate and evaluate around 13 popular densely structured pruning methods and variants against various white box (evasion) adversarial attacks, where our proposed method showcases clear dominance.
Structured v.s. Unstructured Pruning: Accuracy-Efficieny Trade-off
Most of the existing CNN pruning methods be roughly categorized into structured and unstruc- tured pruning. Note we said roughly because there is no universally agreed delineation between structured and unstructured pruning methods. The general consensus is that methods considered more unstructured often enjoy higher degree of freedom on where to apply their pruning strategies (e.g., weight-level pruning) and therefore result in better accuracy retention.
In contrast, structured pruning methods often prune weights in a grouped manner following some kind of architecturally defined constraints (e.g., filter-level pruning). Compared to their unstructured counterparts, structured pruning methods are more hardware-friendl and easier to obtain compression and acceleration commodity hardware, though at the cost of worse accuracy retention. This difference due to unstructurally pruned networks' tendency to have pruned (zeroed) parameters randomly distributed in the weight matrix, leading poor data reuse and locality (Yang et al. 2018 Such unstructurally pruned networks struggle to have wall-clock time speed up without supports like custom-indexing, special operation design, sparse operation libraries, even dedicated hardware setups Yang et al. 2018 Han et al. 2016 He and Xiao, 2023].
popular line of research is to produce pruned networks that are entirely dense, a.k.a. densely structured, where the pruned weights are stored in the normal dense tensor format. Such format of pruned network considered to be most library/hardware- friendly and, therefore, most deployable in a practical context. With such significant benefits, densely structured pruning methods consist of the absolute majority of structured pruning methods.
Among all structured pruning methods,
Filer
Kernel
Channel
Grouped Kemel
3
Cutbu Urameesmmee
3
4
5
3
5
2
Inout Channels
Grouped
Unprumod Original
FitterChannnll Pruning
Komel Pruning
Grouped Kernel Pruning
Figure 1: Visualization of different pruning granularities.
Densely structured pruning methods come with different pruning granularities, where significant portion of prior arts prune at filter or channel level. These two types of pruning are historically considered to be the limit of densely structured pruning, as showcased in Figure if we go down one more level, we will have kernel pruning; however, its pruned network is not dense. This is until recently, authors from Zhong et al. [002 utilized kernel pruning with grouped convolution, where they prune at a grouped kernel level to ensure an entirely dense pruned network. To the best of our knowledge, grouped kernel pruning carries the highest degree of pruning freedom among all densely structured pruning methods.

---

1.2 Pruned Models Are Fragile Under Adversarial Attacks - But Why Do We Care?
Empirical findings like Wang et al. [2018] suggest that although pruned neural networks may have acceptable benign accuracy. they are often more vulnerable to adversarial attacks. Adversarial robustness is recognized as a Long-standing metric evaluate model quality, as a model with undesired adversarial robustness can be easily exploited to produce wrong and potentially harmful output, resulting in fairness and accountability issues.
We would argue such robustness properties are especially valued under the context of (struc- tured) pruning, where pruned models are often deployed to resource-constraint devices with less central oversight available and requiring execution in a more real-time manner. Imagine if an OCR model for real-time check redeeming can be maliciously exploited read the number 9; the result will surely be unpleasant for many parties involved.
To alleviate such problem (though not under a pruning context), prior arts like Wang al. [2020b]
demonstrate the adversarial robustness of a convolutional network is largely correlated to its sensi- tiveness high-frequency components (HFC), where such sensitiveness simple kemnel-eevll operations like kernel smoothness. More on this in Section 2
be mitigated with some
1.3 Solution: Grouped Kernel Pruning with
-eririal--ooouutss--BBaaaee Boossting Kernel Metries
With the recent Grouped Kernel Praming (GKP) framework pushing the pruning freedom of densely structured pruning to (close) kemel-level Zhong et al., 2022], we explore the unique possibility of mixing up anwersariallrobuuttss---beeee the procedure of GKP. We present Smoothly Robust Grouped Kernel Pruning (SR-GKP), a densely structured pruning method that works in simple post-train one-shot manner, but is often capable of delivering competitive benign performance and much stronger adversarial performance against. SOTA filter and channel pruning methods that require more sophisticated procedures. Solution-wise our main claims and contributions are:
kernel metrics such as kernel smoothness
into
Free improvement on adversarial robustness. Our method has no extra (in fact, often less) cost compared to a standard pruning method, making the gained adversarial robustness entirely free. One-shot & post-train & weight-dependent: the simplest procedure with most compatibility. Our method is a mue-shot-post-rrase trxi-trrin---wiigtt -eigghteeeeddenn structured pruning method for CNN that follows the classic train prune- fine-tune procedure, where all excessive components are pruned all at once before fine-tuning. This means it is compatible with any trained CNNs (as it does not interfere with the training pipeline) yet straightforward to execute, the pruning procedure before fine-tuning does not rely on access to data but only the trained model's parameters.
Raise attention to the important but overlooked field of adversarially robust structured pruning. Our method is among the few structured pruning methods capable of delivering pruned networks with improved adversarial performance field with severe problems, but receives little recognition or solutions.
On the investigation side, we are the first to comprchensively reveal:
Drastic adversarial performance difference under similar benign report. We found that while different carefully designed modern densely structured pruning methods may showcase similar benign performance, most are done so at the cost of adversarial robustness.
One less reason for filter/channel pruning: further endorsing GKP. Filter and channel pruning methods have dominated the field of densely structured pruning for years; our work together with /Zhng et al. 2022) and Park et al. 2023) showcased that when done right, grouped kernel pruning-based based methods are superior under both benign and adversarial tasks, making it a promising direction for future densely structured pruning research.
'Note the term "one-shot" can be ambiguous under pruning context. Some literature, including ours, refer to "onc-shot" as the pruning procedure is done all at once before fine-tuning [Liu et al. 2009 Frankle and
Carbin 2019 Renda el al. 2020]: yet, some other work refers to "one-shot" as without requiring a fine- tuning
stage [Chen et al. 2023 Prantar et al. 2023). We hereby clarify that our proposed method still requires
fine-tuning stage after pruning. Further, Only-Traine Train-Once (OTO) [Chen al. 2023 a popular series of
iterative From-sccatch pruning methods known for its automatically applicable property utilizes the term
"one-shot" in a manner in line with "one-stop." highlighting its end-to-end nature and aggess-gnootic capability.
For disambiguation, we hereby clarify that our method is, in fact, limited to CNN pruning. requires a trained model to start with, and does not possess such model-agnostic automatically applicable properties.
3

---

For added bonuses, we are the first ones to reproduce and comprehensively report the benign and adversarial performances of multiple structured pruning methods under fair setting. We believe the lack of such fair and comprehensive reports (on both benign and adversarial tasks) is mainly due to the lack of user-friendly tools. Thus, alongside our method implementation and checkpoint files, we also provide the pruning community a lightweight open-sourced tool capable of plug-and-play style of testing different victim models with various adversarial attacks while supporting all procedures modern pruning method may require.
2 Related Work and Discussion
Due to the page limitation, we will discuss related work regarding white box evasion adversarial attacks, adversarially robust structured pruning, and grouped kernel pruning. Other related topics, such the compressionnacceleaatio implications of structured and unstructured pruning methods, input component frequency with kernel smoothness, as well as other related MI. efficiency and AT safety literature, will be introduced in Appendix B
Adversarial Attacks. Neural networks are known to be vulnerable to adversarial attacks, i.e., small perturbation applied to the inputs can mislead models to make wrong predictions SZegedy et al. [2014). Goodfellow et al. (2014). In practice, popular adversarial attacks can often be categorized as white-box and black-box evasion attacks. The difference being white-box attacks have access to the entirety of the model, including input features, architectures, and model parameters, while black-box attacks' access is often constrained (e.g., only input-output pair). Thus, white-box attacks are almost always more effective and efficient than black-box ones; in fact, many classic black-box attacks constructed in a way to approximate the information that is directly accessible by white-box attacks (e.g., gradient) [Chen al. 2020). We opt for white-box attacks for the scope of this paper, given one significant use case of structurally pruned models is edge-device deployments, where the model is more likely to be accessed. Also, white-box attacks generally offer a more straightforward workflow with harder challenges posed.
Structured Pruning for Adversarial Robustness. Structured pruning methods, which arguably carry the most practical significance, have been heavily studied throughout the years (Molchanow
eet al.
2017 Yu et al. 2018 He et al. 2019 Wang et al. 2019a b Lin et al. 2019 He et al.) [0118] Li al.
2021 Zhong et al. 2022]. Despite their popularity, few of them focus on adversarial robustness.
To the hest of our knowledse ledge, there are only four prior arts presenting structured pruning methods while claiming improved performance
adversarial robustness metrics (Vemparala al. 2021
Ye et al. 2019 SShwagg et al. 2220 Zhao and Wressnegger 2023)_ Unfortunately, Vemparala et al.
[022] does not have a public repository for code, Ye et al. (2019) does not have any experiment
on standard BasicBlock ResNets for comparative investigation despite their popularity. [Blalock
al. 2020). , Sehwag et al. [2020] and ZZao and Wressnegger [2023] mostly propose unstructured
methods with only few structurally pruned ablation studies conducted on limited model-dataset
combinations, with some of their structured pruming implementation not published.
The lack of traffic, infrastructure, or baseline in this area has undoubtedly created deterrents to all interested scholars. To fill the gap, we provide the community open-sourced toolkit capable of testing various victim models against different adversarial attacks under a pruning context, coming with 10+ popular structured pruning methods already integrated for comparative evaluation. Model checkpoints are also available for direct evaluations.
Grouped Kernel Pruning. Our work relics on the GKP framework - particularly inspired by the recent work of Zhong et al. [2022]. The core of GKP is grouped kernel pruning and reconstruction to a grouped convolution format, where this combination has been explored a few times under the context of structure pruning.
Specifically, Zhong et al. [202]] proposed their take on grouped kernel pruning with three-stage procedure: filter clustering, generating then deciding which grouped kernel pruning strategy to employ, then reconstructing to grouped convolution format via permutation. This particular framework solved the previous drawback of requiring complex procedure, yet rich set of experiment results was showcased to demonstrate its performance advantages against many SOTA structured pruning methods. Our proposed method is largely enabled by the extra pruning freedom that GKP provides.

---

Outside of Zhong et al. [202]], concurrent work like Zhang et al. [2022a] and follow-up work like Park et al. 2023 have showcased the excellent benign performance of different GKP implementations under benign context.
Proposed Method
3.1
Motivation: Pruning May Amplify Overfitting High-Prequency Components
Wang et al. 2020b] suggests CNNs are prone overfitting figh-frryeenyy components (HFC) of inputs - type of feature that not robust yet can be easily replicated with adversarial perturbations. Though Wang's 's finding towards unpruned model, such phenomenon be found, and in fact, even amplified, under a structural pruning setting.
Input
Reconstructed with LFC
Reconstructed with HFC
Naive
Unpruned
Output
Pruned
Figure 2: A frog figure from CIFAR-10 test set in its original, LFC, and UFCCrreconttuucted formats. The output indicates the correctness classification results when testing through an unpruned and LINorm pruned ResNet-56 (pruning rate 43.75% be consistent with Table 22)
classic demonstration of such phenomena can be seen Figure Despite the frog-labeled
figure reconstructed with only low-frequency components showing visible resemblance to its original benign format, ResNet-56 model pruned by LINorm filter pruning [Li al., 2017] cannot classify correctly. However, such pruned models can somehow correctly classify the same input reconstructed with only HFCs, even if it already lost all semantics of frog to human audience. This indicates model pruned by methods without having adversarial robustness in consideration is more likely to overfit to HFC.
Table 1: Unpruned and structurally pruned ResNet-56 v. HFC//LFC
CIFAR-I0 test set. represents the cutoff threshold (for 0 S S 1). With a higher 0, the HFC-rrconstrructed images will exclusively include more frequeeyy information; vice-versa for a lower A. Pruning rate z 43.75%; prumed model checkpoints are taken from Table
INPUT
UNPRUNED BASELINE
CC PRUNED
NPPM PRUNED LINORM-B PRUNED
FULL 0.0)
93.24
94.04
93.55
92.62
HFC (0 00.3
77.05
80.22
78.08
79.83
HFC (0 = =0.5)
50.77
57.49
55.47
56.06
HFC (0 (0=0.77 -0.7)
22.79
25.78
21.92
27.15
We emphasize that the above example (Figure 2 is not cherry-picked one. As shown in Table 1 by reconstructing the entire test CIFAR-10 10 with solely their high-frequency components, we find that a structurally pruned model is more prone to fitting HFCs than its unpruned counterpart under various settings. This is potentially because HFCs are easily learnable features under a benign setting, so pruned models want to "make most use" of their remaining weights given the reduced network capacity, and therefore become even more overfitted to HFCs by treating them as short-cut features.
Kernel smoothhess as an indicator for learning from HFC. Fortunately, Wang et al. [2020b suggests kernel smoothness highly correlated the learning HFC. Specifically, Wang al. (20206) found that CNN with "smoother" kemnels where neighbor weights within a 2D kernel have less of value difference - will reduce the overfitting of HFCs, thus making the model more robust against adversarial perturbations. For the ease of the following conversation, we define the kernel smoothness of convolution kernel for I. Hx to be:
hxw
smoothness(k)
Σ jvvaluss border with
K2 k
(1)
5

---

where H and W are the kernel dimensions; in most CNNs, such dimensions are set to 3 x 3.
The finding of kernel smoothness and adversarial robustness presents unique opportunity under the context of GKP. Prior to GKP, densely structured pruning is almost always done at filter/channel level, where kernel-level metrics/operation have little bearing when relaxed (see Appendix C.1.3) However, GKP prunes at (close) kernel level, where a kernel-level metric may still retain its power.
3.2 Mixing Smoothness into Grouped Kernel Pruning Procedure
It is natural to want to encapsulate some kernel-level operations/metries - in this case, kernel smoothness into the procedure of GKP. However, the challenge comes with how we can do an efficient and effective manner. Particularly, how can we achieve improved adversarial performance without sacrificing benign tasks?
For the ease of illustration, W E RCC layer, which consist of C filters, with each filter consisting of C number of H x W 2D kernels. According to Zhong et al. [2022], a standard GKP procedure has two potential stages:
be the weight of the l-th convolutional
Stage 1: Filter grouping stage: where the C filters are clustered into equal-sized filter groups (FG,, FG FG with each filter group FG RC////xC x
Stage 2: Pruning strategies obtaining/pruning stage: where the pruning method generates a set of "candidate" grouped kernel pruning strategies to be evaluated and select from; a grouped kernel is defined GK FG with GK having shape of C / Finally, we evaluate all collected candidate strategies and decide which to pursue.
x H W
However, how apply kernel smoothness-related criteria to such stages is non-trivial question. Our empirical results from some proof-of-concepy experiments suggest some naïve applications either will not work at all or will only work by significantly sacrificing the performance we simply replace Stage 2 above by pruning the grouped kernels with greater >.oO smoothness(k) will experience a huge drop benign accuracy (Appendix C). Therefore, must drive our attention to discover some more sophisticated ways of mixing up such criteria into the above stages.
benign tasks: e.g., if
3.2.1 Smoothness Snaking: Is Filter Clustering the Only Right Answer for GKP?
Per TMI-GKP [Zhong et al. 2022). filters within the same convolutional layer are grouped into several equal-sized groups by a filter clustering schema, which consist some different combinations of dimensionality reduction and clustering techniques. The motivation of grouping by clustering is natural as establishes preferable search space for pruning algorithms, where the power of pruned components likely be retrained via similar unpruned components within the same group. However, later procedures of TMI-GKP (e.g., Stage per Section (32)) then seek to maintain a diverse representation of kept components within each filter group. which begs the question: is filter clustering the only right answer for GKP? If finding diverse set of unpruned grouped kernels the goal, why not have filter groups with filter diversity to start with? And if the answer the first question is "No." t How can we utilize this opportunity to mix up with kernel smoothness criteria?
Grouped Kerne
3
5
2
Grag
Go0pp
3 Ingar Charnels
Sort Smoothness (langer bar as being less smooth)
Propare in Filter Groupe to Fill
Form Grouped Conv Layer with
Unpruned Conv Layer
F5
F6
F2
aeoignments
Figure 3: Smoothness Snaking For Filter Grouping (Stage 1)

---

Grouped- Srouped-based based pruning will always seek out some kind of balance among groups to avoid skewed distribution where all important components are distributed certain groups [Zhong et al., 2022]. Following this principle, we want our filter groups to have balanced smoothness so that we do not end up having any filter group that is "over" or "under-smoothed" to start with for pruning. However, this is equivalent to the partition problem [Korf 1998), which is known to be NP-hard. Given the filter grouping stage is often robust to adjustments we proposed to sort filters according to their smoothness, then assign them iteratively in an S-shaped "snaking" manner across predefined number of filter groups, as shown in Figure 3. namely Smoothness Snaking. Empirical results suggest although smoothness snaking may not be optimal as the dynamic clustering scheme in Zhong et al. [2022] in terms of benign performance, it is able to provide better adversarial robustness under ad udversarial attacks and is much faster to execute (Table g due to the absence of dimensionality reduction & clustering procedures (Appendix C). We consider this to be a successful mix-up.
3.2.2 Smooth Beam Greedy GKP Search
Knowing that the pruning strategies obtaining/decision stage (Stage per Section 3 3.2) of GKP is
sensitive to tampering, a direct application of smoothness criteria would not work (Appendix C This indicates the distance-basee cost formula (Equation 5) proposed in Zhong al. [0022] carries a significant influence on the performance of a pruned network, which is unsurprising given the
vast popularity of distance-based pruning arts Since we can't directly replace this cost formula, we propose to widen the capacity of pruning strategies obtaining stage with smoothness in mind
while keeping the decision stage akin the cost formula. By doing this, the chances of more "smoothness-ut aware" pruning strategies being employed are increased.
Grouped Hardd
Kepp (iKe
Branch g1t:
CKI
Kept Orike
2
2
4
6
Owen unpraned Grouped Fiter
ou muttole Deside Candititte by their Cont - aud cean distnce to Kept OKk Branches Digins boing more diste
Iej oa as Kapt OKs Higher ond rark indinee am prrtermet
Sort Candidate Gite their Cost to Kept CKks Branches
O 26
Kept
OF1
3-2
Branch
Kept GiKss
maii
2 0.25 5 2.25
Branch
8.75
Branch
1-0
Add OKs Kept OKo
Branches
(g) by of as Kept 25 toxos higher curderdd ranis indlcos and pndeene
Og Back shop (ieparor the tranch caatchos
(fighir hecles being more detave to Kapt (icilmmcteer
Figure 4: Smoothness-awart Beam Greedy Search for Grouped Kernel Pruning (Stage 2).
Again, to keep our approach simple, we implemented a custom beam search element to consider more grouped kernels during each advancement (Figure 4h. Per each iteration, all gathered candidate grouped kernel pruning strategies will be evaluated against a mixture of smoothness and cost criteria, where only Beamwidth amount of grouped kernels will be kept for further advancement (until the desired pruning ratio is reached). The scoring formula to determine which subset of strategies may be kept in the beam defined as (larger scores are better/preferred):
S (ranhanch a)) = a (((Cost(G(Kenncc)) + (1 a) +0 (GaoothhnnsssGKaaaann (2) mix up
where @CCriterion (GKoaanhh) for Criterion e [Cost, Smoothness} represents the rank of such G.braahh when all collected G. branch candidates are sorted according to the given Criterion in a descending order. So Φi (Smoothness (CaeeesssGG.ra G b) = would suggest this particular GK has a greater smoothness ranking (a.k.a. "less smooth") then all other G'Krranch candidates in considerations. a is tunable balancing parameter that adjusts the importance of one criterion over
In Zhong et al. 22022] various filter grouping strategies - including random assignment were proposed. Many of them tend to perform reasonably well, albeit less performant nor stable than the proposed dri-eriven solution.
7

---

the other. The Smoothness GK branch) equation is simply the sum of all kernels within all GK kept over Equation [ the Cost MosttGGibann branch) equation is a modified version of Equation 5 in Zhong al [2022]] J where we removed some hyperparameters for simplicity and to reduce tuning workload. Please refer to Figure and Appendix C33 for more details.
4 Experiments and Results
4.1 Experiment Setups
We evaluate the efficacy of our method on ResNet-32/56/110 with the BasicBlock implementation. ResNet-50/101 with the BottleNeck implementation, and VGG-16 (He al. 2016 Simonyan and Zisserman 2015]. For datasets, we choose CIFAR-10 [Krizhevsky] 2009). Tiny- T Jmayy-maeeett [Wu et al. 2017], and ImageNet-IK (Deng al. 2009] for wide range of coverage. For all compared methods and under most model-dataset combinations, we tried our best to replicate them with a R 300 epochs (except for ImageNet, where we only employ 100 epochs) of tneetnnnnng/rttrannnng budget while maintaining all other settings either identical or proportional to their original publications.
4.2 Compared Methods and Evaluating Criteria
We evaluate our proposed method against up to 13 popular densely structured pruning methods and variants shown in Appendix [ Whenever possible, we produce pruned-and-fing ain-fiinttuuee (or retrained) models upon identical unpruned baseline models with similar post-prune MACs and Params. Then, we compare their inference accuracy on benign inputs as well as powered by FGSM and PGD attacks in various perturbation budgets and intensities [Goodfellow et al. 2014 Madry et al. 2018).
inputs
One reporting mechanism that probably unique to our paper in comparison to standard pruning art under the benign space - is for some of our methods, results of multiple epochs are reported, each showcasing method's peak performance against different evaluation metrics (a.k.a. "supersccre").. This is because for benign tasks, following Li et al. 2017) and He et al. [2019] only the epoch checkpoint with the best benign accuracy needs to be reported. But under adversarial context, if a pruning method is capable of producing more than one fully pruned model during the tinetuunnggretraininng stage, oftentimes, the best performer per each evaluation metric does overlap. We believe it is responsible to report them all as there are no dominant evaluation metrics in our experiments; however, this an important advantage for methods that can generate multiple usable pruned models (c.g., one-shot pruning) over methods that can only generate a few or just one fully pruned model (c.g.. layer-wise iterative pruning). We survey the availability of such checkpoints and their epoch cutoff in Table TO "Fully Pruned Epoch" column.
We also comprehensively investigate each pruning method against checklist of questions presented in To A Appendix (D.13) including pruning granularity, procedure, when is the first fully pruned epoch, zero-masked or hard-pruning, and many other important questions. This investigation, along with the epoch budget constraint and baseline control, should provide our audience with a more leveled playing field for fair and informed methods comparison.
Further, to facilitate digesting mass results reported across many methods/variants and against various metrics, present large-scale empirical evaluation in three different ways. Take BasicBlock ResNets on CIFAR-10 as an example (with pruning rate Re 43. 13.75%). we present the raw results as Table TI 12 and 13 where visualize them accordingly as Figure 6 7 and g Last, make a rank chart, Table to provide a straightforward gauge of the competitiveness among different pruning methods.
4.3 Results and Analysis
Our abbreviated results Table and Table showcased the performance of various modern SOTA methods as well as our proposed methods on the two most popular model-dataset combinations [Blalock et al. 2020). For ResNet-56 on CIFAR- 10, our method outperformed every other method on all evaluating criteria, except for PGD; as RAP-ADMM (Ye et al. 2019 outperformed SR-GKP significantly with 61. 16% v. 44.85%. However, it is worth noting that RAP-ADMM does adversarial training on Pert-rerurredd data, so it is not surprising that it performs well against the seen type of attack. Unfortunately, it seems like the adversarially trained RAP-ADMM cannot generalize its
8

---

Benign
8/255, 6, k_step 2/255, maxx fier
100
04.04
60.93
$0.03
$4.00
80
75.15
75.75
75.16
74.83
78:22
Accuracy
$1.12
20433
45.18
Unpruned
CC
FPGM
L1Norm-B
LRF
RAP-ADMM SR-GKP (Ours)
Figure 5: Visualization of ResNet-56 on CIFAR-10 with pruning rate =: 43.75% - note this plot is
done in "superscore" manner for concise presentation; the four bars of method may not belong to the same model checkpoint.
Table 2: ResNet-56 on CIFAR- 10. All pruning methods are performed on the same baseline model. "Best (a)" represents the performance of model checkpoint that meats the showcased MACs/Params reduction that performs the best against criterion (a).
Benign
PGSM,
FGSM_-01
(M)
Unpruned
93.24
75.15
39.64
42.58
126.561
0.853
Best Benign
94,04
74.78
29.25
37.85
93.73
75.75
38.86
69,837
0.616
(h)
Best Benign
93.60
75.31
43.20
43.55
FPOM He 2019
Best (x)
93.37
76.28
44,96
41.6
71,661
0,482
HRank Lin et 2021)
Best (b)
Henign
92.27
72.32
19.11
32.51
79.237
0.584
Best tai
92.07
72.59
18.94
32.21
Benign
LINorm-R
(a)
75.16
72.115
Best (b)
91:20
74.40
45.16
43.41
Best Henign
93.93
73.47
25.59
34.86
LRF Joe
2021
tal
93.68
74,80
27.39
36.78
71,009
0.490
(b)
Rest Renign
93.55
74,82
29.07
37.12
NPPM Gio
2122
Best (a)
93.35
75.50
30.29
38.09
70,843
0.601
(b)
93.43
75.27
31.18
38.13
SFP st
M18
93.15
75.63
43.83
44.10
71,462
0.481
Best Benign
93.95
75.18
42.18
43.46
TMI-GKP Zhong
75.88
43.74
71,855
(h)
93.66
Reat Renign
94.08
75.89
42.60
43.85
SR-GKP (Ouns)
Best (a)
(b)
93.83
76.40
45.17
44.85
71,855
0.482
Table 3: Methods ranked against each other on each model with pruning rate Rs 43.75%, lower is better. XResNet-XX Mean Rank" means method's average ranks across four metrics ResNet-XX: e.g., SR-GKP is ranked #1/#1/#4#22 for its best performance across benign/FGSM 0.01/FGSM 0.1/PGD metrics ResNet-56 against other methods, it'd have ResNet-56 Mean Rank of (1+1+4+2)44 = #2. Methods with incomplete presence across the three models are excluded. This table is provided to facilitate the digestion of Table
12 and 13 which consist of raw results.
Method
RosNett32 Mean Rank ResNet-56 Mean Rank
ResNet- 100 Mean Rank
All Models Mean Rank
Lietal al
DHP
A1200
#10.5
#8.75
#12.25
#10.5
FPGM
#1.75
#2,5
43.75
87.75
75
L.Norm-B
2017
44
#6.75
#6.25
LRF
Joo 2.. 2021
#7.75
48.75
47
47.83
OTO>2 Uivonts Strpooa
hen 2023
#12.5
#12.5
#12.25
412.42
OTOx2 (post-train) Con 2024
86.75
48.75
96
#7.17
RAP-ADMM
SEP
Hte 2013
46.25
43
46.08
TMI-CKE" Zaaiing al. 2023
65.25
#6.25
n4.25
#4.58
SR-GKP (Oirs)
#3.75
#2.5
#2.75

---

Table 4: ResNet-50 on ImageNet-Ik. Note this table includes two baselines: "self-trained" and "torchvision". This is because TMI-GKP requires training epoch snapshots, which is not sup- plied with the torchvision on pretrained ResNet-50. Methods included in this table are either well- performing methods under the CIFAR-I0 experiments or specifically recommended by the reviewers.
Method
Baseline
Besign
PGSM
PGSM, -1.1
PGSM,
PGD
11225
MACs (M)
Params (M)
Ungraned
Self-trsined
75.70
67,57
25,82
16.16
6.65
4122,828
25.687
SR. SRPP
TMI-GKP Zhong al. 2012
Sclftriined
75,29
75,02
68.01
67.62
25:86
26.45
15.83
15.82
8,08
7.26
2725.954
2725.954
17:099
17.069
Ungraned
torchvision
76.13
70.15
28.70
13.93
9.27
4122.528
25.557
DEPC Nanlamee dl 5002
turchvisiio
73. 80
67.45
25.15
12.24
7.43
FPGM Te m 2019
torchoi isios
75.0
68.50
25.84
13.06
7.43
2641.670
18.310
OTTs2 morreeaan cile
2025
torchrinion
75.38
68.04
21.01
12.76
5.26
SHP [e et 2013 2013
502 corchrisios
58.50
55.72
25.82
9:01
11.00
2635. 129
17.302
S8--777
curchvistion
75.34
68.04
26.65
15.04
7.85
2759.672
17800
defense to other adversarial attacks, even though they are similar in nature. Also, RAP-ADMM has the worst benign performance across all showcased methods, yet its training time is significantly longer due to the need to perturb its training data on the fly constantly.
(and similar experiments showcased in Appendix D may answer one of our research questions: are carefully designed modern structured pruning methods also fragile under adversarial attacks? The answer is an unfortunate "Yes," as not only do recent structured pruning methods show serious performance drops under adversarial attacks, such drops often more severe than their predecessors - which often rely on much naïve designs. Figure as well as the ranked chart Table 3 provide a vivid illustration with LRF [Joo et al. 2021 the weakest adversarially robustness across all evaluated methods.
202T method- - showing one of
Upon careful comparison, we noticed that SFP [He et al., 2018) and FPGM [He et al. 2019]
two pre-2020 methods- - tend to be the best filter pruming methods under the double scrutiny of benign and adversarial tasks. However, this only holds true to smaller scale experiments, as SFP is significantly outperformed by SR-GKP on ResNet-50 on ImageNet for benign accuracy (58.50% v. 74.34%). Though FPGM tends to perform better on the same task, it is still behind SR-GKP in general sense (Table 4 We further note ImageNet with adversarial metrics is rarely evaluated under the context of adversarially robust pruning [Chen et al., 2022 Ye et al. 2019]; this is mainly because many of ImageNet sub-classes are closely related in terms of semantics (e.g., man- eating shark tiger shark) (Ozbulak al. 2021] We provide the ImageNet results primarily to show SR-GKT can deliver competitive benign ImageNet performance, where most ImageNet results compared in Table 4 are from methods with strong adversarial performance on CIFAR- 10.
Last, although TMI-GKP [Zhog al,, 2022] is optimized for benign tasks, and its procedure does not consider adversarial scenarios, it naturally comes with strong adversarial robustness. We believe this has lot to do with the increased pruning freedom enabled by the GKP granularity, which further highlights the potential of GKP-based methods outside its achievements in the benign space. Due to the page limitation, we hereby only showcase a much-abbreviated version of our experiments. We strongly encourage our readers to check out our full experiments at Appendix D with lot more comprehensive coverage on many more dataset-model combinations, as well as ablation studies.
5
Conclusion
Our work studies the area of adversarially robust structured pruning, topic presenting severe problems but lacking proper recognition or exploration. On the investigation side, we reveal that just like their naïve predecessors
carefully designed modern structured pruning methods are also fragile under adversarial attacks, yet different pruning methods may yield drastically different adversarial performance while hiding behind similar benign reports. On the solution side, we propose SR- SR-GKP: a simple one-shot GKP method that showcases competitive benign performance with a significant advantage under adversarial attacks against comparable SOTA methods while requiring no extra cost from a pruning procedure perspective.
We believe the overlooked nature of this field is mainly two-fold: the lack of pruning freedom to utilize findings of other fields, and the lack of user-friendly tools and resources for doing adversarial evaluations under pruning context. We present our take and contribution to both issues by showcas- ing the capability of GKP-hased methods and providing our community with an open-sourced tool and model checkpoints for future studies.
10

---

Acknowledgments
This research was supported, in part, by NSF Awards OAC-2117439, OAC-2112606, IIS-2224843, and IIS- 1900900. This work made use of the High Performance Computing Resource in the Core Facility for Advanced Research Computing at Case Western Reserve University. We give our special thanks to the CWRU HPC team for their timely and professional help and maintenance.
References
D. Blalock, J. J. Gonzalez Ortiz, J. Frankle, and J. Guttag. What is the state of neural network pruning? Proceedings of machine learning and systems, 2:129- 146, 2020.
C.-Y. Chang, Y.-N. Chuang, Z. Jiang, K.-H. Lai, A. Jiang, and N. Zou. Coda: Temporal domain generalization via concept drift simulator, 2023a.
C.-Y. Chang, Y.-N. Chuang, K.-H. Lai, X. Han, X. Hu, and N. Zou. Towards assumption-free bias mitigation, 2023b.
C.-Y. Chang, Y.-N. Chuang, G. Wang, M. Du, and N. Zou. Dispel: Domain generalization via domain-specific liberating, 2023c.
Chen, I. Jordan, and J. Wainwright. Hopskipjumpattack: query-efficient decision-based attack. In 2020 JEEE Symposium on Security and Privacy, SP 2020, San Francisco, CA, USA, May 18-21, 2020, pages 1277-1294 IEEE, 2020. doi: 10.1100/540000.2000000000 URL
httpp://doi. owg//o.//.... 1010/10///000/00
00200000455
Chen, H. Zhang, Zhang, S. Chang, Liu, P. Chen, and Z. Wang. Linearity grafting: Relaxed neuron pruning helps certifiable robustness. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, 2022.
T. Chen, L. Liang, D. Tianyu, Z. Zhu, and I. Zharkov. Otov2: Automatic, generic, user-friendly. In International Conference Learning Representations, 2023.
Y.-N. Chuang, G. Wang, F. Yang, 7. Liu, X. Cai, M. Du, and X. Hu. Efficient xai techniques: (axonomic survey. arXiv preprint arXiv:2302.03225, 2023a.
Y.-N. Chuang, G. Wang, F. Yang, Q. Zhou, P. Tripathi, X. Cai, and X. Hu. Cortx: Contrastive framework for real-time explanation. arXiv preprint urXhv:2003.0299 2023b.
T. Dao, B. Chen, K. Liang, J. Yang, Z. Song, A. Rudra, and C. Ré. Pixelated butterfly: Simple and efficient sparse training for neural network models, 2022a.
T. Dao, B. Chen, N. S. Sohoni, A. Desai, M. Poli, J. Grogan, A. Liu, A. Rao, A. Rudra, and C. Re. Monarch: Expressive structured matrices for efficient and accurate training. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 4690-4721. PMLR, 17-23 Jul 2022b.
Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and I.. Fei-Fei. Imagenet: A large-scale hierarchical image datahase. In 2009 IEEE conference on computer vision and pattern recognition, pages 248 255. leee, 2009.
T. Dettmers, A. Pagnoni, A. Holtzman, and I.. Zettlemoyer. Qlora: Efficient finetuning of quantized 11ms. arXiv preprint arXiv:2305.14311 2023.
Frankle and M. Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019. URL httLhttp////ooeerrrrie net/ Forum?id==rJ1-bRcct
E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh. OPTQ: Accurate quantization for generative pre-trained transformers. In The Fleventh International Conference on Learning Representations, 2023. URL http:///ppaaeeeeeeeeeteerrrrraaa

---

S. Gao, F. Huang, W. Cai, and H. Huang. Network pruning via performance maximization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9270-9280, June 2021.
Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. arXiv 1412.6572, 12 2014.
Gui, Wang, H. Yang. C. Yu, Z. Wang, and J. Liu. Model compression with adversarial robustness: A unified optimization framework. Advances in Neural Information Processing Systems, 32, 2019.
Han, X. Liu, H. Mao, J. Pu, A. Pedram, M. A. Horowitz, and J. Dally. Eic: Efficient inference engine compressed deep neural network. ACM SIGARCH Computer Architecture News, 44(3): 243-254, 2016.
K. He, X. Zhang. S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 20/6, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778. IEEE Computer Society, 2016. doi: 10.1 00.100//VVP22206.900
Y. He and I.. Xiao. Structured pruning for deep convolutional neural networks: A survey, 2023.
Y. He, X. Zhang, and J. Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE international conference on computer vision, pages 1389-1397, 2017.
Y. He, G. Kang. X. Dong, Y. Fu, and Yang. Soft filter pruning for accelerating deep convolutional neural networks. In J. Lang, editor, Proceedings of the Twenty-Sevente International Joint Conference on Artificial Intelligence, IJCAI 2018, July 13-19, 2018. Stockholm, Sweden, pages 2234 2240. ijcai.org 2018. doi: 10.24963/jjcai.000 2033/3 8/309.
Y. He, P. Liu, Z. Wang, Z. Hu, and Y. Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 4340 4349. Computer Vision Foundation IEEE, 2019, doi: 10. (09/CVVR.2011.0004.
Jian, Z. Wang, Y. Wang. J. Dy, and loannidis. Pruning adversarially robust neural networks without adversarial examples. In 2022 IEEE International Conference Data Mining (ICDM), pages 993-998. IEEE, 2022.
Z. Jiang, X. Han, C. Fan, F. Yang, A. Mostafavi, and X. Hu. Generalized demographic parity for group fairness. In International Conference on Learning Representations, 2022.
D. Joo, E. Yi, S. Back, and J. Kim. Linearly replaceable filters for deep network channel pruning. In AAAI Conference on Artificial Intelligence, 2021.
R. E. Korf. A complete anytime algorithm for number partitioning. Artificial Intelligence, 106(2): 181-203, 1998. ISSN 0004-3702.
A. Krizhevsky. Learning multiple layers of features from tiny images. pages 32-33, 2009.
F. Lagunas, E. Charlaix, V. Sanh, and A. M. Rush. Block pruning for faster transformers. arXiv preprint arXiv:2109.0444 2021.
Y. LeCun, J. Denker, and S. Solla. Optimal brain damage. In D. Touret- zky, editor. Advances in Neural Information Processing Systems, volume 2. Morgan- Kaufmann, 1989 URL https::/procceediniggg www.tiiiiiinnnninnnon.eo..e wwwuurpp..ccpppppee httpppppr..ille//pprr///88/ ff11666882acc1100022266612282233 .df
H. Li, A. Kadav, I. Durdanovie, H. Samet, and H. P. Graf. Pruning filters for efficient convnets, In International Conference on Learning Representations, 2017. URL httppps:///peeeeviieeee net/ forum?id=r.n Corum?id=rJJFCTS13
Y. Li, S. Gu, K. Zhang, I.. Van Gool, and R. Timofte. Dhp: Differentiable meta pruning via hypernetwrkss In Proceedings of the European Conference Computer Vision, 2020.
12

---

Li, S. Lin, Liu, Q. Ye, M. Wang, F. Chao, F. Yang, J. Ma, Q. Tian, and R. Ji. Towards compact cnns via collaborative compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 6438 (438-6447, 2021.
Z. Li, T. Chen, L. Li, B. Li, and Z. Wang. Can pruning improve certified robustness of neural networks? arXiv preprint arXiv:2206.07311 2022.
J. Lin, J. Tang, H. Tang, S. Yang. X. Dang, and S. Han. Awq: Activation-aware weight quantization for Im compression and acceleration. arXiv, 2023.
M. Lin, R. Ji, Y. Wang, Y. Zhang, B. Zhang, Y. Tian, and L. Shao. Hrank:
Filter pruning using high- rank feature map. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA. June 13-19, 2020, pages 1526-1535. Computer Vision Foundation / IEEE, 2020. doi: 10. 1D.11099CVPR4220022220000.
S. Lin, R. Ji, C. Yan, B. Zhang, L. Cao, Q. Ye, Huang, and D. Doermann. Towards optimal structured cnn pruning via generative adversarial learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Z. Liu, M. Sun, T. Zhou, G. Huang, and T. Darrell. Rethinking the value of network pruning. In International Conference on Learning Representations, 2019.
Z. Liu, G. Wang, S. Zhong, Z. Xu, D. Zha, R. Tang, Z. Jiang, K. Zhou, V. Chaudhary, S. Xu, et al. Winner-take-all column row sampling for memory efficient adaptation of language model. arXiv preprint arXivv2230..55200 2023.
A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018. URL
https://openrevvie: http:/iwwweeeefotoeeeeeeee
Molchanov, S. Tyree, T. Karras, Aila, and Kautz. Pruning convolutional neural networks for resource efficient inference. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, Frunce, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
T. Narshana, C. Murti, and C. Bhattacharyya. Dfpc: Data flow driven pruning of coupled channels without data. In The Eleventh International Conference on Learning Representations, 2022.
U. Ozbulak, M. Pintor, A. V. Messem, and W. D. Neve. Evaluating adversarial attacks on imagenet: A reality check misclassification classes. In NewrIPS 2021 Workshop ImageNet: Past, Present, and Future, 2021. URL http::: wttprriiiwnenfffffoeeeeeee
J. Park, Y. Kim, J. Kim, J.-Y. Choi, and S. Lee, Dynamic structure pruning for compressing cnns. ArXiv, abs/2303.09736, 2023.
A. Renda, J. Frankle, and M. Carbin, Comparing rewinding and fine-tuning in neural network pruning. In International Conference on Learning Representations, 2020. URL htttpss://ppenreeiiiiii Sttefforum?id-Sggsjone
V. Sehwag, S. Wang, P. Mittal, and S. Jana. Hydra: Pruning adversarially robust neural networks. Advances in Neural Information Processing Systems, 33, 2020.
K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In Y. Bengio and Y. LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL sttp://arxiv.. org/ aat////bb///...///////////////// 1556
C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, J. Goodfellow, and R. Fergus. Intriguing properties of neural networks. In Y. Bengio and Y. LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. URL //tp:vv.///iv.a//. org/ abs/ /1///312..00 1312.6199
R. Tang, Y.NN Chuang, and X. Hu. Secure your model: A simple but effective key prompt protection mechanism for large language models.
13

---

M. R. Vemparala, N. Fasfous, A. Frickenstein, S. Sarkar. Q. Zhao, S. Kuhn, L. Frickenstein, A. Singh, C. Unger. N. S. Nagaraja. C. Wressnegger, and W. Stechele. Adversarial robust model compression using in-train pruning. 2021 IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops (CVPRW), pages 66-75, 2021.
Y. Wan, Zhao, A. Chadha, N. Peng, and K.-W. Chang. Are personalized stochastic parrots more dangerous? evaluating persona biases in dialogue systems, 2023.
G. Wang, Y.-N. Chuang, M. Du, F. Yang, Q. Zhou, P. Tripathi, X. Cai, and X. Hu. Accelerating shapley explanation via contributive cooperator selection. In International Conference on Machine Learning, pages 22576-22590. PMLR, 2022.
G. Wang, Z. Liu, Z. Jiang. N. Liu, N. Zou, and X. Hu. Division: Memory efficient training via dual activation precision. 2023a.
H. Wang, X. Wu, S. Ge, Z. C. Lipton, and E. P. Xing. Smooth kernels improve adversarial robust- ness and perceptually-alignee gradients, 2020a. /RLLhttp///opeeeeiiies wwt.forumfffiniiio id BJerUCEtPB
H. Wang, X. Wu, Z. Huang, and E. P. Xing. High-frequency component helps explain the generaliza- tion of convolutional neural networks. In 2020 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8681-8691, 2020b. doi: 10. 1411099/VPR44660.200000000
H. Wang, C. Qin, Y. Zhang, and Y. Fu. Neural pruning via growing regularization. In International Conference on Learning Representations, 2021. URL httpp::/ppparrrii...tttffrrrrrrp 0966_Is_ni
L. Wang, G. W. Ding, R. Huang, Y. Cao, and Y. C. Lui. Adversarial robustness of pruned neural networks. In Submission to International Conference on Learning Representations (Workshop), 2018. URL wwt.//vee.reeeeeeeeeeeeee
S. Wang, R. Veldhuis, C. Brune, and N. Strisciuglio. What do neural networks learn in image classification? frequency shortcut perspective, 2023b.
W. Wang, C. Fu, J. Guo, D. Cai, and X. He. COP: customized deep model compression via regularized correlation-based filter-level pruning. In S. Kraus, editor, Proceedings of the Twenty- Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16. 2019, pages 3785-3791 ijcai.org. 2019a. doi: 10.24963/ijcai.2019/222
W. Wang, S. Zhao, M. Chen, J. Hu, D. Cai, and H. Liu. DBP: discrimination based block-level pruning for deep model acceleration. CoRR, abs/1912.10178. 2019b.
J. Wu, Q Zhang, and G. Xu. Tiny imagenet challenge. Technical Report, 2017.
G. Xiao, J. Lin, and S. Han. Offsite-tuning: Transfer learning without full model. arXiv, 2023.
C. Yang, A. Buluç, and J. D. Owens. Design principles for sparse matrix multiplication on the gpu. In European Conference on Parallel Processing, pages 672 687. Springer, 2018
S. Ye, X. Lin, Xu, S. Liu, Cheng. J.-H. Lambrechts, H. Zhang, A. Zhou, Ma, and Wang. Adversarial robustness vs. model compression, or both? pages 111-120, 10 2019. doi: 10.1109/ ICCV:2019,00000.
R. Yu, A. Li, C. Chen, Lai, I. Morariu, X. Han, M. Gao, C. Lin, and L. S. Davis. NISP: pruning networks using neuron importance score propagation. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 9194 9203. Computer Vision Foundation IEEE Computer Society, 2018. doi: 10.1109/CVPR.2018 8.00958.
D. Zha, Z. P. Bhat, K.-H. Lai, F. Yang, Z. Jiang, S. Zhong, and X. Hu. Data-centric artificial intelligence: survey, 2023.
G. Zhang, Xu, Li, and A. X. Guo. Group-based network pruning via nonlinear relationship between convolution filters. Applied Intelligence, Jan. 2022a. doi: 10. 10.10077:400499-011--00000
14

---

S. Zhang, Y. Liu, Y. Sun, and N. Shah. Graph-less neural networks: Teaching old MLPs new tricks via distillation. In International Conference on Learning Representations, 2022b. URL https://openreevix. http::///oparreeeseeeee/oooouooo
Q Zhao and C. Wressnegger. Holistic adversarially robust pruning. In Proc. of the International Conference on Learning Representations (ICLR), May 2023.
S. Zhong, G. Zhang, N. Huang, and S. Xu. Revisit kernel pruning with lottery regulated grouped convolutions. In International Conference Learning Representations, 2022. URL https
15

---

A Limitation and Broader Impact
Note although our investigation revealed serious performance issues under adversarial tasks in terms of structured pruning methods, where our proposed method provides sensible mitigation; our findings are still limited to the benign and artificially perturbed input. Though we expect grouped kernel pruning- muaiing-based methods to deliver good performance under other reasonable evaluating metrics due to the improved pruning freedom, we will leave a more comprehensive investigation against other robustness metrics for future work. We also provide brief discussion of such related metrics in Appendix [.44
B Extended Related Work and Discussion
B.1 Structured v.s. Unstructured Pruning
Pruning methods can be roughly divided into structured pruning and unstructured pruning according to the pruning granularity. Specifically, unstructured pruning often means we prune each weight independently. In contrast, structured pruning bundles weights into groups, then prune the whole group instead of the individual weight (e.g., block-wise Lagunas al. [202]] channel-wise He et al. 2017]. group-wise pruning Zhong et al. 20222).
Unstructured pruning can maintain the model performance better with the same number of parametens. However, unstructured pruned models yield marginal wall-clock time efficiency or even slower than the unpruned model at the low sparsity regime. This is because unstructured pruned matrices need to be stored in sparse matrix format, as the zeros are randomly distributed in these matrices Yang et al.) [2018]. Operations executed on sparse matrices (e.g., sparse matrix multiplication, sparse embedding table look-up) are notoriously inefficient on commodity hardware, e.g., GPUs and CPUs, due the limited data reuse and random memory access Yang et al. [2018]. Han et al. [2016).
In contrast, although structured pruning has less flexibility compared unstructured one, it is much more hardware/iibrar--fiendd since structurally pruned matrices can still be stored in the dense matrix format. Thus, operations executed on structured pruned matrices the same those in the unpruned model, which are highly optimized. Consequently, the compression provided by structured pruning can often translate into the real wall-clock time speedup upon proper implementation.
B.2 Unstructured Pruning for Adversarial Robustness
In Section introduced the few structured pruning methods which claim increased adversarial robustness, indicating the lack of presence of adversarially robust structured pruning methods. However, many unstructured pruning literature have explored the possibility of pruned models with improved adversarial robustness, such as [Shwwa et al. 22000. Zhao and Wressnegger [2023). Ye et al. [2099] Jian et al.) [2022), [L et al. (2022), , Gui al. [2019] Given is natural for scholars to adopt insights from unstructured pruning methods into structured context, we hereby provide this list of unstructured pruning arts for inspiration purposes.
B.3 Learning of High-Frequency Components and its Adversarial Implications
As consult adversarial- boossting--ooggggggoooooo kernel metrics and operations, we heavily rely on the findings from Wang et al. [2020b]. a work that discusses how learning components with different frequencies may affect the adversarial robustness of CNN, and how some kernel-level metrics like kernel smoothness influence such type of learning. We have elaborated more on this in the Section 3.1 above. Following work can also be referenced when it comes to how component frequency may affect (or be adopted to improve) adversarial robustness, efficiency, and general neural network learning Wang et al. 2020a 2023b al
B.4 More on ML efficiency and AI safety
The point of studying adversarially robust structured pruning methods is to progress ML efficiency while being aware of its safety. We note that structured pruning not the sole method that can achieve efficiency benefits, yet adversarial robustness is nowhere near comprehensive metric for Al
16

---

safety. Approaches such as quantization, network architecture search, knowledge distillation, lossy approximation, efficiency- aware architecture tweak, and transformation to hardware-friendee patterns also provide efficiency gains [Frantar et al. 2023 Lin et al., 2023 Wang et al. 2020a Zhang et al.
2022b Liu al. 2023 Xiao et al. [023 [ Dettmers al. 2023 Dao al. 2022b a yet aspects like explainability, domain generalization,
model/data security, and robustness
under a noisy environment are also considered important metrics for building reliable, safe, and trustworthy AI solutions [Wang et al., 2022 Wan al., [023 Chuang al. 2023a b Chang et al.
2023c a Jiang et al. 2022 Chang et al. 2023b Tang et al. Zha et al. 2023
C
Additional Details on Proposed Method
C.1 Naïve Mix-Up Attempts of Kernel Smoothness
As per Section 3.2 there are generally two stages in a GKP procedure: Filter Grouping and Grouped Kernel Pruning. The following experiments shall attempt mix-up kernel smoothness criteria into each of such stages, and we can therefore find out which stage "friendly" to such mix-up operations and how such operations should look like.
C.1.1 During GKP Filter Grouping (Stage 1)
In TMI-GKP [hhong et al. 2022 the filter grouping stage is driven by the tickets magnitude increase score, known as TMI-driven Clustering. We utilize it as a baseline to investigate whether our smoothness- filter grouping operation Smoothness Snaking (Section 3.2.1 and Figure 3 can maintain the baseline performance and provide improvements. Note, we denote TMI-GKP's grouped kernel pruning scheme (GK Pruning) as Greedy in the following Table
and
Table 5: Comparison different Filter Grouping methods in GKP. All compared models are pruned to identical MACs/Params for fairness.
Model
Filter Grouping
GK Pruning
Criterion
Benign
(a) FGSM,-00
(b) FGSM,-0.1
Unpruned
92.80
71.93
31.35
Best Benign
92.99
69.15
19.59
TMI-driven Clustering
Greedy
Best (a)
92.03
70.61
ResNet-32
Best
92.77
30.51
Best Benign
92.77
71.47
30.64
Smoothness Snaking
Greedy
Best (a)
Best (b)
92.65
70.82
30.65
Unpruned
93.24
75.15
39.64
Best Benign
93.95
75.18
42.18
TMI-driven Clustering
Greedy
Best (a)
93.37
75.88
42,55
ResNet-56
Best (b)
93.66
75.74
44.09
Best Benign
93.62
75.68
41.21
Smoothness Snaking
Greedy
Best (a)
93.40
76.05
43.03
Best (b)
93.44
75.69
44.62
can be observed that though Smoothness Snaking may yield a slightly lower benign performance than its TMI-driven baseline, may improve the adversarial robustness when used with the same grouped kernel pruning procedure. We would also note Smoothness Snaking is significantly faster (up 1,600x) than TMI-driven clustering due the absence of dimensionality reduction and clustering procedure (see Table & for details).
C.1.2 During GKP Pruning Strategies Obtaining/Pruning (Stage 2)
Following the section above, here we investigate the performance of different Grouped Kernel Pruning methods. Our proposed method is denoted as Smooth Beam Greedy (see Figure details), TMI-GKP's grouped kernel pruning method is denoted as Greedy as above, and Least Smooth represents vanilla adaptation of smoothness-drive pruning, where the grouped kernels that are least smooth pruned.
17

---

Table 6: Comparison of different Grouped Kernel Pruning methods in GKP, All compared models are pruned to identical MACs/Params for fairness.
Model
Filter Grouping
GK Pruning
Criterion
Benign
(a)
FGSMc-0.01
(b) FGSM:-o.1
Unpruned
92.80
71.93
31.35
Best Benign
92.77
71.47
30.64
Smoothness Snaking
Greedy
Best (a)
Best
92.65
70.82
30.65
ResNet-32
Best Benign
90.09
63.91
17.22
Smoothness Snaking
Least Smooth
Best (a)
88.90
70.41
15.33
Best (b)
48.37
47.80
28.55
Best Benign
92.97
70.57
29.31
Smoothness Snaking
Smooth Beam Greedy
Best (a)
92.88
71.52
Best (b)
92.86
70.79
31,32
Unpruned
93.24
75.15
39.64
Best Benign
93.62
75.68
41.21
Smoothness Snaking
Greedy
Best (a)
93
76.05
43.03
Best (b)
75.69
44.62
ResNet-56
Best Berign
Smoothness Snaking
Least Smooth
Best (a)
90.01
74.63
14.10
Best (h)
90.59
73.60
26.40
Best Benign Best (a)
94.08
75.89
42.60
Smoothness Snaking
Smooth Bear Greedy
93,83
76.40
45.17
Best (b)
we may tell that Smooth Beam Greedy may significantly improve the adversarial robustness of the pruned networks, yet, it also provides remedies the decrease in benign performance due to the smoothness snaking operation. may also worth noting that the Least Smooth operation extremely detrimental to almost all tracking metrics, suggesting vanilla mix-up is inappropriate.
From Table 6
C.1.3 Bonus Investigation: Is Smoothnes--wwre Filter Pruning Possible?
One main purpose of our paper is to endorse the potential of GKP under adversarial tasks, after [Zhong et al. 2022] Zhang al. 2022a Park al., 2023] showcased the power of GKP in benign context, thus "one less reason for filter pruning." But to make such claim proper, we will need investigate whether it possible do the same smoothness-aware mix-up with filter pruning procedure.
SFP (He al. 2018 the baseline, which is considered one of the strongest filter pruning methods on CIFAR- 10 Krizhevsky) 22099. Then, we try to apply the same ranked- based kemel-smoothness mix-up algorithm in Equation 2 to find such strong filter pruning baseline can withstand the same mix-up under filter level.
Here in Table 7
Table 7: Filter pruning method SFP He et al. 2018 applied with the same mix-up algorithm in Equation Note "CSB" represents "cost smoothness balancer". which is also in Equation 2 so a higher CSB means more biased towards the distance-basee Cost metrics. All compared models are pruned to identical MACs/Params for faireess.
PGD *
8/255 Easp Barp=2/20 2/255
Model
Method
Benign
FGSM,=0.0t
FGSM,
jierm3
SFP
91.94
69.25
30.34
30.18
ResNet-32
CSB 0.5
83.49
40.86
21.12
24.28
CSB 0.75
91.03
65.94
22.92
25.40
SFP
93.15
75.63
43.83
44.10
ResNet-56
CSB 0.5
81.83
60.46
19.59
28.45
CSB 0.75
92.26
72.11
37.42
36.64
CSB 0.9
93.09
74.78
40.64
41.41
18

---

It can be observed that SFP under such mix-up is completely unusable due to the degradation of benign performance; which suggests the same mix-up strategy, though effective on GKP, is not transferable to filter pruning.
C.2
Speed-Up Analysis
C.2.1
Pruning Procedure Speed Comparison: TMI-Driven Clustering v.s. Smoothness Snaking
Table 8: Wall-clock runtime comparison between SR-GKP (Ours) and TMI-GKP Zhong al.
2022
Method
ResNet-32 Group
ResNet-32 Total
ResNet-56 Group
ResNet-56 Total
ResNer- 10 Group
ResNet- 100 Total
TMI-GKP
47m
20m I0s
2h 32m 12s
2h 36m 22s
4h 53m 33s
5h 30m 18s
SR-GKP
3s
1lm 36s
20m
Ib 10m 166
In Table g we showcased the significant runtime advantage of SR-GKP to TMI-GKP due to the absence of clustering and dimensionality reduction procedures.
C.2.2 Inference Speed Discussion: Standard Convolution v.s. Grouped Convolution
In the realm of structured pruning, there are two types of popular implementations in practice, which may result in different readings efficiency metrics, such inference cost reduction (FLOPs/MACs), inference speed-up, and model size compression (number of parameters):
Masking (soft pruning) To deploy a structured binary mask upon the original unpruned model, clear the gradients of some (or all) pruned components before the weight update step during fine-tuning, therefore theoretically "structural pruning" the model. In this implementation, the pruned model will not reduce in dimension; thus, no acceleration and compression benefits can be directly observed.
This is a popular implementation - as surveyed in Table TO MerooMMaske?"" column because some iterative pruning methods would like the pruned/zeroed components to be reactive during fine-tuning, then pruning another set of components instead. If the pruning granularity structured (e.g., filter-lvel), the fine-tuned model can be converted to a hard pruned model by the end of the update as defined below.
Component removal (hard pruning) To remove the pruned components entirely, where the pruned model will reduce in size, providing immediate compression benefits.
Our method, SR-GKP, is implemented as hard pruning. This means that if you inspect the weights of GKP-pruned network, they in regular shapes, and there is no zeroed weight be found a.k.a. "densely structured." Which may lead to a direct reduction in FLOPs/MACs and model size.
In terms of inference acceleration, previous grouped convolution implementations cannot provide speed-up benefits against a standard convolution (even with much fewer MACs/Params) because the standard convolution operator has been extensively optimized, while the grouped one hasn't 't (e.g., pytorch issue #10229 and #18631). But after torch 2.0, this is not the case at large. For simplicity, here compare the forward wall-clock between standard Conv2d of (C_ iC_in_ 512, C_out = 512, kernel_size = (3, 3)) with the exact same Conv2d but with groups = (meaning half of its kemelss structurally removed).
Table 9: Inference wall-clock comparison between standard convolution operator and a GKP convolution operator (w/ groups-2 and pruning rate = 50%). Input size set as (64, 3, 224, 224).
Operator
Forward
Macs
Params
Unpruned Standard Conv
129.56 ms 72.96 ms (56.31%)
550528 275264 (50%)
359296 179648 (50%)
GKP-prunee Grouped Conv
19

---

As showcased in Table 9 an organic inference time speed-up of grouped convolution over standard convolution can be observed.
(Note, we emphasized at large above because we indeed can find shapes that are slower with grouped convolution in torch 2.0. We believe this is very much framework optimization issue that is beyond the scope of our paper and shall be on the road map of the torch community, as well as other more inference-specifie frameworks, granted their already observed improvements.)
C.3
SR-GKP Procedure
C.3.1
Simplifying the Cost Formula from TMI-GKP
For better readability, we hereby follow the notation of TMI-GKP [Zhong et al. [022] (Equation 4), where assume V represents of kept grouped kernels provided by pruning strategy, where g the convolutional layer in question. The quality V*, under our design, is deemed by:
Score grouped kernel pruning
(V*, g'')'
Σ Eu.
w(su: - 3( Σ
w(p,, 8;)).
(3)
MI=IEV
Where w(S sy) represents the Euclidean distance between grouped kernels Sy and Sy, but s; represents the kept grouped kernel that has the least w(p., s;) to a pruned grouped kernel Thus, the former of this equation calculates the inner distance sum of grouped kernels within strategy V*, and the latter term represents the outer distance between pruned grouped kernel and its closest kept grouped kernel. Intuitively, we would like the former ideal V to have great diversity. Following the same idea, we would like to have the latter term small, as we want the kept kernels cover the representation power of certain pruned kernel. By using -3 to connect two term, we have V = arg max (Score(V* ,gf)) for all V*s obtained in Stage 2
be large, as we would prefer our
4
(Figure 4)
Though similar, we differ from TMI-GKP [hhog al.
2022 (Equation 4) in two spots: first, set
3 3- (Y') the two terms automatically. Secondly, for the latter term of Equation 3 we only match one pruned kernel Ps with one kept kernel 8;, instead of several of them. We made these modifications for the main purpose of removing hyperparameters. We kept the grouped kernel strategy selection stage similar TMI-GKP, but only expanded its search scope using the Smooth Beam Greedy search (Figure because prior investigation suggests this stage is sensitive to smoothness mix-up operations.
//Pum- PPmum where Paum represent the number of pruned grouped kernels in layer gf, balancing
20

---

D Extended Experiments and Results
D.1 Preliminary
D.1.1 Details of Experiment Setups
For all experiments on VGG- 16 and around 300 epochs meeetuning/retrannnng budget. Experiments conducted on ResNet-50/101 are budgeted with 100 epochs. We allow compared methods to utilize their own training schedule and vanilla SGD optimizer setup.
we aim to provide all pruning methods an
SR-GKP utilizes an initial 1r of 0.01, with a step size of 100 or 30, depending on if it is on the 300 epoch 100 epoch schedule. BasicBlock ResNets with VGG use a weight decay of 5e-4 but BottleNeck ResNets use a weight decay of 1e-4. SR-GKP strictly employs a batch size of 64 for all CIFAR-10 experiments, 128 for Tmag-em experiments, and 256 for ImageNet-Il Ik experiments.
D.1.2 Details of Evaluation Criteria
We manually ensure all pruned models have a similar MACs/Params reduction from the identical baseline models. Then, following Li et al. [2017], we report all model checkpoints - yielded during the pruning procedure that may reach the target MACs/Params reduction. For performance (aceuracy) evaluation, we test each model against FGSM [Goodfellow et al. 2014 and PGD [Madry
et al. 2018 under various settings and intensities.
D.1.3 Details of Compared Methods
We provide method overview in Table 10 provide our readers with a more comprehensive understanding of such methods.
Table 10: Details Compared Methods. Note "C/F/GK" under "Type" represents Chan- nel/Filter/rrouped Kernel Pruning. Whether method requires "Special Setup" is determined by whether it follows the most vanilla train prune fine-tune procedure. "Fully Pruned Epoch" reflects if given 300 fine-tune/retrain budget, what would be the first epoch that meets the target MACs/Params reduction? (a.k.a. checkpoints after this epoch are reportable as "superscore" readings defined in Section 4.2) MaskedM reflects whether the pruning method can easily yield compressed model without masking (a. hard pruning).
Pype
Special Setup'
Zem-Maaked
Fully Epoch
Li
2021
Yirquires Caypenet) N (GAN)
DHT
(Emutisccutt))
FPGN
benutive
Ieerutive
close
2014
One
No training)
HARP
2023
JCLR
One
HRRnik
[Ehrersf: 2006)
CVPR
F
Berulive
N
Y
325 327
LINomi
ICLR
F
One-sbot
(dynamic pusting rate)
N
LKF Joo
AAAI
One sbot
Y (requires data, addinz 1xl, dark knowledge)
N
NPP51 (uo 3r-l--01]]
CVPR
One shot
Y (Copemet)
N
OTOx2 [CTer 20 Puni)
ICLR
F
Iterative (fro scrach)
N
300
CAP-AD66ST1122 117 2019
JOCV
F
leentiv
Y (adv, training)
151
SFY [[le el 2011
DCAI
F
Benniie
Y (sefi pruning)
300
TMI-DK Pashrrra 2012
ICIR
GK
One-shot
SR. CKP (0u5)
NeuiPS
GK
One sbot
Further, following the background introduction of pruning implementation in Appendix C.2.2 hereby justify why a post-train one-shot hard pruning method - such as SR-GKP may "require no extra (in fact, often less) cost than standard pruning procedure."
Under the classic train prune fine-tune/retrain paradigm (where our method lives), the training cost determined at two stages: training the unpruned baseline model and fine-tuning the pruned model.
SR-GKP carries no additional training cost as it post- pruning method, it won't affect the training cost of the first stage. As for fine-tuning, the SR-GKP one-shot pruned model is already compressed and will go through vanilla SGD update with no extra operation; we also specifically ensured that all compared methods are using an identical (or similar to the best of the method's ability) epoch budget for fine-tuning as stipulated Section 4.2 thus, requires no additional (in fact, often less) cost the fine-tuning stage well.
21

---

For illustration convenience, here we walk through two exemplary comparisons:
Compared to TMI-GKP [Zhong al. 2022 the founding method of Grouped Kernll Pruning- a SR-GKP pruned model shares the exact same format with a TMI-GKP pruned model. Both pruned models will go through identical fine-tuning procedure. In this case, SR-GKP yields additional trining/iine-tuiigg cost. Additionally, we' d like to direct your attention to Table [ which indicates SR-GKP has much faster execution time than TMI-GKP in terms of pruning procedure, thus being a more "executionally efficient" method overall.
Compared to SFP [He al. 2018 popular filter pruning method that exhibits strong adversarial performance - SFP produces a zero-masked model while SR-GKP's pruned model hard pruned. Such zeroed weights will reactivate during its eetrainingffin-ttninng which filter to zero out between epochs. In this case, the one-shot pruned SR-GKP induces signifi- cantly less cost it does not require weight updates the full model, but only compressed one; and b) does not perform extra pruning operations during the tme-tuning/retraining stage.
where the algorithm decides
22

---

D.2 Additional Experiments
D.2.1
ResNet-32/56/100 on CIFAR-10 with pruning rate 43.75% (in addition to Table 2)
Please refer Table IT with Figure Table 12 with Figure 7 and Table 13 with Figure 8 for details.
We also provide Table in ranked chart format for an easy digest of the three aforementioned tables.
Slangn
FORM e 0.01
POSM m.
MGD 62b5, Casp 2015, machte
100
KIM
50
Accuracy
311
UH+
RPOM
UthomiA
LtMkrre B
LIF
O1O/C
O10xz PP) RAP-ADMYY
8FP
TM 6KP 84 84-04P (Dant
Figure 6: "Superscore" visualization of ResNet-32 on CIFAR- 10 with pruning rate = 43.75%
Table 11: Full experiments of ResNet-32 on CIFAR-10 with pruning rate s 43.75%. All pruning methods are performed on the same baseline model. "Best (a)" represents the performance of a model checkpoint that meets the showcased MACs/Params reduction and performs best against criterion (a).
Resign
RGSM.....
PGSMM
(e)
9/255 2//022/55
(M)
Unpruned
92.80
71.93
31.35
35,40
69.479
0.464
Best Resign
03.01
70.06
25.25
30.40
OC Fi at 2021
Best (a)
92.69
70.92
25,84
30.75
39.261
0.312
Best (b)
92.82
70.50
26.79
31.20
Best Besignn
92.26
67.25
21.35
26.12
DEPP 1
2020
Best (x)
91.88
68.14
26.64
26.80
40.09
0.283
91.80
27.23
29.61
Ho
(4)
(b)
Renign
24.03
LINorm A al. 2017
Best (a)
91.03
70.00
25.92
30.70
39.861
0.252
Best (b)
90.38
67.61
32.50
29.67
Best Resign
91.58
67.04
25. 24
28.91
NoNomm-B 8 Lietal 2017
Best (x)
91.03
71.59
28: 11
35.20
39.600
0.313
Best (b)
90,59
69.67
34,62
34.02
Best Besign
93.01
69.38
25.82
30.07
LRF ot 2021
Best (x)
92.63
70.22
25.
29.74
38 791
0.260
92.79
27.54
Renign
27.97
RAP-ADMM
2019
25.80
39.370
CTTv2 scrttch)
OTO+2 (post tmain) Chen et 2023
92.14
70.6
28.01
31.90
SHP He et 2013
91.94
69.25
30.34
30.18
40.375
0.206
Best Benign
92.99
69.15
19.59
28.09
TML TMP-GKP GKP Zhong al 2022
Best (a)
92.11
70.61
15.85
29.44
39.545
0.263
Best (b)
92.77
60.90
30.51
32.85
Rest Renign
92.97
70.57
29.31
32.01
SR GKP (Ours
(a)
92.88
71.52
30.39
33.36
39.545
0.263
23

---

POOM
8/50, 2155,,
100
Accuracy
786
P904
6eng"
Degg
Hook
olbe O10ca AMP 41P-480
Whee Fook)
Figure 7: "Superssorr"""suuazzziioo
visualization of ResNet-56 CIFAR-10 10 with pruning rate 43.75%
Table 12: Full experiments of ResNet-56 on CIFAR-10 10 with pruning rate x 43.75%. All pruning methods are performed on the same baseline model. "Best (a)" represents the performance of a model checkpoint that meets the showcased MACs/Params reduction and performs best against criterion (a).
Method
Criterion
PGIM
PCT)) Jowi
MACs
Unpruned
Besign
29.25
(5)
Best (b)
93.70
75.56
29.89
38.90
Best Resign
02.42
69:50
20.93
32.56
DHP
ct al.
3020
Best (a)
92.34
70.66
30.17
33.64
73.289
0.480
Best (b)
92.07
31.54
70.42
32.83
Best Besignn
93.60
75.31
43.20
43.55
FPGM
He
2019
Best (x)
93.37
76.28
44.95
44.66
71,661
0.482
GAL
70.38
47.32
Benign (b)
79.237
0.584
(a)
Renign
LINoem A al. 2017
Best (a)
91.65
74.30
43.59
42.74
67.995
0.487
Best (b)
91.34
71.94
48.19
42.10
Best Resign
92.62
72.97
41.30
41.79
No 8 Liet 2017
Best (x)
91.94
75.16
42.49
43.71
72.115
0.586
Best (b)
91.70
74.40
45.15
43.41
Best Besignn
93 93
73.47
25.59
34.86
LRF ot 2021
Best (x)
93.66
74,80
27.39
36.75
71.009
0.490
93.63
28.20
Resign
RAP-ADMM
27.84
71.661
CTTV2 scrttch)
79.780
OTOV2 (post tmain) Chen et 2023
93.02
74.29
40.45
41.58
66.196
0.554
SHP He et 2013
93.15
75.63
43.83
44.10
71.462
0.451
Best Benign
93.95
75.18
42.18
43:46
TML TMP-GKP GKP Zhong al 2022
Best (a)
93.37
75.88
42.55
43.74
71,855
0.482
Best (b)
93.66
75.74
44.00
44.51
Rest Renign
94.08
75.89
42.60
43.85
SR GKP (Ours
(a)
45.17
21.855
0.482

---

Besign FOSME-OO
FOBM 6
FOD 8035, SMO 2426 2/26 mat hor
100
80
Accuracy
Urpraed
oroc Cr
Figure 8: "Superscore" visualization of ResNet- 110 CIFAR-10 with pruning rate 43.75%
Table 13: Full experiments of ResNet- 10 on CIFAR-10 with pruning rate 43.75%. All pruning methods are performed on the same baseline model. "Best (a)" represents the performance a model checkpoint that meets the showcased MACs/Params reduction and performs best against criterion (a).
Method
FGSM,-s.t
PGD 5//00 1/30
Unprunal
Best Benign
94.31
75.12
27.31
38.24
OC
Li al. 2001
Best (a)
94.17
75.88
28.
39.78
144.144
1.046
Best (b)
94.25
75.52
29.00
39.64
Best Benign
92.53
67.98
25.08
26.83
DHP
al. 30211
Best (a)
92.21
68.86
25,51
27.39
101350
0.612
Best (b)
92.25
67.82
26.64
26.67
Best Benign
94,
79.32
52,16
52,46
JPGM
(a)
79.78
53.35
114.357
54.15
2019
2020
Benign
Besign
N.Noom-AA
(5)
39.71
Best (b)
92.21
75.26
45.48
45.49
Best Resign
22.96
75.26
41.19
44.56
LiNorm-B 8 Lietal 2017
Best (a)
92. 40
76.10
41.36
47.05
144.909
1.064
Best (b)
91.71
73.58
41.92
42.69
Best Besign
94.49
76.60
29.50
42.15
LRF
Too al 2021
Best (x)
94.22
76.93
29.27
42.59
144.405
0.997
30.67
32.84
146
(b)
OTOv3 sentich)
Chen
23.94
OTOv2 (poo imii))
Chen
2033
77.25
47.95
SHP
He ct 2013
94.44
78.75
53.62
52.57
144:274
0.976
Best Resign
94.90
77.50
47.15
49.00
TMI Zhong al
2022
Best (x)
94.63
78.02
45.95
48.81
144.551
0.976
Best (b)
94.90
77.50
47.15
49.00
Best Renign
95.00
78.01
46.53
48.36
SR GKP (Ous)
(a)
47.49
144.551
0.966
94.60
73.92
49.09
25

---

D.2.2 ResNet-32/56/110 on CIFAR-10 with pruning rate = 62.5%)
Please refer Table 15
16 and 17
We also provide Table 14 in ranked chart format for easy
digest of the three aforementioned tables.
Table 14: Methods ranked against each other on each model with pruning rate R 62.5%, lower is better. "ResNet-XX Mean Rank" means method's average ranks across four metrics ResNet-XX: e.g. SR-GKP is ranked #1 1/#3/#3 for its best performance across benign/FGSMM 0.01/FGSM 0.I/PGD metries on ResNet- 10 against other methods, so it'd have ResNet-1 10 Mean Rank of (1+1+343)44 = #2. Methods with incomplete presence across the three models are excluded, This table provided to facilitate the digestion of Table 13 16
and 17 which consist raw results.
ResNet-32
Rank
ResNet-
Rank
CC al 2021
#3.5
#5.75
#3.42
DHI L1
20.00
#3.75
#7.5
#7.75
#6.33
FPGM
LINomi-A
La el 2017
#5.5
#3.75
#5.75
#5.00
NPPM Gao
2021
#5
#4
#5.5
#4.83
[Heeral: 2018
#5.17
SR-GKI((Uus)
#2.75 (2ud-best)
#2.25
#2
#2.33
Table 15: Full experiments of ResNet-32 on CIFAR-10 with pruning rate 62.5%. All pruning are performed on the same baseline model. "Best (a)" represents the performance of a model checkpoint that meets the showcased MACs/Params reduction and performs best against criterion (a).
Method
Criterion
Benign
(a) FGSM,-0.0
(h) PGSM,-o..
(c) PGTY 2/250 (ec_iem Jum-
MACs (M)
Params (M)
Unpouned
Best Benign
92
70.71
15:45
30.87
Li al. 2021
Best (a)
91.83
69.99
28.84
30.91
26,904
0,210
92.01
Best Renign
91.73
66.8
28.00
26.71
DHP Liet
2020
Best (a)
91
67.71
26.83
26.89
66.84
26.39
Best Benign
91.32
65.41
20.91
24.97
FPGM
He et al. 2019
Best (a)
90.47
67.77
15.95
26.61
Best (b)
91.04
56.51
24.47
25.68
Benign
89
20.44
27.29
LINorm-A
Li et al. 2017
Best (a)
89.52
67.65
18.07
27.93
26.511
0.163
Best (b)
89.23
66.75
23.21
27.88
Benign
L1Norm-8 8 Lic all 2017
Best (n)
89
67.14
17.75
26.91
26. 157
0.146
Best (b)
89.42
66.66
21.63
27.55
Benign
LRF Joe al. 2021
Best (a)
92.46
70.56
20.91
28.90
29.915
0.196
Best (b)
92.43
69.50
25.40
29.12
Renign
NPPM Goo et al.
Best (a)
91.79
67.56
22.39
25.91
26.998
0.198
Best (b)
91.67
67.16
23,97
25.60
SFP He al. 2018
90.28
66.71
20.35
25.47
Benign
92.21
21.91
25.98
SR-GKP (Duns)
Best (n)
91.52
69.33
14.83
27.94
26.717
0.176
Best (b)
92.04
66.37
23,55
25.30
26

---

Table 16: Full experiments of ResNet-56 on CIFAR-10 10 with pruning rate N 62.5%. All pruning methods are performed on the same baseline model. "Best (a)" represents the performance of a model checkpoint that meets the showcased MACs/Params reduction and performs best against criterion (a).
Benign
(a) PGSM,-0.33
(h) FGSM
(M)
(M)
Ungruned
93.24
75.15
39.64
42,58
126.561
0.853
93.57
73.63
25.30
35.40
Last
Best (a)
74.29
24.96
35.54
48.692
0.421
Best (b)
93.37
62.56
26.12
35.25
Best Benign
91 .666
70.66
29.75
31.40
(a)
Best (b)
91.48
70.41
30.15
31.27
Best Benign
92 92.64
71.80
35.17
35.17
FPGM
Best (b)
92.62
71.86
35.99
35.61
HRank
Lin etal
2020
90.63
69.49
17.14
29.51
Benign
LINorm-A
et
2017
Best (a)
91.12
71.76
37.01
36,97
47.562
0.355
Best (b)
91.47
70.10
39.83
37.03
91
32,80
33.61
LINomm-B
2017
(a)
71.24
33.22
35.00
47.794
0.322
Best (b)
91.00 91.07
69:25
26.19
33.66
Best Benign
93.07
73.23
29.66
35.24
(a)
Best (b)
93.02
72.91
30.39
34.03
SEP
He
2018
92.24
72.21
33,65
35.39
Benign
SR-GKP (Durs)
Hest (n)
02.69
73.54
35.42
38 39
48.409
0.323
Best (b)
92.69
73.38
36.53
38.95
Table 17: Full experiments of ResNet-110 10 on CIFAR-10 with pruning rate ~ 62.5%. All pruning methods are performed on the same baseline model. "Best (a)" represents the performance of a model checkpoint that meets the showcased MACs/Params reduction and performs best against criterion (a).
Benign (4) PGISM_...a
POSM,
(c) PNGD)-6/200, 22p
MAC's (M)
Params (M)
Unpeuned
94.26
78.55
48.54
50.85
254.995
1.728
Rest 94.29
73.77
24.50
36.32
Best (b)
94.03
73.70
26.03
36.44
Best Benign
92,73
71,39
23.19
35.51
DHP ea 20/20)
92.35
72.41
23.70
36.22
Best
92.50
71.27
25.13
34.86
Best Benign
91.11
76.11
17.62
47.51
FPGM He al. 2019
Best
94,00
76.52
48.39
47.75
HRank Lin et al. 2020
91.94
70.13
15.04
30.19
LINom-A A Liet at. 2017
Rost Benign
Best &
91.51
92.50
74.99
73.06
42.62
40.19
43.45
41.77
97.952
0.622
Best Benign
94.04
74.81
41.83
41.28
LINom-B Li 2017
Best (x)
93.79
75.55
42.36
41.99
101.256
0.484
Best
93.86
74,96
43.13
41.99
Benign
LRF Ico et 2021
Best (a)
93.88
76.96
33.44
42.52
94.479
0.638
Best (b)
93.98
76.21
34.55
41.77
Benign
93.93
93.76
31.12
99.913
Best
93.79
75.02
32.62
39.26
SFP He ata
2018
92.98
76.08
$2.15
47.26
Benign
SR- GKP (Ours)
Best (a)
94.17
76.52
43.98
46.01
97.217
0.654
Best (b)
91,27
76.17
44.56
45.74
D.2.3 VGG-16 on CIFAR-10
Please refer to Table 18 with Figure for details.
27

---

FGSM
FGD 8/256,
100
69-73
53.30
60-16
83.05
80
Accuracy
00.00
60
$5.30
40
41.01
20
Urpruned
00
HRark
Lthoms
TM OXP
BR OAP 10un
Figure 9: "SSuperscore" visualization of VGG-16 CIFAR-10.
Table 18: Full experiments of VGG-16 CIFAR-IO. All pruning methods are performed on the same baseline model. "Best (a)" represents the performance model checkpoint that meets the showcased MACs/Params reduction and performs best against criterion (a).
Method
Criterion
Benign
(a) FGSM,-oot
(b) FGSM
PGD #8/255
Fay
MAC' (M)
Parons (M)
Unpruned
93.94
83.73
50.51
64.99
313,433
14.728
Best Benign
94,14
83.09
47,83
62.81
178.107
Best (b)
93.97
83.14
48.97
62.85
GAL Lin al. 2019
91.29
81.69
61.96
67.29
203.224
7.732
93.57
81.45
56 26.28
(a)
212.264
Best (b)
93.54
81.52
34.76
56.34
Best Benign
92.88
80.85
37.21
56.43
LiNorm
Li al. 2017
Best (a)
92.41
81.53
31,10
56.17
179.561
9.135
(h)
TMI-GKP (Zhnng et
2022
Best Benign
94.07
83.48
55,30
66.06
178.184
8.293
Best Henign
93.95
83.49
56.64
65.57
SR-GKP (Oun))
tal
93.77
83.85
$7.48
66.51
178.184
8.293
(b)
D.2.4 ResNet-56/101 on Imag-
Please refer to Table for details.
Table 19:
Full experiments of ResNet-56/101 Tiny-Imagenet. All pruning methods are performed on same baseline model. "Best (a)" represents the performance model checkpoint that meets the showcased MACs/Params reduction and performs best against criterion (a).
Medil
Mahood
Oriezioo
Berign
RCSMM
(b) FGSM
IoPGSMM
PCDS
(AC+(M)
Poramns
Unpored
55.90
$1.55
78.79
8.011
15.00
505 751
D1565
ResNee-16
Bost Berign
$4.14
7,71
17.18
115c7
Besian
$1.45
50.07
7,62
315,600
0.547
Ungrused
65.51
65.12
10.13
$7.60
10061
42,900
Boi Benign
67.21
65.52
46.00
8.95
37.34
ResNee- 101
SR GKP (Ous))
Bost
Best
65.00
64.11
64.91
17.00
10.04
25 14
$721.15
24.226
Best
65.69
64.91
Best Besign
66.09
64.05
5.00
5721.133
24.256
ResNo-101
TMI-KK Zhrng a 2002
Boil (8) 63.90
Beil
63.53
62.23
61.34
33.71
33.47
1077
16.45
30.61
30.47
5721
$721.115
24.226
24.226
Best 60)
65.90
61.05
10.77
3067
5721 113
24.256
D.2.5
Comparison with Channel Pruned HARP
HARP by Zhao and Wressnegger (023 is one of few adversarially robust structured pruning methods available for comparison, as it has a channel pruning version offered in its appendix (HARP-CP). However, HARP utilizes several different training schedules than other methods, and we therefore cannot compare it with other methods using unified baseline (as in Table To fulfill the comparison, we opt align SR- GKP with HARP under different potential training schedule settings offered in HARP literature or experiment setup. Please refer to Table 20 for details.
28

---

Table 20: Comparison with Channel-Pruned HARP (HARP-CP) by Zhao and Wressnegger 2023 on VGG-16 with CIFAR-10 dataset. (AT Adversarial Training; NT Natural Trainingg AS epoch
with best adversarial accuracy is saved; NS epoch with best benign accuracy is saved; LS = last
epoch is saved: SS "Superscore" readings according to Section 4.2 and Table I0
Method
Baseline Budget
Pruning Budget
Fine-tuning Budget
Best Benign
FGSM.=0.0
F0
PGD
HARP-CP
(AS)
NT. 300 (NS)
3.30
HARP-CP
NT. 300 (NS)
AT, 20 (LS))
NT. 300 (NS)
87.04
23.49
10.71
0.05
HARP-CP
(AS)
AT, (LS)
AT. 100 (AS)
72.06
64.08
17.96
50.93
SR-GKP (Ours)
NT, 300, (I.S)
0
NT, 300 (SS)
93.95
83.85
59.03
66.62
SR-GKP (Ours)
NT, 300, (LS)
0
NT. 100 (SS)
93.48
82.62
52.35
62.18
We believe it is safe to conclude that SR-GKP showcased clear dominance over the HARP-CP.
Although one network-datass combination is not comprehensive evaluation, Figure 13 of HARP suggests HARP-CA CC extremely close RAP-ADMM (Ye et al. 2019)
method we already compared in terms of both benign and adversarial acc; and from Figure 6 7 as well Table 3 we may conclude that RAP-ADMM, performs vastly below SR-GKP on various metrics. We find interesting that HARP and RAP-ADMI, worse than SR-GKP on most adversarial tasks, though SR-GKP only sees benign inputs; indicating the effectiveness of SR-GKP. We suspect this is mainly due to the fact HARP and RAP-ADM are not structured pruning-focue focused methods.
two methods utilizing adversarial training still perform
D.3 Ablation Studies
Please refer to Table Table and 22ble for ablation studies on hyperparameter a Equation 2
We denote it CSB as it is essence "cost-smoothness balancer." It can be observed that relatively high CSB - meaning giving more bias to the distance-based cost metrics - may yield better performance.
Table 21: Ablation study of cost-smoothness balancer "CSB" (a in Equation 2 on ResNet-32 with
CIFAR-10.
Method
Criterion
Benign
(a) FGSMF-GMoR 0.01
(b) FGSM; 0.1
MAC' (M)
Params (M)
Unpruned
92.80
71.93
31.35
69.479
0.464
Best Benign
92.79
70.39
28.09
0.1 CSB
Best
92.59
71.50
29.03
39.545
0.263
Best (b)
92.47
70.94
29.69
Best Benign
93.01
70.45
27.91
0.25 CSB
Best (a)
39.545
0.263
92.70
71.32
30.46
Best (b)
Best Benign
93.07
68.74
28.07
0.5 CSB
Best (a)
92.54
70.63
21.42
39.545
0.263
Best (b)
92.78
70.29
30.47
Best Benign
92.93
70.55
30.02
0.75 CSB
Best (a)
39.545
0.263
92.70
71.09
30.89
Best (b)
Best Benign
92.97
70.57
29.31
0.9 CSB
Best (a)
92.88
71.52
30.39
39.545
0.263
Best (b)
92.86
70.79
31.32
29

---

Table 22: Ablation study of cost-smoothness balancer "CSB" (a in Equation on ResNet-56 with
CIFAR-10.
Method
Criterion
Benign
FGSM.
(b) FGSM,-0.
MACs (M)
Params (M)
Unpruned
93.24
75.15
39.64
126.561
0.853
Best Benign
93.72
75.17
39.90
0.1 CSB
Best (a)
71.855
0,482
93.38
75.79
42.53
Best (b)
Best Benign
93.83
75.68
40.40
0.25 CSB
Best (a)
93,70
76.49
42.35
71.855
0,482
Best (b)
93.57
75.72
43.92
Best Benign
93.76
75.61
41.20
0.5 CSB
Best (a)
93.41
76.69
42.58
71.855
0,482
Best (b)
93.45
75.97
42.69
Best Benign Best (a)
94.08
75.89
42.60
0.75/0.9 CSB
71.855
0,482
93.83
76.40
45.17
Best (b)
Table 23: Ablation study of cost-smoothness balancer "CSB" (a in Equation 2 on ResNet-110 with CIFAR-10.
Method
Criterion
Benign
(a) FGSM
(b) FGSM,=0.1
MACs (M)
Params (M)
Unprumed
94.26
78.56
48.83
254.995
1.728
Best Benign
94.60
78.44
44.20
0.1 CSB
Best (a)
94.47
79.62
46.03
144.551
0.976
Best (b)
94.35
79.41
47.5
Best Benign
94.50
77.86
47.07
0.25 CSB
Best (a)
94.35
78.65
46.04
144.551
0.976
Best (b)
94.40
78.52
48.10
Best Benign
95.00
78.01
46.53
0.5 CSB
Best (a)
94.69
79.14
47.49
144.551
0.976
Best (b)
94.60
78.92
49.09
Best Benign
94.49
77.97
45.93
0.75 CSB
Best (a)
94.26
78.91
47
144.551
0.976
Best (b)
94.32
78.44
48.89
Best Benign
94.54
78.21
47.16
0.9 CSB
Best (a)
94.29
78.81
46.71
144.551
0.976
Best (b)
94.31
78.25
48.55
30


Reviews:
[{"review_id": "F6DoJSAZ6V", "reviewer": "Reviewer_Tjyo", "summary": "The contribution of the paper is two fold. First, it presents a large scale benchmarking study of the adversarial robustness of various filter pruning methods. It finds that such methods often lead to a large degradation in the adversarial robustness of the pruned models, though they preserve benign accuracy. The paper then proposes an alternative form of pruning to solve this issue. The proposed method borrows ideas from grouped kernel pruning and kernel smoothness to prune out groups of kernels while making sure that the model does not overfit to adversarially vulnerable high frequency components of the image. This leads to better adversarial performance than baselines. ", "questions": "1. Can the authors provide information about the training cost", "limitations": "The limitations are sufficiently discussed.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The empirical evaluation of the paper is solid.\n2. The paper proposes a simple hypothesis of pruned models fitting to HFCs to explain their poor robustness, and unit-tests it well. \n3. The proposed method effectively combines two existing works, and provides solid gains. ", "weaknesses": "1. The writing of the paper can be improved. There are multiple typos, and the paper could benefit from formally defining terms and notations earlier on. A more thorough discussion of the GKP and kernel smoothing techniques would also help.\n2. The claim of no additional training cost over standard pruning methods is not verified. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer `Tiyo` for your appreciation of our work, especially in terms of the benchmark and ablation study parts. We invested a lot of time/manpower/compute into them, as we believe such results and to-be-open-sourced codebase may greatly help out the structural pruning community to efficiently identify problems and conduct fair comparison. Yet we hope our ablation studies may facilitate future development in the GKP realm.\n\n### **[W1 - Term clarification and typos]: Thanks!** \nWe will address those ambiguities/typos, as well as give the paper a detailed polish, should there be camera-ready version.\n\n### **[W2, Q1 - Show support for SR-GKP not requiring additional training cost]: Sure!**\n\n\nUnder the classic *train - prune - fine-tune/retrain* paradigm (where our method lives), the training cost is determined at two stages: training the unpruned baseline model and fine-tuning the pruned model. \n\nWe claim SR-GKP carries no additional training cost as it is a post-train pruning method, so it won't affect the training cost of the first stage. As for fine-tuning, the SR-GKP one-shot pruned model is already compressed and will go through a vanilla SGD update with no extra operation; we also specifically ensured that all compared methods are using an identical (or similar to the best of the method's ability) epoch budget for fine-tuning — thus, it requires no additional (in fact, often less) cost on the fine-tuning stage as well.\n\nFor illustration convenience, here we walk through two exemplary comparisons: \n\n**Compares to TMI-GKP [1]** — the founding method of Grouped Kernel Pruning — a SR-GKP pruned model shares the exact same format with a TMI-GKP pruned mode, then such pruned models will go through the identical fine-tuning procedure. **In this case, SR-GKP yields no additional training/fine-tuning cost.** Additionally, we'd like to direct your attention to Table 7 ([page 16](https://openreview.net/attachment?id=Pjky9XG8zP&name=supplementary_material#page=16)), which indicates SR-GKP has a much faster execution time than TMI-GKP in terms of pruning procedure, thus being a more \"executionally efficient\" method overall.\n\n**Compare to SFP [2]** — a popular filter pruning method exhibits strong adversarial performance — where its pruned model is zero-masked. Such zeroed weights will reactivate during its retraining/fine-tuning, where the algorithm decides which filter to zero out between epochs. **In this case, the one-shot pruned SR-GKP induces significantly less cost** as a) it does not require weight update to the full model, but only on a compressed one; and b) it does not perform extra pruning operation during the fine-tuning/retraining stage.\n\nWe hope our response clarifies the reviewer's questions and further demonstrates the effectiveness of our method, and we will add similar information to D.1.3 ([page 17](https://openreview.net/attachment?id=Pjky9XG8zP&name=supplementary_material#page=17)) to enrich the clarity of our method comparison. Given the borderline scoring of our work, we hope the reviewer may consider raising the score, especially on the presentation aspect.\n\n---\n\nRef.   \n[1] Zhong et al., Revisit Kernel Pruning with Lottery Regulated Grouped Convolutions. ICLR 2022   \n[2] Ye et al., Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks. IJCAI 2018"}, {"review_id": "En3Ac2vkCd", "reviewer": "Reviewer_sejQ", "summary": "This paper proposes a structured pruning method that introduces kernel smoothness into grouped kernel pruning to obtain adversarial robustness. Without additional overhead (compared to standard pruning methods), the method can obtain considerable accuracy and adversarial robustness simultaneously. The authors compare with popular structured pruning methods, and the results validates the effectiveness of the proposed method.", "questions": "- The author claims the method is one-shot. However, the one-shot pruning typically refers to the pruning method without the need of fine-tuning, such as OTO [2]. The proposed method seems still requiring fine-tuning afterwards.\n\n  [2] Only train once: a one-shot neural network training and pruning framework.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The motivation is reasonable: introducing kernel metrics related to adversarial robustness into the pruning process to obtain robustness.\n\n- The authors designed sufficient ablation experiments to validate that the proposed 'Smoothness Snaking' Filter Grouping strategy and 'Smooth Beam Greedy' pruning strategy are more effective than other strategies.", "weaknesses": "- The novelty is limited: kernel smoothness and grouped kernel pruning have been studied in some works. The core contribution of the paper seems to be a direct combination of two ideas. \n\n- Some baselines are omitted in the experiments, such as HARP, which can be extended to structured pruning for robustness.\n\n  [1] Holistic adversarially robust pruning. \n\n- Lack of performance comparison under varying pruning ratio.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your time, as well as for recognizing the motivation of our task and the solidness of our approach. To address your mentioned weaknesses:\n\n### **[W1 - Limited novelty]: Please refer to our [general comment](https://openreview.net/forum?id=Pjky9XG8zP&noteId=1EzOrtnZm3).** \n\n### **[W2 - Comparison with HARP]: Sure! SR-GKP still dominates.**\n\nGood suggestion. We missed it as HARP is presented as a weight pruning paper with a channel pruning extension. The only overlapped experiment between channel-pruned HARP (hereinafter \"HARP-CP\") and SR-GKP is VGG-16, and here's the results:\n\n| Method | Baseline Budget | Pruning Budget | Fine-tuning Budget | Best Benign | Best FGSM w/ $\\varepsilon = 0.01$ | Best FGSM w/ $\\varepsilon = 0.1$ | Best PGD |\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n| HARP-CP | NT, 300 epochs (AS) | AT, 20 epoch (LS) | NT, 300 epochs (NS) | 91.66 | 55.11 | 23.13 | 3.30 |\n| HARP-CP | NT, 300 epochs (NS) | AT, 20 epoch (LS) | NT, 300 epochs (NS) | 87.04 | 23.49 | 10.71 | 0.05 |\n| HARP-CP | NT, 300 epochs (AS) | AT, 20 epoch (LS) | AT, 100 epochs (AS) | 72.06 | 64.08 | 17.96 | 50.93 |\n| SR-GKP | NT, 300 epochs (LS) | 0 epoch | NT, 300 epochs (SS) | 93.95 | 83.85 | 59.03 | 66.62 |\n| SR-GKP | NT, 300 epochs (LS) | 0 epoch | NT, 100 epochs (SS) | 93.48 | 82.62 | 52.35 | 62.18 | \n\n(AT = Adversarial Training; NT = Natural Training; AS = epoch with best adv. acc is saved; NS = epoch with best benign acc is saved; LS = last epoch is saved; SS = Superscore according to [Table 8](https://openreview.net/attachment?id=Pjky9XG8zP&name=supplementary_material#page=18).)\n\n**We believe it is safe to conclude that SR-GKP showcased clear dominates over the HARP-CP.**\n\nAlthough one network-dataset combination is not a comprehensive evaluation, Figure 13 of HARP ([OpenReview link](https://openreview.net/pdf?id=sAJDi9lD06L#page=21)) suggests HARP-CP is extremely close to RAP-ADMM [1] — a method we already compared — in terms of both benign and adversarial acc; and from [Figure {6, 7, 8}](https://openreview.net/attachment?id=Pjky9XG8zP&name=supplementary_material#page=18) we may conclude that RAP-ADMM performs vastly below SR-GKP on various metrics. **We find it interesting as HARP and RAP-ADMM — two methods utilizing adversarial training — still perform worse than SR-GKP on most adversarial tasks, though SR-GKP only sees benign inputs; indicating the effectiveness of SR-GKP**. We suspect this is mainly due to HARP and RAP-ADMM are not structured pruning-focused methods.\n\nWe will also include a discussion about HARP in our related work session; again, we thank the reviewer for mentioning this work.\n\n### **[W3 - More pruning ratios]: Sure! Again, SR-GKP is going strong.**\n\nHere's $PR \\approx 62.5\\%$. Given our raw experiment results would be massive like [Table {9, 10, 11}](https://openreview.net/attachment?id=Pjky9XG8zP&name=supplementary_material#page=20), we plotted a couple graphs to facilitate your reading — please kindly direct your attention to the to-all-reviewer comment, as that's where we may attach a PDF ([redirection to general comment](https://openreview.net/attachment?id=1EzOrtnZm3&name=pdf)).\n\nWe believe it would be fair to say that SR-GKP still delivers the best comprehensive performance with $PR \\approx 62.5\\%$. Methods like SFP and FPGM still stay as the best competitors, but they generally cannot deliver consistently good benign & adversarial performances across different depths of ResNets, and are weak on ImageNet benign readings. SR-GKP seems to be the only constantly strong method among all tested experiments and metrics (within top #3 for almost all trials).\n\n---\n### **[Q1 - Ambiguity on term \"one-shot.\"]: Our definition is probably more main-stream.**\n\nThis particular term, much like \"kernel,\" carries different connotations under the pruning context. We are confident that our utilization of one-shot — which implies the pruning procedure is done all at once with no training/fine-tuning done in between (in contrast to iterative pruning, where a prune-train cycle is required) — is a popular, if not the more \"mainstream,\" definition; this is evident by the following heavily-cited pruning papers [2, 3, 4]:\n\n> [3]: *\"... this pruning approach is one-shot: the network is trained once, p% of weights are pruned, and the surviving weights are reset (then retrained)\"*   \n[4]: *\"... the outline above prunes the network to a target sparsity level all at once, known as one-shot pruning.\"*\n\n\nWe authors are also familiar with the OTO procedure, and it is our understanding that OTO is more like a *from-scratch* (in contrast to post-train) *iterative pruning* method, as it iteratively introduces group sparsity during the training of a (often time, randomly initialized) model. Section 3 of [5] ([OpenReview Link](https://openreview.net/pdf?id=7ynoX1ojPMt#page=3)) and hyperparameters like `start_pruning_steps` ([OTO GitHub Link](https://github.com/tianyic/only_train_once/blob/main/tutorials/01.quick_start_resnet18_cifar10.ipynb)) support our understanding. \n\nWe plan to stay on using this \"one-shot\" to describe SR-GKP, though we understand there are popular papers, e.g., SparseGPT [6], that interpret one-shot pruning as the reviewer. We will add a proper clarification for disambiguation.\n\n---\nRef.   \n[1] Ye et al., Adversarial Robustness vs Model Compression, or Both? ICCV 2019   \n[2] Liu & Sun et el., Rethinking the Value of Network Pruning. ICLR 2019   \n[3] Frankle & Carbin, The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks. ICLR 2019 (Best Paper Award)   \n[4] Renda et al., Comparing Rewinding and Fine-tuning in Neural Network Pruning. ICLR 2020  \n[5] Chen et al., OTOv2: Automatic, Generic, User-Friendly. ICLR 2023   \n[6] Frantar and Alistarh, SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot. ICML 2023   \n"}, {"review_id": "pbmKddnl5q", "reviewer": "Reviewer_xyz8", "summary": "This paper investigates the adversarial robustness of modern structured pruning methods in deep neural networks. The paper insists that while naively pruned networks show poor performance under simple adversarial attacks, there is a lack of thorough investigation into the adversarial performance of carefully designed modern structured pruning methods. The authors aim to address this gap and provide remedies named Smoothly Robust Grouped Kernel Pruning (SR-GKP), a one-shot post-training pruning method for improving adversarial robustness without incurring significant additional costs.", "questions": "1. What is the best(a) and best(b) in Table 2? No explanations for this.\n2. Why does the RAP-ADMM performance against FGSM in Figure 5 lower than the unpruned version?", "limitations": "Yes, the authors have addressed the limitations and potential negative societal impact of their work.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1. Clarity: Clear writing. Easy to understand paper.\n2. The paper is technically sound.", "weaknesses": "1. The novelty of the method is not high. The method is highly dependent on TMI-GKP [Ref_1]. The method looks very similar to previous TMI-GKP. Even if the paper has a method with smoothness, it still sounds like one of the variants of TMI-GKP under the same umbrella.\n2. The adversarial performance has an incremental improvement compared to original TMI-GKP in Table 2. Both best benign performance and best(a) performance have a small improvement (less than 2%), which is not a significant contribution.\n3. The paper is based on empirical study. To increase the value of the paper, it would be better if the authors can provide a theoretical analysis on why the proposed method is helpful to both benign accuracy and adversarial accuracy (especially, why benign accuracy is increased?).\n4. Typos and mistakes make readers confused. \nTable 2 caption: ResNet-56 on CIFAR-10. All pruning are performed on the same baseline model. “Best (a)” represents the performance of a model checkpoint that meats\nLine 263: So Smoothness(GKbranch) would suggest this particular GKbranch has a greater smoothness value (a.k.a. “less smooth”) then all other GKbranch candidates in considerations.\nLine 314: Figure ?? provides a vivid illustration with LRF — a 2021 method— showing the weakest adversarially robustness across the plotted methods\n\n---\n\n**Post-Rebuttals**\n\nI would like to thank authors for their detailed response and for their effort in improving paper according to reviewer’s suggestions. After reading the wall-clock runtime comparison in Table 7 pointed out by authors in supplementary material, I respectfully agree with the author’s opinion on the novelty of the paper. The authors have adequately addressed most of my concerns regarding clarity except for the theoretical analysis. I have updated my rating to accept for this paper through author-reviewer discussion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your feedback.\n\n### **[W1 - Limited novelty]: Please refer to our [general comment](https://openreview.net/forum?id=Pjky9XG8zP&noteId=1EzOrtnZm3).** \n\n### **[W2 - Incremental improvement to TMI-GKP]: We respectfully disagree.**\n\nWe respectfully disagree with the reviewer's assessment. Below is an abbreviated table with SR-GKP's performance noted in comparison to TMI-GKP, we'd argue the gaps are substantial on many metrics.\n\n> Gap between SR-GKP and TMI-GKP on CIFAR10 experiments in %, `+` indicates SR-GKP is better.\n\n| Model-Dataset | Best Benign | Best FGSM $\\varepsilon = 0.01$ | Best FGSM $\\varepsilon = 0.1$ | Best PGD |\n|:-:|:-:|:-:|:-:|:-:|\n| ResNet32-CIFAR10        | -0.02 | **+0.91** | **+0.81** | **+1.31** |\n| ResNet56-CIFAR10        | +0.13 | +0.52 | **+1.08** | +0.34 |\n| ResNet110-CIFAR10       | +0.10 | **+1.12** | **+1.94** | **+1.01** |\n| VGG16-CIFAR10           | -0.12 | +0.37 | **+3.73** | +0.56 |\n\n> Gap between SR-GKP and TMI-GKP on ImageNet/TinyImageNet experiments in %, `+` indicates SR-GKP is better.\n\n| Model-Dataset | Best Benign | Best FGSM $\\varepsilon = 0.001$ | Best FGSM $\\varepsilon = 0.01$ | Best FGSM $\\varepsilon = 0.1$ | Best PGD |\n|:-:|:-:|:-:|:-:|:-:|:-:|\n| ResNet56-TinyImageNet   | **+3.35** | **+4.07** | **+1.99** | **+0.09** | +1.66 |\n| ResNet101-TinyImageNet  | **+2.52** | **+2.68** | **+4.56** | +0.19 | **+4.82** |\n| ResNet50-ImageNet       | +0.27 | +0.4 | +0.59 | +0.01 | **+0.82** |\n\nFurther, if we may kindly direct your attention to Table 7 ([Page 16](https://openreview.net/attachment?id=Pjky9XG8zP&name=supplementary_material#page=16)), our method achieves similar benign performance to TMI-GKP, but with a much faster wall-clock runtime on the pruning stage. Given TMI-GKP and SR-GKP share the same fine-tuning stage, our method is more \"executionally efficient\" overall.\n\n### **[W3 - Need theoretical analysis]: To the best of our knowledge, nothing direct is possible, but here is a discussion.**\n\nMost provable justifications for NN often require a setup distant from the actual models used in applied tasks. e.g., two-layer ReLU-activated ultra-wide MLP [1] vs ResNets. Further, given the procedure of GKP adjusts the compute graph of the original model (standard conv —> grouped conv), theoretical analysis is particularly hard. \n\nThat being said, there is literature — following the style of \"principled empirical evaluations\" — able to provide some insights under the general context of pruning, smoothness, and adversarial robustness. Here's a brief list:\n\n1. [2, 3] Suggest dropout — a long-standing pruning operation — may reduce over- and under-fitting by preventing over co-adaptation and making the model's gradient more aligned. [4] touches on the seemingly contradicting claim of pruning-improving-generalization and over-parameterization double decent phenomenon and showcases that pruning may induce a generalization-favored regularization effect with a proper sparsity.\n2. [5, 6] Suggest NNs often learn LFC first, then the HFC. Given LFC are often more robust features, a model generated by post-train pruning methods might retrain more learning on LFC, thus, leaving more room to fit on HFC and cause worse adversarial performance.\n3. [7, 8] Showcase smoothness may reduce fitting on HFC.\n\nWe believe the reviewer would agree that this kind of investigation would require a sophisticated experiment chain that largely deviated from the setup of \"measuring practical performances of methods under a standard input setting\" (which is the main theme of our work's experiments), and worthy of a paper of its own. Thus, we leave such kinds of studies to future works.\n\n### **[W4 - Typos]: Thanks for the careful read, will fix!** \n\n---\n\n\n\n### **[Q1 - Caption clarification]**\n\n“Best (a)” represents a valid model checkpoint (that meats the showcased MACs/Params reduction) with the best performance against eval metrics (a). In this case, (a) means FGSM w/ $\\varepsilon=0.01$. This is a render error, which we attempted to make a remedy by supplying a correction at the top of our appendix ([Page 13](https://openreview.net/attachment?id=Pjky9XG8zP&name=supplementary_material#page=18)). We apologize for the confusion caused.\n\n### **[Q2 - Why do RAP-ADMM FGSM readings lower than the unpruned baseline]**\n\nWe don't quite grasp the foundation of this question, as it is common for a pruned model to exhibit lower acc reading than its unpruned counterpart (regardless the task is adversarial or not), simply because the pruned network has less capacity. Particularly to [Figure 5](https://openreview.net/attachment?id=Pjky9XG8zP&name=supplementary_material#page=8), methods like CC and LRF has lower FGSM reading than the unpruned baseline.\n\nWe suspect the reviewer meant to ask why RAP-ADMM has a much higher PGD performance than its unpruned baseline, as that is indeed an outlier behavior that might have stirred the reviewer's interest. This is simply because RAP-ADMM does adversarial training on PGD-perturbed input ([Line 304-309](https://openreview.net/attachment?id=Pjky9XG8zP&name=supplementary_material#page=9)).\n\n---\n\n\n[1] Du & Zai et al., Gradient Descent Provably Optimizes Over-parameterized Neural Networks. ICLR 2019   \n[2] Srivastava et al., Dropout: A Simple Way to Prevent Neural Networks from Overfitting. JMLR 2014   \n[3] Liu et al., Dropout Reduces Underfitting. arXiv 2023   \n[4] Jin et al., Pruning’s Effect on Generalization Through the Lens of Training and Regularization. NeurIPS 2022   \n[5] Xu et al., Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks. ICLR 2020   \n[6] Wang et al., What do neural networks learn in image classification? A frequency shortcut perspective. ICCV 2023   \n[7] Wang et al., High Frequency Component Helps Explain the Generalization of Convolutional Neural Networks. CVPR 2020   \n[8] Wang et al., Smooth Kernels Improve Adversarial Robustness and Perceptually-Aligned Gradients. ICLR 2020 Submission   "}, {"review_id": "C8RqN8Vfo8", "reviewer": "Reviewer_rEnW", "summary": "The authors base their work on top of Grouped Kernel Pruning (GKP) to alleviate the issue of adversarial robustness when pruning CNNs structurally. The authors propose Smoothly Robust Grouped Kernel Pruning (SR-GKP) that alters how the filter grouping occurs in GKP and how the grouped kernels are selected. The authors claim that this method is one-shot and data-free. Post one-shot pruning, the authors fine-tune the pruned models on the original training set. The authors claim that this method has no extra cost as opposed to other pruning methods to gain adversarial robustness.", "questions": "Q1. As stated on lines 276-277 you state that you fine-tune the model. However, you state that your method is data-free. Why?\n\nQ2. When discarding grouped kernels (Stage 2), how do you decide when to stop pruning a layer?\n\nQ3. How do you decide an appropriate value of $\\alpha$? Does the optimal $\\alpha$ value remain the same across datasets and architectures?\n\nQ4. When pruning a layer (structured), you essentially reduce the output tensor size. This affects the kernels in the following layers (they also need to be pruned). Do you include them to decide whether a grouped kernel is to be pruned?", "limitations": "The method makes progress in alleviating the issue of adversarial robustness. However, the pruned models are still prone to suffer from adversarial inputs, as shown in the empirical results.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "S1. The problem considered is important\n\nS2. The method is clever and technically sound.\n\nS3. Empirical evidence shows merit in using this approach.", "weaknesses": "W1. The baselines for the ImageNet experiments are weak (Table 3). Some suggestions for incorporating recent baselines: [a], [b], [c] (GitHub repos are available).\n\nW2. Lack of ablation studies w.r.t. $\\alpha$\n\nW3. Lack of adherence to the NeurIPS 2023 template (Figure 2)\n\nW4. Writing needs to be improved.\nSome typos: \n- Line 193. find\n- Line 222 and 243. (Appendix C) [brackets]\n- Line 248. Did you mean tamper?\n- Line 310. (and similar experiments showcased in ??)\n\n\n\n\n\n\n\n\n[a] Only Train Once: A One-Shot Neural Network Training And Pruning Framework. Chen et al. 2021\n\n[b] Neural Pruning via Growing Regularization. Wang et al. 2021\n\n[c] Structural Pruning via Latency-Saliency Knapsack. Shen et al. 2022\n\n[d] DFPC: Data flow driven pruning of coupled channels without data. Narshana et al. 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "(No char left, but thx!)\n\n### **[W1 - Weak baseline on ImageNet]: The choice of compared methods on ImageNet is derived from our CIFAR10 experiments. We didn't cherry/lemon pick them.**\n\nGiven we need to replicate full ImageNet experiments to obtain adv. results, we opt only to compare methods showcased strong performance in CIFAR10 exps, which are SFP and FPGM ([Table {9, 10, 11}](https://openreview.net/attachment?id=Pjky9XG8zP&name=supplementary_material#page=18)). It just so happened that both methods have weak benign readings on ImageNet, which suggests it is hard to deliver strong results across different dataset-model combinations, making SR-GKP's strong across-the-board performance particularly cherished.\n\nWe believe this method-filtering mechanism is sound. Yet, we want to note further that ImageNet with adv. metrics are rarely evaluated under the context of pruning + adv. robustness [1, 2], mainly because many of ImageNet subclasses are closely related in terms of semantics (e.g., man-eating shark vs tiger shark) [3]. We provide the ImageNet results primarily to show SR-GKP can deliver competitive benign ImageNet performance. We will emphasize this in our main text to add clarity.\n\nThat being said since the reviewer is interested, we incorporated those methods on ImageNet and CifarResNets to the best of their repo's reproducibility:\n\n| Method on ResNet50-ImageNet | Benign | FGSM $\\varepsilon = 0.001$ | FGSM $\\varepsilon = 0.01$ | FGSM $\\varepsilon = 0.1$ | PGD |\n|:-:|:-:|:-:|:-:|:-:|:-:|\n| Unpruned | 76.13 | 70.18 | 28.70 | 13.93 | 9.27 |\n| OTOv2 (post-train) | **75.38** | **68.04** | 21.01 | 12.76 | 5.26  |\n| DFPC | 73.80 | 67.45 | 25.15 | 12.24 | 7.10 |\n| FPGM | 75.04 | 68.50 | 25.84 | 13.06 | 7.43 |\n| SFP | 58.50 | 55.72 | 25.82 | 9.01 | 11.00 |\n| SR-GKP | 75.34 | **68.04** | **26.65** | **15.94** | 7.85 |\n\n| Method | Model on CIFAR10 | Benign | FGSM $\\varepsilon = 0.01$ | FGSM $\\varepsilon = 0.1$ | PGD | \n|:-:|:-:|:-:|:-:|:-:|:-:|\n| OTOv2 (from-scratch) | ResNet32  | 90.97 | 66.36 | 17.28 | 27.15 |\n| OTOv2 (post-train)   | ResNet32  | 92.14 | 70.63 | 28.01 | **31.96** |\n| SR-GKP               | ResNet32  | **93.02** | **71.48** | **31.11** | 31.63 |\n| | | | | | |\n| OTOv2 (from-scratch) | ResNet56  | 91.57 | 65.91 | 23.63 | 25.52 |\n| OTOv2 (post-train)   | ResNet56  | 93.02 | 74.29 | 40.46 | 41.58 |\n| GReg-1               | ResNet56  | 91.06 | 70.56 | 23.66 | 33.48 |\n| SR-GKP               | ResNet56  | **94.08** | **76.40** | **45.17** | **44.85** |\n| | | | | | |\n| OTOv2 (from-scratch) | ResNet110 | 91.58 | 71.43 | 23.94 | 34.74 |\n| OTOv2 (post-train)   | ResNet110 | 93.99 | 77.25 | 47.96 | 48.46 |\n| SR-GKP               | ResNet110 | **95.00** | **79.14** | **49.09** | **50.01** |\n\nSome asked methods are not listed due to reproduction hardship: e.g., GReg induces memory overflow on our 80G A100 GPUs; DFPC's data-driven implementation is not yet shared; HALP requires a handcrafted config and hardware-specific latency LUT. Due to character limitation we will leave it as that; we'd be happy to provide more info during the discussion session, should the reviewer be interested.\n\n### **[W2, Q3 - Lack of ablation study on $\\alpha$]: It is in Appendix D.3**\n\nWe kindly direct the reviewer's attention to [Appendix D.3](https://openreview.net/attachment?id=Pjky9XG8zP&name=supplementary_material#page=22), where there is a full-scale ablation study on $\\alpha$. We'd say the choice of $\\alpha$ is pretty consistent.\n\n### **[W3&4 Template and typos]: Thanks, will fix!** \n\n---\n\n### **[Q1 - Why claim \"data-free\" when fine-tuning is required?]: Ambiguous term — good point, will replace.**\n\nWe meant the pruning stage of SR-GKP can be done without access to data. Upon inspection, it looks like some pruning arts use the term \"data-free\" (no access to data) and \"data-agnostic\" (independent of data) interchangeably. e.g., SynFlow [4], a data-agonistic method, is referred as data-free in [5]. [6], a namely \"data-free\" method, still *\"enabling operations like fine-tuning...\"*.\n\nTo avoid ambiguities, we will replace it with \"weight-dependent\" should there be a camera-ready version, as we believe the reviewer's perspective is less ambiguous and appears to be more mainstream in recent pruning arts [DFPC, 7]. \n\n### **[Q2 - When to stop pruning a layer?]: When it reaches the target (universal) pruning ratio.**\n\n### **[Q4 - SR-GKP reduces output tensor size between layers]: Not really.**\n\nSR-GKP does not reduce the output tensor size. Rather, it reduces the computation required to generate those (between-layer) outputs. Take [Figure 1](https://openreview.net/attachment?id=Pjky9XG8zP&name=supplementary_material#page=2) as an example: the unpruned model has 6 features maps as output for having 6 filters. Yet in GKP, it still has 6 feature maps as it has 3 grouped filters each with a size of 2; but fewer kernels (3, vs 6) are utilized to generate such feature maps. \n\n### **[Limitation - SR-GKP pruned models still suffer from adv. inputs]: True, but SR-GKP already showcased better performance than unpruned baseline, as well as many methods rely on adv. training.**\n\nGiven an unlimited perturbation budget, no adv. defense can be immune to all adv. attacks, regardless if the victim model is pruned or not. We consider our method a worthy progression for the subtitled reasons, yet the investigation-side of our work also unveils many previously unknown findings to the pruning community.\n\n---\n\n[1] Chen and Zhang et al., Linearity Grafting... Robustness. ICML 2022   \n[2] Ye et al., Adversarial Robustness ... Both? ICCV 2019   \n[3] Ozbulak & Pintor et al., Evaluating Adversarial ... Classes. NeurIPS 2021 (Workshop)   \n[4] Tanaka & Kunin et al., Pruning neural networks... Flow. NeurIPS 2020   \n[5] Pellegrini et al., Neural Network Pruning Denoises ... Tasks. PMLR 2022   \n[6] Srinivas et al., Data-free Parameter ... Networks. arXiv 2015   \n[7] Yvinec et al., RED: Looking for Redundancies ... Networks. NeurIPS 2021   "}]

Author rebuttal:
# General Response

We thank all reviewers for your valuable time and comments. We are glad that many reviewers found:

* **Our task to be well-motivated**
    * `rEnW`: *"The problem considered is important."*
    * `sejQ`: *"The motivation is reasonable."*
* **Our approach to be technically sound** 
    * `rEnW`: *"The method is clever and technically sound."*
    * `xyz8`: *"The paper is technically sound."*
    * `Tjyo`: *"The paper proposes ... and unit-tests it well."*
    * `Tjyo`: *"The proposed method effectively combines two existing works, and provides solid gains."*
    * `sejQ`: *"The authors designed sufficient ablation experiments to validate that the proposed ... strategies."* 
* **Our empirical evaluation to be solid**
    * `Tjyo`: *"The empirical evaluation of the paper is solid."*
    * `rEnW`: *"Empirical evidence shows merit in using this approach."*

Again, we are grateful for your recognition. 

In terms of limitations, other than the cosmetic issues and some factual remarks, two common notions are:

* **More experiments**
    * `sejQ` asked for the inclusion of HARP [1], a robustness-aware weight pruning method with a channel pruning extension. We delivered.
    * `sejQ` asked for more variety in pruning rates. We delivered.
    * `rEnW` asked for more ImageNet baseline. We explained how we reached such baseline selection and delivered the asked comparison to the best of their repo's reproducibility; we even went beyond and delivered the reproducible ones on CifarResNets.
* **Limited novelty**
    * `sejQ`: *"The novelty is limited: kernel smoothness and grouped kernel pruning have been studied in some works. The core contribution of the paper seems to be a direct combination of two ideas."*
    * `xyz8`: *"The novelty of the method is not high... Even if the paper has a method with smoothness, it still sounds like one of the variants of TMI-GKP under the same umbrella."*

While novelty is certainly important, we believe the novelty of a paper can be roughly viewed from two aspects: **empirical novelty** — as if the paper unveils properties and behavior that is unknown to the general audience (e.g., lottery ticket style [2]); and **technical novelty** — as if the propose solution designs anything new to achieve its goal (e.g., RWKV [3]).

**We argue that our work is rich in empirical novelty, as before it, one cannot answer the following important questions with certainty**:
1. Are carefully designed structured pruning methods with similar benign performance on par with each other on adversarial tasks? Are newer methods more robust? 
2. Is Grouped Kernel Pruning naturally more robust to adversarial attacks than filter/channel pruning?
3. Is adversarial training absolutely necessary for obtaining a robust pruned model?
4. Is it possible to have a structurally pruned model that performs well under both benign and adversarial tasks?

In terms of technical novelty, we agree that our method looks like a direct combination of kernel smoothness and GKP from the surface. However, we'd like to emphasize that the **combination between kernel smoothness and GKP is unique**, as mixing kernel smoothness  — a kernel-level metric — with filter pruning scheme will result in a catastrophic performance drop ([C.1.3](https://openreview.net/attachment?id=Pjky9XG8zP&name=supplementary_material#page=16)). Yet, **such mix-up is non-trivial to figure out**, as a direct mixture will not work, so we have to apply principled ablation studies to find out the right way of integration ([C.1.1-C.1.2](https://openreview.net/attachment?id=Pjky9XG8zP&name=supplementary_material#page=14)). 


**We also intensionally kept our method simple**, as GKP is an extremely young pruning family with only three well-developed methods under its wing [4, 5, 6]. Though it has gained some recognition for its strong benign performance (with [6] winning AAAI Oral), the procedure and capability of GKP can be foreign to many readers. Under this context, if we proposed a very sophisticated and trick-heavy GKP method that performs well on adversarial tasks, one would likely question if such gains are from the extra addition, or is it due to GKP granularity being naturally more adversarially robust and more flexible on incorporating robustness-related maneuvers?

We argue that with the necessity of including backgrounds on GKP, structured pruning, adversarial robustness, (tools like) kernel smoothness, and a vast amount of empirical results, **a relatively vanilla GKP implementation is the most appropriate vehicle to deliver our claim: that GKP variants are superior on both benign and adversarial tasks**, and thus, *"one less reason for filter pruning."*

We further note — though we believe our reviewers and AC are well-aware — that technical novelty is not the only important contribution. We believe our investigation on how to do GKP differently ([Section 3.2.1](https://openreview.net/attachment?id=Pjky9XG8zP&name=supplementary_material#page=6)), benchmark report on various methods against different evaluation metrics, as well as our to-be-open-sourced codebase and checkpoints for fair and comprehensive evaluation... **may provide essential benefits to future GKP developers and the pruning community at large.**


---
Ref.   
[1] Zhao & Wressnegger, Holistic Adversarially Robust Pruning, ICLR 2023   
[2] Frankle & Carbin, The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks. ICLR 2019    
[3] Peng et al., RWKV: Reinventing RNNs for the Transformer Era. arXiv 2023   
[4] Zhong et al., Revisit Kernel Pruning with Lottery Regulated Grouped Convolutions. ICLR 2022   
[5] Zhang et al., Group-based network pruning via nonlinear relationship between convolution filters. Applied Intelligence, 2022   
[6] Park et al., Dynamic Structure Pruning for Compressing CNNs. AAAI 2023   

---

Meta-review: Although the reviewers acknowledged the importance of the problem tackled, the motivation behind this work, the empirical evaluation, and the soundness of the method, they expressed some concerns about the writing of the paper, the verification of some claims, the novelty of the proposed method, the lack of comparison to some baselines, the ambiguity of some terms ("one-shot", "data-free"), and the lack of theoretical analysis. The authors' feedback and the thorough discussion between the reviewers and the authors eventually convinced all reviewers, who reached a consensus for acceptance.

The AC agrees that this study, at the intersection of network pruning and adversarial robustness, is of interest to a large portion of the community. The results are interesting, and the method, although relatively simple, is intuitive and effective. As such, this paper can be accepted to NeurIPS. However, the AC also agrees with the reviewers that writing quality should be improved and urges the authors to carefully review their paper for the final version.
Decision: Accept (poster)