Title: One Less Reason for Filter Pruning: Gaining Free Adversarial Robustness with Structured Grouped Kernel Pruning
Venue: NeurIPS 2023
Accepted: yes
Keywords: pruning

structured pruning

adversarial robustness

grouped kernel pruning

CNN

one-shot
Abstract: Densely structured pruning methods utilizing simple pruning heuristics can deliver immediate compression and acceleration benefits with acceptable benign performances. However, empirical findings indicate such naively pruned networks are extremely fragile under simple adversarial attacks. Naturally, we would be interested in knowing if such a phenomenon also holds for carefully designed modern structured pruning methods. If so, then to what extent is the severity? And what kind of remedies are available? Unfortunately, both the questions and the solution remain largely unaddressed: no prior art is able to provide a thorough investigation on the adversarial performance of modern structured pruning methods (spoiler: it is not good), yet the few works that attempt to provide mitigation often do so at various extra costs with only to-be-desired performance.

In this work, we answer both questions by fairly and comprehensively investigating the adversarial performance of 10+ popular structured pruning methods. Solution-wise, we take advantage of *Grouped Kernel Pruning (GKP)*'s recent success in pushing densely structured pruning freedom to a more fine-grained level. By mixing up kernel smoothness — a classic robustness-related kernel-level metric — into a modified GKP procedure, we present a one-shot-post-train-weight-dependent GKP method capable of advancing SOTA performance on both the benign and adversarial scale, while requiring no extra (in fact, often less) cost than a standard pruning procedure. Please refer to our [GitHub repository](https://github.com/henryzhongsc/adv_robust_gkp) for code implementation, tool sharing, and model checkpoints.

One Less Reason for Filter-Pruning: Gaining Free Adversarial Robustness with Structured Grouped Kernel Pruning

Abstract

Densely structured pruning methods utilizing simple pruning heuristics can deliver immediate compression and acceleration benefits with acceptable benign performances. However, empirical findings indicate such naïvely pruned networks are extremely fragile under simple adversarial attacks. Naturally, we would be interested in knowing if such a phenomenon also holds for carefully designed modern structured pruning methods. If so, then to what extent is the severity? And what kind of remedies are available? Unfortunately, both questions remain largely unaddressed: no prior art is able to provide a thorough investigation on the adversarial performance of modern structured pruning methods (spoiler: it is not good), yet the few works that attempt to provide mitigation often do so at various extra costs with only to-be-desired performance.

In this work, we answer both questions by fairly and comprehensively investigating the adversarial performance of 10+ popular structured pruning methods. Solutionwise, we take advantage of Grouped Kernel Pruning (GKP) 's recent success in pushing densely structured pruning freedom to a more fine-grained level. By mixing up kernel smoothness - a classic robustness-related kernel-level metric into a modified GKP procedure, we present a one-shot-post-train-weight-dependent GKP method capable of advancing SOTA performance on both the benign and adversarial scale, while requiring no extra (in fact, often less) cost than a standard pruning procedure. Please refer to our GitHub repository for code implementation, tool sharing, and model checkpoints.

1 Introduction

Convolutional neural networks (CNNs) have demonstrated solid performance on tasks centered around computer vision. However, with modern CNNs growing in both widths and depths, the issue of over-parameterization has drawn increasing attention due to such networks often requiring large computational resources and memory capacity. To mitigate the burden, network pruning - the study of removing redundant parameters from original networks without significant performance loss has become a popular approach for its simplicity and directness [LeCun et al., 1989, Blalock et al., 2020, He and Xiao, 2023].

Despite the popularity of the pruning field in general, few prior arts have been available to provide improved adversarial robustness under the constraint of (densely) structured pruning ; even though empirical findings show vanilla structured pruning methods implemented with naïve pruning strategies often experience huge performance drop on such adversarial tasks [Wang et al., 2018, Sehwag et al., 2020, Vemparala et al., 2021]. More concerning, no prior art has made an effort to provide a comprehensive investigation on whether the same phenomenon also exists under carefully designed modern structured pruning methods, where such methods are often capable of delivering excellent benign accuracy retention after pruning (sometimes, even improvements).

Below, we provide a walk-through of why a densely structured method and having an adversarially robust pruned model are preferable and important, to how we developed our solution by leveraging the power of increased structural pruning freedom (grouped kernel pruning) with kernel-level metrics (kernel smoothness). In the later sections of this paper, we replicate and evaluate around 13 popular densely structured pruning methods and variants against various white box (evasion) adversarial attacks, where our proposed method showcases clear dominance.

1.1 Structured v.s. Unstructured Pruning: Accuracy-Efficiency Trade-off

Most of the existing CNN pruning methods can be roughly categorized into structured and unstructured pruning. Note we said roughly because there is no universally agreed delineation between structured and unstructured pruning methods. The general consensus is that methods considered more unstructured often enjoy a higher degree of freedom on where to apply their pruning strategies (e.g., weight-level pruning) and therefore result in better accuracy retention.

In contrast, structured pruning methods often prune weights in a grouped manner following some kind of architecturally defined constraints (e.g., filter-level pruning). Compared to their unstructured counterparts, structured pruning methods are more hardware-friendly and easier to obtain compression and acceleration on commodity hardware, though at the cost of worse accuracy retention. This difference is due to unstructurally pruned networks' tendency to have pruned (zeroed) parameters randomly distributed in the weight matrix, leading to poor data reuse and locality [Yang et al., 2018]. Such unstructurally pruned networks struggle to have wall-clock time speed up without supports like custom-indexing, special operation design, sparse operation libraries, or even dedicated hardware setups [Yang et al., 2018, Han et al., 2016, He and Xiao, 2023].

Among all structured pruning methods, one popular line of research is to produce pruned networks that are entirely dense, a.k.a. densely structured, where the pruned weights are stored in the normal dense tensor format . Such format of a pruned network is considered to be most library/hardwarefriendly and, therefore, most deployable in a practical context. With such significant benefits, densely structured pruning methods consist of the absolute majority of structured pruning methods.

1

2

3

4

5

6

1

2

3

4

5

6

Input Channels

Output Channels

Unpruned Original

1

2

3

4

5

6

1

2

3

4

5

6

Filter/Channel Pruning

1

2

3

4

5

6

1

2

3

4

5

6

Filter

Kernel

1

2

3

4

5

6

1

3

4

6

2

5

2

4

6

2

5

3

5

6

4

6

1

2

4

1

3

Channel

Densely structured pruning methods come with different pruning granularities, where a significant portion of prior arts prune at a filter or channel level. These two types of pruning are historically considered to be the limit of densely structured pruning, as showcased in Figure 1: if we go down one more level, we will have kernel pruning; however, its pruned network is not dense. This is until recently, authors from Zhong et al. [2022] utilized kernel pruning with grouped convolution, where they prune at a grouped kernel level to ensure an entirely dense pruned network. To the best of our knowledge, grouped kernel pruning carries the highest degree of pruning freedom among all densely structured pruning methods.

1.2 Pruned Models Are Fragile Under Adversarial Attacks - But Why Do We Care?

Empirical findings like Wang et al. [2018] suggest that although pruned neural networks may have acceptable benign accuracy, they are often more vulnerable to adversarial attacks. Adversarial robustness is recognized as a long-standing metric to evaluate model quality, as a model with undesired adversarial robustness can be easily exploited to produce wrong and potentially harmful output, resulting in fairness and accountability issues.

We would argue such robustness properties are especially valued under the context of (structured) pruning , where pruned models are often deployed to resource-constraint devices with less central oversight available and requiring execution in a more real-time manner. Imagine if an OCR model for real-time check redeeming can be maliciously exploited to read the number 1 as 9 ; the result will surely be unpleasant for many parties involved.

To alleviate such a problem (though not under a pruning context), prior arts like Wang et al. [2020b] demonstrate the adversarial robustness of a convolutional network is largely correlated to its sensitiveness to high-frequency components (HFC), where such sensitiveness can be mitigated with some simple kernel-level operations like kernel smoothness. More on this in Section 2.

1.3 Solution: Grouped Kernel Pruning with Adversarial-Robustness-Boosting Kernel Metrics

With the recent Grouped Kernel Pruning (GKP) framework pushing the pruning freedom of densely structured pruning to a (close) kernel-level [Zhong et al., 2022], we explore the unique possibility of mixing up adversarial-robustness-boosting kernel metrics - such as kernel smoothness - into the procedure of GKP. We present Smoothly Robust Grouped Kernel Pruning (SR-GKP), a densely structured pruning method that works in a simple post-train one-shot manner, but is often capable of delivering competitive benign performance and much stronger adversarial performance against SOTA filter and channel pruning methods that require more sophisticated procedures. Solution-wise, our main claims and contributions are:

Free improvement on adversarial robustness. Our method has no extra (in fact, often less) cost compared to a standard pruning method, making the gained adversarial robustness entirely free.

One-shot & post-train & weight-dependent: the simplest procedure with most compatibility. Our method is a one-shot-post-train-weight-dependent 1 structured pruning method for CNN that follows the classic train - prune - fine-tune procedure, where all excessive components are pruned all at once before fine-tuning. This means it is compatible with any trained CNNs (as it does not interfere with the training pipeline) yet straightforward to execute, as the pruning procedure before fine-tuning does not rely on access to data but only the trained model's parameters.

Raise attention to the important but overlooked field of adversarially robust structured pruning. Our method is among the few structured pruning methods capable of delivering pruned networks with improved adversarial performance - a field with severe problems, but receives little recognition or solutions.

On the investigation side, we are the first to comprehensively reveal:

Drastic adversarial performance difference under a similar benign report . We found that while different carefully designed modern densely structured pruning methods may showcase similar benign performance, most are done so at the cost of adversarial robustness.

One less reason for filter/channel pruning: further endorsing GKP . Filter and channel pruning methods have dominated the field of densely structured pruning for years; our work - together with Zhong et al. [2022] and Park et al. [2023] - showcased that when done right, grouped kernel pruning-based methods are superior under both benign and adversarial tasks, making it a promising direction for future densely structured pruning research.

For added bonuses, we are the first ones to reproduce and comprehensively report the benign and adversarial performances of multiple structured pruning methods under a fair setting. We believe the lack of such fair and comprehensive reports (on both benign and adversarial tasks) is mainly due to the lack of user-friendly tools. Thus, alongside our method implementation and checkpoint files, we also provide the pruning community a lightweight open-sourced tool capable of a plug-and-play style of testing different victim models with various adversarial attacks while supporting all procedures a modern pruning method may require .

2 Related Work and Discussion

Due to the page limitation, we will discuss related work regarding white box evasion adversarial attacks, adversarially robust structured pruning, and grouped kernel pruning. Other related topics, such as the compression/acceleration implications of structured and unstructured pruning methods, input component frequency with kernel smoothness, as well as other related ML efficiency and AI safety literature, will be introduced in Appendix B.

Adversarial Attacks. Neural networks are known to be vulnerable to adversarial attacks, i.e., a small perturbation applied to the inputs can mislead models to make wrong predictions Szegedy et al. [2014], Goodfellow et al. [2014]. In practice, popular adversarial attacks can often be categorized as white-box and black-box evasion attacks. The difference being white-box attacks have access to the entirety of the model, including input features, architectures, and model parameters, while black-box attacks' access is often constrained (e.g., only input-output pair). Thus, white-box attacks are almost always more effective and efficient than black-box ones; in fact, many classic black-box attacks are constructed in a way to approximate the information that is directly accessible by white-box attacks (e.g., gradient) [Chen et al., 2020]. We opt for white-box attacks for the scope of this paper, given one significant use case of structurally pruned models is edge-device deployments, where the model is more likely to be accessed. Also, white-box attacks generally offer a more straightforward workflow with harder challenges posed.

Structured Pruning for Adversarial Robustness. Structured pruning methods, which arguably carry the most practical significance, have been heavily studied throughout the years [Molchanov et al., 2017, Yu et al., 2018, He et al., 2019, Wang et al., 2019a,b, Lin et al., 2019, He et al., 2018, Li et al., 2021, Zhong et al., 2022]. Despite their popularity, few of them focus on adversarial robustness. To the best of our knowledge, there are only four prior arts presenting structured pruning methods while claiming improved performance on adversarial robustness metrics [Vemparala et al., 2021, Ye et al., 2019, Sehwag et al., 2020, Zhao and Wressnegger, 2023]. Unfortunately, Vemparala et al. [2021] does not have a public repository for code, Ye et al. [2019] does not have any experiment on standard BasicBlock ResNets for comparative investigation despite their popularity. [Blalock et al., 2020], Sehwag et al. [2020] and Zhao and Wressnegger [2023] mostly propose unstructured methods with only a few structurally pruned ablation studies conducted on limited model-dataset combinations, with some of their structured pruning implementation not published.

The lack of traffic, infrastructure, or baseline in this area has undoubtedly created deterrents to all interested scholars. To fill the gap, we provide the community an open-sourced toolkit capable of testing various victim models against different adversarial attacks under a pruning context, coming with 10+ popular structured pruning methods already integrated for comparative evaluation. Model checkpoints are also available for direct evaluations.

Grouped Kernel Pruning. Our work relies on the GKP framework - particularly inspired by the recent work of Zhong et al. [2022]. The core of GKP is grouped kernel pruning and reconstruction to a grouped convolution format, where this combination has been explored a few times under the context of structure pruning.

Specifically, Zhong et al. [2022] proposed their take on grouped kernel pruning with a three-stage procedure: filter clustering, generating then deciding which grouped kernel pruning strategy to employ, then reconstructing to grouped convolution format via permutation. This particular framework solved the previous drawback of requiring a complex procedure, yet a rich set of experiment results was showcased to demonstrate its performance advantages against many SOTA structured pruning methods. Our proposed method is largely enabled by the extra pruning freedom that GKP provides.

Outside of Zhong et al. [2022], concurrent work like Zhang et al. [2022a] and follow-up work like Park et al. [2023] have showcased the excellent benign performance of different GKP implementations under a benign context.

3 Proposed Method

3.1 Motivation: Pruning May Amplify Overfitting to High-Frequency Components

Wang et al. [2020b] suggests CNNs are prone to overfitting high-frequency components (HFC) of inputs - a type of feature that is not robust yet can be easily replicated with adversarial perturbations. Though Wang's finding is towards an unpruned model, such phenomenon can be found, and in fact, even amplified, under a structural pruning setting.

Input

Reconstructed

Reconstructed

Naive

with LFC

with HFC

Unpruned

Output

Pruned

A classic demonstration of such phenomena can be seen in Figure 2. Despite the frog -labeled figure reconstructed with only low-frequency components showing visible resemblance to its original benign format, a ResNet-56 model pruned by L1Norm filter pruning [Li et al., 2017] cannot classify it correctly. However, such pruned models can somehow correctly classify the same input reconstructed with only HFCs, even if it already lost all semantics of a frog to a human audience. This indicates a model pruned by methods without having adversarial robustness in consideration is more likely to overfit to HFC.

We emphasize that the above example (Figure 2) is not a cherry-picked one. As shown in Table 1, by reconstructing the entire test set of CIFAR-10 with solely their high-frequency components, we find that a structurally pruned model is more prone to fitting HFCs than its unpruned counterpart under various settings. This is potentially because HFCs are easily learnable features under a benign setting, so pruned models want to 'make most use' of their remaining weights given the reduced network capacity, and therefore become even more overfitted to HFCs by treating them as short-cut features.

Kernel smoothness as an indicator for learning from HFC. Fortunately, Wang et al. [2020b] suggests kernel smoothness is highly correlated to the learning of HFC. Specifically, Wang et al. [2020b] found that a CNN with 'smoother' kernels - where neighbor weights within a 2D kernel have less of a value difference - will reduce the overfitting of HFCs, thus making the model more robust against adversarial perturbations. For the ease of the following conversation, we define the kernel smoothness of a convolution kernel k for k ∈ R H × W to be:

where H and W are the kernel dimensions; in most CNNs, such dimensions are set to 3 × 3 .

The finding of kernel smoothness and adversarial robustness presents a unique opportunity under the context of GKP. Prior to GKP, densely structured pruning is almost always done at a filter/channel level, where kernel-level metrics/operation have little bearing when relaxed (see Appendix C.1.3). However, GKP prunes at a (close) kernel level, where a kernel-level metric may still retain its power.

3.2 Mixing Smoothness into Grouped Kernel Pruning Procedure

It is natural to want to encapsulate some kernel-level operations/metrics - in this case, kernel smoothness - into the procedure of GKP. However, the challenge comes with how we can do it in an efficient and effective manner. Particularly, how can we achieve improved adversarial performance without sacrificing benign tasks?

For the ease of illustration, let W ℓ ∈ R C ℓ out × C ℓ in × H ℓ × W ℓ be the weight of the ℓ -th convolutional layer, which consist of C ℓ out filters, with each filter consisting of C ℓ in number of H ℓ × W ℓ 2D kernels. According to Zhong et al. [2022], a standard GKP procedure has two potential stages:

Stage 1: Filter grouping stage : where the C ℓ out filters are clustered into n equal-sized filter groups { FG ℓ i , FG ℓ j , . . . , FG ℓ n } , with each filter group FG ℓ ∈ R C ℓ out /n × C ℓ in × H ℓ × W ℓ .

Stage 2: Pruning strategies obtaining/pruning stage : where the pruning method generates a set of 'candidate' grouped kernel pruning strategies to be evaluated and select from; a grouped kernel is defined as a GK ∈ FG ℓ with GK having a shape of C ℓ out /n × 1 × H ℓ × W ℓ . Finally, we evaluate all collected candidate strategies and decide which to pursue.

However, how to apply kernel smoothness-related criteria to such stages is a non-trivial question. Our empirical results from some proof-of-concept experiments suggest some naïve applications either will not work at all or will only work by significantly sacrificing the performance on benign tasks: e.g., if we simply replace Stage 2 above by pruning the grouped kernels with greater ∑ k ∈ GK smoothness ( k ) , we will experience a huge drop on benign accuracy (Appendix C). Therefore, we must drive our attention to discover some more sophisticated ways of mixing up such criteria into the above stages.

3.2.1 Smoothness Snaking: Is Filter Clustering the Only Right Answer for GKP?

Per TMI-GKP [Zhong et al., 2022], filters within the same convolutional layer are grouped into several equal-sized groups by a filter clustering schema , which consist of some different combinations of dimensionality reduction and clustering techniques. The motivation of grouping by clustering is natural as it establishes a preferable search space for pruning algorithms, where the power of pruned components can likely be retrained via similar unpruned components within the same group. However, later procedures of TMI-GKP (e.g., Stage 2 per Section 3.2) then seek to maintain a diverse representation of kept components within each filter group, which begs the question: is filter clustering the only right answer for GKP? If finding a diverse set of unpruned grouped kernels is the goal, why not have filter groups with filter diversity to start with? And if the answer to the first question is 'No.' How can we utilize this opportunity to mix up with kernel smoothness criteria?

F1

F5

F4

F6

F2

F3

Group 1

Group 2

Group 3

1

2

3

4

5

6

1

2

3

4

5

6

Input Channels

Unpruned Conv Layer

Filter

=

F1

=

F2

=

F3

=

F4

=

F5

=

F6

Group 1

Group 2

Group 3

F1

F1

F5

F1

F5

F4

F1

F5

F4

F6

F1

F5

F4

F6

F2

Group 2

Group 3

F1

F5

F4

F6

F2

F3

Group 1

Sort Filters by Smoothness

(longer bar as being less smooth)

Prepare

n

Filter Groups to Fill

1

2

3

4

5

6

1

3

5

2

4

6

Form Grouped Conv Layer

with

groups = n

Output Channels

'Snake-like' filter assignments

Group 1

Group 2

Group 3

Grouped-based pruning will always seek out some kind of balance among groups to avoid a skewed distribution where all important components are distributed to certain groups [Zhong et al., 2022]. Following this principle, we want our filter groups to have balanced smoothness - so that we do not end up having any filter group that is 'over' or 'under-smoothed' to start with for pruning. However, this is equivalent to the partition problem [Korf, 1998], which is known to be NP-hard. Given the filter grouping stage is often robust to adjustments 2 , we proposed to sort filters according to their smoothness, then assign them iteratively in an S-shaped 'snaking' manner across a predefined number of filter groups, as shown in Figure 3, namely Smoothness Snaking . Empirical results suggest although smoothness snaking may not be as optimal as the dynamic clustering scheme in Zhong et al. [2022] in terms of benign performance, it is able to provide better adversarial robustness under adversarial attacks and is much faster to execute (Table 8) due to the absence of dimensionality reduction & clustering procedures (Appendix C). We consider this to be a successful mix-up.

3.2.2 Smooth Beam Greedy GKP Search

Knowing that the pruning strategies obtaining/decision stage (Stage 2 per Section 3.2) of GKP is sensitive to tampering, a direct application of smoothness criteria would not work (Appendix C). This indicates the distance-based cost formula (Equation 5) proposed in Zhong et al. [2022] carries a significant influence on the performance of a pruned network, which is unsurprising given the vast popularity of distance-based pruning arts. Since we can't directly replace this cost formula, we propose to widen the capacity of pruning strategies obtaining stage with smoothness in mind while keeping the decision stage akin to the cost formula. By doing this, the chances of more 'smoothness-aware' pruning strategies being employed are increased.

1

2

1

2

3

4

5

6

GK2

GK1

GK5

GK6

GK4

GK2

GK3

0

1

2

3

4

(b) Decide the first Kept GK

via

multiple restarts then

sort

Candidate GKs by their

Cost

- euclidean distance -

to Kept GKs Branches

(higher index as being more distant)

Branch #1:

0.75

×

0 + 0.25

×

1 = 0.25

0.75

×

3 + 0.25

×

2 = 2.75

0.75

×

2 + 0.25

×

3 = 2.25

0.75

×

1 + 0.25

×

0 = 0.75

GK1

Kept GKs:

GK3

GK2

GK6

GK4

GK5

0

1

2

GK5

GK6

GK4

0

1

2

GK4

GK5

GK4

GK1

GK3

GK5

GK1

GK3

GK4

GK1

GK4

GK2

GK1

GK2

GK6

(c) Add

bw

of GKs as Kept GKs

(

bw

= beam width = 2 in this trace; GKs with

higher cost rank indices are preferred)

GK1

Branch #1:

GK3

GK1

Branch #2:

GK2

(d) Sort Candidate GKs by their

Cost

to Kept GKs Branches

GK1

Kept GKs:

GK3

GK2

(e) Add bw of GKs as Kept GKs

GK6

(a) Given an unpruned Grouped Filter

Kept GKs:

Kept GKs

GK1

Branch #1-1:

GK3

GK5

GK1

Branch #1-2:

GK3

GK4

GK1

Branch #2-1:

GK2

GK4

GK1

Branch #2-2:

GK2

GK6

Kept GKs

Cost

and

Smoothness

(higher index as being more distant to Kept GKs/smoother)

GK1

Branch #1:

GK3

GK1

Branch #2:

GK2

Kept GKs

(g) Add bw of GKs as Kept GKs

(

α

= 0.75 in this trace, GKs with higher

GK4

GK4

(h) Back to step (d)

(repeat until the branch length matches

the target pruning rate)

0 - 1

3 - 2

2 - 3

1 - 0

combined rank indices are preferred)

Again, to keep our approach simple, we implemented a custom beam search element to consider more grouped kernels during each advancement (Figure 4). Per each iteration, all gathered candidate grouped kernel pruning strategies will be evaluated against a mixture of smoothness and cost criteria, where only a Beam width amount of grouped kernels will be kept for further advancement (until the desired pruning ratio is reached). The scoring formula to determine which subset of strategies may be kept in the beam is defined as (larger scores are better/preferred):

where ϕ ( Criterion ( GK branch ) ) for Criterion ∈ { Cost , Smoothness } represents the rank of such GK branch when all collected GK branch candidates are sorted according to the given Criterion in a descending order. So ϕ ( Smoothness ( GK branch ) ) = 0 would suggest this particular GK branch has a greater smoothness ranking (a.k.a. 'less smooth') then all other GK branch candidates in considerations. α is a tunable balancing parameter that adjusts the importance of one criterion over

the other. The Smoothness ( GK branch ) equation is simply the sum of all kernels within all GK kept over Equation 1; the Cost ( GK branch ) equation is a modified version of Equation 5 in Zhong et al. [2022], where we removed some hyperparameters for simplicity and to reduce tuning workload. Please refer to Figure 4 and Appendix C.3 for more details.

4 Experiments and Results

4.1 Experiment Setups

We evaluate the efficacy of our method on ResNet-32/56/110 with the BasicBlock implementation, ResNet-50/101 with the BottleNeck implementation, and VGG-16 [He et al., 2016, Simonyan and Zisserman, 2015]. For datasets, we choose CIFAR-10 [Krizhevsky, 2009], Tiny-ImageNet [Wu et al., 2017], and ImageNet-1k [Deng et al., 2009] for a wide range of coverage. For all compared methods and under most model-dataset combinations, we tried our best to replicate them with a ≈ 300 epochs (except for ImageNet, where we only employ 100 epochs) of fine-tuning/retraining budget while maintaining all other settings either identical or proportional to their original publications.

4.2 Compared Methods and Evaluating Criteria

We evaluate our proposed method against up to 13 popular densely structured pruning methods and variants shown in Appendix D. Whenever possible, we produce pruned-and-fine-tuned (or retrained) models upon identical unpruned baseline models with similar post-prune MACs and Params. Then, we compare their inference accuracy on benign inputs as well as adversarially-perturbed inputs powered by FGSM and PGD attacks in various perturbation budgets and intensities [Goodfellow et al., 2014, Madry et al., 2018].

One reporting mechanism that is probably unique to our paper - in comparison to standard pruning art under the benign space - is for some of our methods, results of multiple epochs are reported, each showcasing a method's peak performance against different evaluation metrics (a.k.a. 'superscore'). This is because for benign tasks, following Li et al. [2017] and He et al. [2019], only the epoch checkpoint with the best benign accuracy needs to be reported. But under an adversarial context, if a pruning method is capable of producing more than one fully pruned model during the fine-tuning/retraining stage, oftentimes, the best performer per each evaluation metric does not overlap. We believe it is responsible to report them all as there are no dominant evaluation metrics in our experiments; however, this is an important advantage for methods that can generate multiple usable pruned models (e.g., one-shot pruning) over methods that can only generate a few or just one fully pruned model (e.g., layer-wise iterative pruning). We survey the availability of such checkpoints and their epoch cutoff in Table 10: 'Fully Pruned Epoch' column.

Wealso comprehensively investigate each pruning method against a checklist of questions as presented in Table 10 of Appendix D.1.3, including pruning granularity, procedure, when is the first fully pruned epoch, zero-masked or hard-pruning, and many other important questions. This investigation, along with the epoch budget constraint and baseline control, should provide our audience with a more leveled playing field for fair and informed methods comparison.

Further, to facilitate digesting mass results reported across many methods/variants and against various metrics, we present our large-scale empirical evaluation in three different ways. Take BasicBlock ResNets on CIFAR-10 as an example (with pruning rate ≈ 43 . 75% ), we present the raw results as Table 11, 12, and 13; where we visualize them accordingly as Figure 6, 7, and 8. Last, we make a rank chart, Table 3, to provide a straightforward gauge of the competitiveness among different pruning methods.

4.3 Results and Analysis

Our abbreviated results - Table 2 and Table 4 - showcased the performance of various modern SOTA methods as well as our proposed methods on the two most popular model-dataset combinations [Blalock et al., 2020]. For ResNet-56 on CIFAR-10, our method outperformed every other method on all evaluating criteria, except for PGD; as RAP-ADMM [Ye et al., 2019] outperformed SR-GKP significantly with 61 . 16% v. 44 . 85% . However, it is worth noting that RAP-ADMM does adversarial training on PGD-perturbed data, so it is not surprising that it performs well against the seen type of attack. Unfortunately, it seems like the adversarially trained RAP-ADMM cannot generalize its

20

40

60

80

100

Unpruned

CC

FPGM

L1Norm-B

LRF

RAP-ADMM

SR-GKP (Ours)

44.85

61.15

36.78

43.71

43.55

38.90

42.58

45.17

27.84

28.20

45.16

43.20

29.26

39.64

76.40

75.19

74.80

75.16

75.31

75.75

75.15

94.08

78.37

93.93

92.62

93.60

94.04

93.24

Benign

FGSM

ε

= 0.01

FGSM

ε

= 0.1

PGD

ε

= 8/255,

ε

_step = 2/255, max_iter = 3

defense to other adversarial attacks, even though they are similar in nature. Also, RAP-ADMM has the worst benign performance across all showcased methods, yet its training time is significantly longer due to the need to perturb its training data on the fly constantly.

Table 2 (and similar experiments showcased in Appendix D) may answer one of our research questions: are carefully designed modern structured pruning methods also fragile under adversarial attacks? The answer is an unfortunate 'Yes,' as not only do recent structured pruning methods show serious performance drops under adversarial attacks, such drops are often more severe than their predecessors - which often rely on much naïve designs. Figure 5 as well as the ranked chart Table 3 provide a vivid illustration with LRF [Joo et al., 2021] - a 2021 method - showing one of the weakest adversarially robustness across all evaluated methods.

Upon careful comparison, we noticed that SFP [He et al., 2018] and FPGM [He et al., 2019] two pre-2020 methods - tend to be the best filter pruning methods under the double scrutiny of benign and adversarial tasks. However, this only holds true to smaller scale experiments, as SFP is significantly outperformed by SR-GKP on ResNet-50 on ImageNet for benign accuracy ( 58 . 50% v. 74 . 34% ). Though FPGM tends to perform better on the same task, it is still behind SR-GKP in a general sense (Table 4). We further note ImageNet with adversarial metrics is rarely evaluated under the context of adversarially robust pruning [Chen et al., 2022, Ye et al., 2019]; this is mainly because many of ImageNet sub-classes are closely related in terms of semantics (e.g., maneating shark v. tiger shark) [Ozbulak et al., 2021]. We provide the ImageNet results primarily to show SR-GKP can deliver competitive benign ImageNet performance, where most ImageNet results compared in Table 4 are from methods with strong adversarial performance on CIFAR-10.

Last, although TMI-GKP [Zhong et al., 2022] is optimized for benign tasks, and its procedure does not consider adversarial scenarios, it naturally comes with strong adversarial robustness. We believe this has a lot to do with the increased pruning freedom enabled by the GKP granularity, which further highlights the potential of GKP-based methods outside its achievements in the benign space. Due to the page limitation, we hereby only showcase a much-abbreviated version of our experiments. We strongly encourage our readers to check out our full experiments at Appendix D with a lot more comprehensive coverage on many more dataset-model combinations, as well as ablation studies.

5 Conclusion

Our work studies the area of adversarially robust structured pruning, a topic presenting severe problems but lacking proper recognition or exploration. On the investigation side, we reveal that -just like their naïve predecessors - carefully designed modern structured pruning methods are also fragile under adversarial attacks, yet different pruning methods may yield drastically different adversarial performance while hiding behind similar benign reports. On the solution side, we propose SR-GKP: a simple one-shot GKP method that showcases competitive benign performance with a significant advantage under adversarial attacks against comparable SOTA methods while requiring no extra cost from a pruning procedure perspective.

We believe the overlooked nature of this field is mainly two-fold: the lack of pruning freedom to utilize findings of other fields, and the lack of user-friendly tools and resources for doing adversarial evaluations under a pruning context. We present our take and contribution to both issues by showcasing the capability of GKP-based methods and providing our community with an open-sourced tool and model checkpoints for future studies.

6 Acknowledgments

This research was supported, in part, by NSF Awards OAC-2117439, OAC-2112606, IIS-2224843, and IIS-1900990. This work made use of the High Performance Computing Resource in the Core Facility for Advanced Research Computing at Case Western Reserve University. We give our special thanks to the CWRU HPC team for their timely and professional help and maintenance.

References

Blalock, J. J. Gonzalez Ortiz, J. Frankle, and J. Guttag. What is the state of neural network pruning? Proceedings of machine learning and systems , 2:129-146, 2020.

C.-Y. Chang, Y.-N. Chuang, Z. Jiang, K.-H. Lai, A. Jiang, and N. Zou. Coda: Temporal domain generalization via concept drift simulator, 2023a.

C.-Y. Chang, Y.-N. Chuang, K.-H. Lai, X. Han, X. Hu, and N. Zou. Towards assumption-free bias mitigation, 2023b.

C.-Y. Chang, Y.-N. Chuang, G. Wang, M. Du, and N. Zou. Dispel: Domain generalization via domain-specific liberating, 2023c.

Chen, M. I. Jordan, and M. J. Wainwright. Hopskipjumpattack: A query-efficient decision-based attack. In 2020 IEEE Symposium on Security and Privacy, SP 2020, San Francisco, CA, USA, May 18-21, 2020 , pages 1277-1294. IEEE, 2020. doi: 10.1109/SP40000.2020.00045. URL https://doi.org/10.1109/SP40000.2020.00045 .

Chen, H. Zhang, Z. Zhang, S. Chang, S. Liu, P. Chen, and Z. Wang. Linearity grafting: Relaxed neuron pruning helps certifiable robustness. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , 2022.

Chen, L. Liang, D. Tianyu, Z. Zhu, and I. Zharkov. Otov2: Automatic, generic, user-friendly. In International Conference on Learning Representations , 2023.

Y.-N. Chuang, G. Wang, F. Yang, Z. Liu, X. Cai, M. Du, and X. Hu. Efficient xai techniques: A taxonomic survey. arXiv preprint arXiv:2302.03225 , 2023a.

Y.-N. Chuang, G. Wang, F. Yang, Q. Zhou, P. Tripathi, X. Cai, and X. Hu. Cortx: Contrastive framework for real-time explanation. arXiv preprint arXiv:2303.02794 , 2023b.

Dao, B. Chen, K. Liang, J. Yang, Z. Song, A. Rudra, and C. Ré. Pixelated butterfly: Simple and efficient sparse training for neural network models, 2022a.

Dao, B. Chen, N. S. Sohoni, A. Desai, M. Poli, J. Grogan, A. Liu, A. Rao, A. Rudra, and C. Re. Monarch: Expressive structured matrices for efficient and accurate training. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 4690-4721. PMLR, 17-23 Jul 2022b.

Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314 , 2023.

Frankle and M. Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations , 2019. URL https://openreview.net/ forum?id=rJl-b3RcF7 .

Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh. OPTQ: Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=tcbBPnfwxS .

Gao, F. Huang, W. Cai, and H. Huang. Network pruning via performance maximization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 9270-9280, June 2021.

Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. arXiv 1412.6572 , 12 2014.

Gui, H. Wang, H. Yang, C. Yu, Z. Wang, and J. Liu. Model compression with adversarial robustness: A unified optimization framework. Advances in Neural Information Processing Systems , 32, 2019.

Han, X. Liu, H. Mao, J. Pu, A. Pedram, M. A. Horowitz, and W. J. Dally. Eie: Efficient inference engine on compressed deep neural network. ACM SIGARCH Computer Architecture News , 44(3): 243-254, 2016.

He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016 , pages 770-778. IEEE Computer Society, 2016. doi: 10.1109/CVPR.2016.90.

He and L. Xiao. Structured pruning for deep convolutional neural networks: A survey, 2023.

He, X. Zhang, and J. Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE international conference on computer vision , pages 1389-1397, 2017.

He, G. Kang, X. Dong, Y. Fu, and Y. Yang. Soft filter pruning for accelerating deep convolutional neural networks. In J. Lang, editor, Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden , pages 2234-2240. ijcai.org, 2018. doi: 10.24963/ijcai.2018/309.

He, P. Liu, Z. Wang, Z. Hu, and Y. Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019 , pages 4340-4349. Computer Vision Foundation / IEEE, 2019. doi: 10.1109/CVPR.2019.00447.

Jian, Z. Wang, Y. Wang, J. Dy, and S. Ioannidis. Pruning adversarially robust neural networks without adversarial examples. In 2022 IEEE International Conference on Data Mining (ICDM) , pages 993-998. IEEE, 2022.

Jiang, X. Han, C. Fan, F. Yang, A. Mostafavi, and X. Hu. Generalized demographic parity for group fairness. In International Conference on Learning Representations , 2022.

Joo, E. Yi, S. Baek, and J. Kim. Linearly replaceable filters for deep network channel pruning. In AAAI Conference on Artificial Intelligence , 2021.

E. Korf. A complete anytime algorithm for number partitioning. Artificial Intelligence , 106(2): 181-203, 1998. ISSN 0004-3702.

Krizhevsky. Learning multiple layers of features from tiny images. pages 32-33, 2009.

Lagunas, E. Charlaix, V. Sanh, and A. M. Rush. Block pruning for faster transformers. arXiv preprint arXiv:2109.04838 , 2021.

LeCun, J. Denker, and S. Solla. Optimal brain damage. In D. Touretzky, editor, Advances in Neural Information Processing Systems , volume 2. MorganKaufmann, 1989. URL https://proceedings.neurips.cc/paper_files/paper/1989/ file/6c9882bbac1c7093bd25041881277658-Paper.pdf .

Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf. Pruning filters for efficient convnets. In International Conference on Learning Representations , 2017. URL https://openreview.net/ forum?id=rJqFGTslg .

Li, S. Gu, K. Zhang, L. Van Gool, and R. Timofte. Dhp: Differentiable meta pruning via hypernetworks. In Proceedings of the European Conference on Computer Vision , 2020.

Li, S. Lin, J. Liu, Q. Ye, M. Wang, F. Chao, F. Yang, J. Ma, Q. Tian, and R. Ji. Towards compact cnns via collaborative compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6438-6447, 2021.

Li, T. Chen, L. Li, B. Li, and Z. Wang. Can pruning improve certified robustness of neural networks? arXiv preprint arXiv:2206.07311 , 2022.

Lin, J. Tang, H. Tang, S. Yang, X. Dang, and S. Han. Awq: Activation-aware weight quantization for llm compression and acceleration. arXiv , 2023.

Lin, R. Ji, Y. Wang, Y. Zhang, B. Zhang, Y. Tian, and L. Shao. Hrank: Filter pruning using highrank feature map. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020 , pages 1526-1535. Computer Vision Foundation / IEEE, 2020. doi: 10.1109/CVPR42600.2020.00160.

Lin, R. Ji, C. Yan, B. Zhang, L. Cao, Q. Ye, F. Huang, and D. Doermann. Towards optimal structured cnn pruning via generative adversarial learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2019.

Liu, M. Sun, T. Zhou, G. Huang, and T. Darrell. Rethinking the value of network pruning. In International Conference on Learning Representations , 2019.

Liu, G. Wang, S. Zhong, Z. Xu, D. Zha, R. Tang, Z. Jiang, K. Zhou, V. Chaudhary, S. Xu, et al. Winner-take-all column row sampling for memory efficient adaptation of language model. arXiv preprint arXiv:2305.15265 , 2023.

Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations , 2018. URL https://openreview.net/forum?id=rJzIBfZAb .

Molchanov, S. Tyree, T. Karras, T. Aila, and J. Kautz. Pruning convolutional neural networks for resource efficient inference. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings . OpenReview.net, 2017.

Narshana, C. Murti, and C. Bhattacharyya. Dfpc: Data flow driven pruning of coupled channels without data. In The Eleventh International Conference on Learning Representations , 2022.

Ozbulak, M. Pintor, A. V. Messem, and W. D. Neve. Evaluating adversarial attacks on imagenet: A reality check on misclassification classes. In NeurIPS 2021 Workshop on ImageNet: Past, Present, and Future , 2021. URL https://openreview.net/forum?id=oWk2dULs1x .

J.-H. Park, Y. Kim, J. Kim, J.-Y. Choi, and S. Lee. Dynamic structure pruning for compressing cnns. ArXiv , abs/2303.09736, 2023.

Renda, J. Frankle, and M. Carbin. Comparing rewinding and fine-tuning in neural network pruning. In International Conference on Learning Representations , 2020. URL https://openreview. net/forum?id=S1gSj0NKvB .

Sehwag, S. Wang, P. Mittal, and S. Jana. Hydra: Pruning adversarially robust neural networks. Advances in Neural Information Processing Systems , 33, 2020.

Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In Y. Bengio and Y. LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015. URL http://arxiv.org/abs/1409.1556 .

Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. J. Goodfellow, and R. Fergus. Intriguing properties of neural networks. In Y. Bengio and Y. LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings , 2014. URL http://arxiv.org/abs/1312.6199 .

Tang, Y.-N. Chuang, and X. Hu. Secure your model: A simple but effective key prompt protection mechanism for large language models.

R. Vemparala, N. Fasfous, A. Frickenstein, S. Sarkar, Q. Zhao, S. Kuhn, L. Frickenstein, A. Singh, C. Unger, N. S. Nagaraja, C. Wressnegger, and W. Stechele. Adversarial robust model compression using in-train pruning. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) , pages 66-75, 2021.

Wan, J. Zhao, A. Chadha, N. Peng, and K.-W. Chang. Are personalized stochastic parrots more dangerous? evaluating persona biases in dialogue systems, 2023.

Wang, Y.-N. Chuang, M. Du, F. Yang, Q. Zhou, P. Tripathi, X. Cai, and X. Hu. Accelerating shapley explanation via contributive cooperator selection. In International Conference on Machine Learning , pages 22576-22590. PMLR, 2022.

Wang, Z. Liu, Z. Jiang, N. Liu, N. Zou, and X. Hu. Division: Memory efficient training via dual activation precision. 2023a.

Wang, X. Wu, S. Ge, Z. C. Lipton, and E. P. Xing. Smooth kernels improve adversarial robustness and perceptually-aligned gradients, 2020a. URL https://openreview.net/forum?id= BJerUCEtPB .

Wang, X. Wu, Z. Huang, and E. P. Xing. High-frequency component helps explain the generalization of convolutional neural networks. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 8681-8691, 2020b. doi: 10.1109/CVPR42600.2020.00871.

Wang, C. Qin, Y. Zhang, and Y. Fu. Neural pruning via growing regularization. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id= o966_Is_nPA .

Wang, G. W. Ding, R. Huang, Y. Cao, and Y. C. Lui. Adversarial robustness of pruned neural networks. In Submission to International Conference on Learning Representations (Workshop) , 2018. URL https://openreview.net/forum?id=SJGrAisIz .

Wang, R. Veldhuis, C. Brune, and N. Strisciuglio. What do neural networks learn in image classification? a frequency shortcut perspective, 2023b.

Wang, C. Fu, J. Guo, D. Cai, and X. He. COP: customized deep model compression via regularized correlation-based filter-level pruning. In S. Kraus, editor, Proceedings of the TwentyEighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019 , pages 3785-3791. ijcai.org, 2019a. doi: 10.24963/ijcai.2019/525.

Wang, S. Zhao, M. Chen, J. Hu, D. Cai, and H. Liu. DBP: discrimination based block-level pruning for deep model acceleration. CoRR , abs/1912.10178, 2019b.

Wu, Q. Zhang, and G. Xu. Tiny imagenet challenge. Technical Report , 2017.

Xiao, J. Lin, and S. Han. Offsite-tuning: Transfer learning without full model. arXiv , 2023.

Yang, A. Buluç, and J. D. Owens. Design principles for sparse matrix multiplication on the gpu. In European Conference on Parallel Processing , pages 672-687. Springer, 2018.

Ye, X. Lin, K. Xu, S. Liu, H. Cheng, J.-H. Lambrechts, H. Zhang, A. Zhou, K. Ma, and Y. Wang. Adversarial robustness vs. model compression, or both? pages 111-120, 10 2019. doi: 10.1109/ ICCV.2019.00020.

Yu, A. Li, C. Chen, J. Lai, V. I. Morariu, X. Han, M. Gao, C. Lin, and L. S. Davis. NISP: pruning networks using neuron importance score propagation. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018 , pages 9194-9203. Computer Vision Foundation / IEEE Computer Society, 2018. doi: 10.1109/CVPR.2018.00958.

Zha, Z. P. Bhat, K.-H. Lai, F. Yang, Z. Jiang, S. Zhong, and X. Hu. Data-centric artificial intelligence: A survey, 2023.

Zhang, S. Xu, J. Li, and A. J. X. Guo. Group-based network pruning via nonlinear relationship between convolution filters. Applied Intelligence , Jan. 2022a. doi: 10.1007/s10489-021-02907-0.

Zhang, Y. Liu, Y. Sun, and N. Shah. Graph-less neural networks: Teaching old MLPs new tricks via distillation. In International Conference on Learning Representations , 2022b. URL https://openreview.net/forum?id=4p6_5HBWPCw .

Zhao and C. Wressnegger. Holistic adversarially robust pruning. In Proc. of the International Conference on Learning Representations (ICLR) , May 2023.

Zhong, G. Zhang, N. Huang, and S. Xu. Revisit kernel pruning with lottery regulated grouped convolutions. In International Conference on Learning Representations , 2022. URL https: //openreview.net/forum?id=LdEhiMG9WLO .

A Limitation and Broader Impact

Note although our investigation revealed serious performance issues under adversarial tasks in terms of structured pruning methods, where our proposed method provides sensible mitigation; our findings are still limited to the benign and artificially perturbed input. Though we expect grouped kernel pruning-based methods to deliver good performance under other reasonable evaluating metrics due to the improved pruning freedom, we will leave a more comprehensive investigation against other robustness metrics for future work. We also provide a brief discussion of such related metrics in Appendix B.4.

B Extended Related Work and Discussion

B.1 Structured v.s. Unstructured Pruning

Pruning methods can be roughly divided into structured pruning and unstructured pruning according to the pruning granularity. Specifically, unstructured pruning often means we prune each weight independently. In contrast, structured pruning bundles weights into groups, then prune the whole group instead of the individual weight (e.g., block-wise Lagunas et al. [2021], channel-wise He et al. [2017], group-wise pruning Zhong et al. [2022]).

Unstructured pruning can maintain the model performance better with the same number of parameters. However, unstructured pruned models yield marginal wall-clock time efficiency or even slower than the unpruned model at the low sparsity regime. This is because unstructured pruned matrices need to be stored in sparse matrix format, as the zeros are randomly distributed in these matrices Yang et al. [2018]. Operations executed on sparse matrices (e.g., sparse matrix multiplication, sparse embedding table look-up) are notoriously inefficient on commodity hardware, e.g., GPUs and CPUs, due to the limited data reuse and random memory access Yang et al. [2018], Han et al. [2016].

In contrast, although structured pruning has less flexibility compared to unstructured one, it is much more hardware/library-friendly since structurally pruned matrices can still be stored in the dense matrix format. Thus, operations executed on structured pruned matrices are the same as those in the unpruned model, which are highly optimized. Consequently, the compression provided by structured pruning can often translate into the real wall-clock time speedup upon proper implementation.

B.2 Unstructured Pruning for Adversarial Robustness

In Section 2, we introduced the few structured pruning methods which claim increased adversarial robustness, indicating the lack of presence of adversarially robust structured pruning methods. However, many unstructured pruning literature have explored the possibility of pruned models with improved adversarial robustness, such as Sehwag et al. [2020], Zhao and Wressnegger [2023], Ye et al. [2019], Jian et al. [2022], Li et al. [2022], Gui et al. [2019]. Given it is natural for scholars to adopt insights from unstructured pruning methods into a structured context, we hereby provide this list of unstructured pruning arts for inspiration purposes.

B.3 Learning of High-Frequency Components and its Adversarial Implications

As we consult adversarial-robustness-boosting kernel metrics and operations, we heavily rely on the findings from Wang et al. [2020b], a work that discusses how learning components with different frequencies may affect the adversarial robustness of a CNN, and how some kernel-level metrics like kernel smoothness influence such type of learning. We have elaborated more on this in the Section 3.1 above. Following work can also be referenced when it comes to how component frequency may affect (or be adopted to improve) adversarial robustness, efficiency, and general neural network learning [Wang et al., 2020a, 2023b,a].

B.4 More on ML efficiency and AI safety

The point of studying adversarially robust structured pruning methods is to progress ML efficiency while being aware of its safety. We note that structured pruning is not the sole method that can achieve efficiency benefits, yet adversarial robustness is nowhere near a comprehensive metric for AI

safety. Approaches such as quantization, network architecture search, knowledge distillation, lossy approximation, efficiency-aware architecture tweak, and transformation to hardware-friendly patterns also provide efficiency gains [Frantar et al., 2023, Lin et al., 2023, Wang et al., 2020a, Zhang et al., 2022b, Liu et al., 2023, Xiao et al., 2023, Dettmers et al., 2023, Dao et al., 2022b,a]; yet aspects like explainability, domain generalization, fairness/bias-mitigation, model/data security, and robustness under a noisy environment are also considered important metrics for building reliable, safe, and trustworthy AI solutions [Wang et al., 2022, Wan et al., 2023, Chuang et al., 2023a,b, Chang et al., 2023c,a, Jiang et al., 2022, Chang et al., 2023b, Tang et al., Zha et al., 2023].

C Additional Details on Proposed Method

C.1 Naïve Mix-Up Attempts of Kernel Smoothness

As per Section 3.2, there are generally two stages in a GKP procedure: Filter Grouping and Grouped Kernel Pruning . The following experiments shall attempt to mix-up kernel smoothness criteria into each of such stages, and we can therefore find out which stage is 'friendly' to such mix-up operations and how such operations should look like.

C.1.1 During GKP Filter Grouping (Stage 1)

In TMI-GKP [Zhong et al., 2022], the filter grouping stage is driven by the tickets magnitude increase score, known as TMI-driven Clustering . We utilize it as a baseline to investigate whether our smoothness-aware filter grouping operation Smoothness Snaking (Section 3.2.1 and Figure 3) can maintain the baseline performance and provide improvements. Note, we denote TMI-GKP's grouped kernel pruning scheme (GK Pruning) as Greedy in the following Table 5.

It can be observed that though Smoothness Snaking may yield a slightly lower benign performance than its TMI-driven baseline, it may improve the adversarial robustness when used with the same grouped kernel pruning procedure. We would also note Smoothness Snaking is significantly faster (up to 1,600x) than TMI-driven clustering due to the absence of dimensionality reduction and clustering procedure (see Table 8 for details).

C.1.2 During GKP Pruning Strategies Obtaining/Pruning (Stage 2)

Following the section above, here we investigate the performance of different Grouped Kernel Pruning methods. Our proposed method is denoted as Smooth Beam Greedy (see Figure 4 for details), TMI-GKP's grouped kernel pruning method is denoted as Greedy as above, and Least Smooth represents a vanilla adaptation of smoothness-driven pruning, where the grouped kernels that are least smooth are pruned.

From Table 6, we may tell that Smooth Beam Greedy may significantly improve the adversarial robustness of the pruned networks, yet, it also provides remedies to the decrease in benign performance due to the smoothness snaking operation. It may also be worth noting that the Least Smooth operation is extremely detrimental to almost all tracking metrics, suggesting a vanilla mix-up is inappropriate.

C.1.3 Bonus Investigation: Is Smoothness-aware Filter Pruning Possible?

One main purpose of our paper is to endorse the potential of GKP under adversarial tasks, after [Zhong et al., 2022, Zhang et al., 2022a, Park et al., 2023] showcased the power of GKP in a benign context, thus 'one less reason for filter pruning.' But to make such a claim proper, we will need to investigate whether it is possible to do the same smoothness-aware mix-up with a filter pruning procedure.

Here in Table 7, we use SFP [He et al., 2018] as the baseline, which is considered one of the strongest filter pruning methods on CIFAR-10 [Krizhevsky, 2009]. Then, we try to apply the same rankedbased kernel-smoothness mix-up algorithm as in Equation 2 to find out if such a strong filter pruning baseline can withstand the same mix-up under a filter level.

It can be observed that SFP under such mix-up is completely unusable due to the degradation of benign performance; which suggests the same mix-up strategy, though effective on GKP, is not transferable to filter pruning.

C.2 Speed-Up Analysis

C.2.1 Pruning Procedure Speed Comparison: TMI-Driven Clustering v.s. Smoothness Snaking

In Table 8, we showcased the significant runtime advantage of SR-GKP to TMI-GKP due to the absence of clustering and dimensionality reduction procedures.

C.2.2 Inference Speed Discussion: Standard Convolution v.s. Grouped Convolution

In the realm of structured pruning, there are two types of popular implementations in practice, which may result in different readings on efficiency metrics, such as inference cost reduction (FLOPs/MACs), inference speed-up, and model size compression (number of parameters):

Masking (soft pruning) To deploy a structured binary mask upon the original unpruned model, but clear the gradients of some (or all) pruned components before the weight update step during fine-tuning, therefore theoretically 'structural pruning' the model. In this implementation, the pruned model will not reduce in dimension; thus, no acceleration and compression benefits can be directly observed.

This is a popular implementation - as surveyed in Table 10 'Zero-Masked?' column - because some iterative pruning methods would like the pruned/zeroed components to be reactive during fine-tuning, then pruning another set of components instead. If the pruning granularity is structured (e.g., filter-level), the fine-tuned model can be converted to a hard pruned model by the end of the update as defined below.

Component removal (hard pruning) To remove the pruned components entirely, where the pruned model will reduce in size, providing immediate compression benefits.

Our method, SR-GKP, is implemented as hard pruning. This means that if you inspect the weights of a GKP-pruned network, they are in regular shapes, and there is no zeroed weight to be found - a.k.a. 'densely structured.' Which may lead to a direct reduction in FLOPs/MACs and model size.

In terms of inference acceleration, previous grouped convolution implementations cannot provide speed-up benefits against a standard convolution (even with much fewer MACs/Params) because the standard convolution operator has been extensively optimized, while the grouped one hasn't (e.g., pytorch issue #10229 and #18631 ). But after torch 2.0 , this is not the case at large . For simplicity, here we compare the forward wall-clock between a standard Conv2d of ( C_in = 512, C_out = 512, kernel_size = (3, 3) ) with the exact same Conv2d but with groups = 2 (meaning half of its kernels are structurally removed).

As showcased in Table 9, an organic inference time speed-up of grouped convolution over standard convolution can be observed.

(Note, we emphasized at large above because we indeed can find shapes that are slower with grouped convolution in torch 2.0 . We believe this is very much a framework optimization issue that is beyond the scope of our paper and shall be on the road map of the torch community, as well as other more inference-specific frameworks, granted their already observed improvements.)

C.3 SR-GKP Procedure

C.3.1 Simplifying the Cost Formula from TMI-GKP

For better readability, we hereby follow the notation of TMI-GKP [Zhong et al., 2022] (Equation 4), where we assume V ∗ represents a set of kept grouped kernels provided by a pruning strategy, where g ℓ the convolutional layer in question. The quality V ∗ , under our design, is deemed by:

Where w ( s u , s v ) represents the Euclidean distance between grouped kernels s u and s v , but s i represents the kept grouped kernel that has the least w ( p i , s i ) to a pruned grouped kernel p i . Thus, the former term of this equation calculates the inner distance sum of grouped kernels within strategy V ∗ , and the latter term represents the outer distance between a pruned grouped kernel and its closest kept grouped kernel. Intuitively, we would like the former term to be large, as we would prefer our ideal V ∗ to have great diversity. Following the same idea, we would like to have the latter term small, as we want the kept kernels to cover the representation power of a certain pruned kernel. By using a -β to connect two term, we have V ∗ best = arg max V ∗ ( Score ( V ∗ , g ℓ )) for all V ∗ s obtained in Stage 2 (Figure 4).

Though similar, we differ from TMI-GKP [Zhong et al., 2022] (Equation 4) in two spots: first, we set β = ( V ∗ 2 ) /p num, where p num represent the number of pruned grouped kernels in layer g ℓ , balancing the two terms automatically. Secondly, for the latter term of Equation 3, we only match one pruned kernel p i with one kept kernel s i , instead of several of them. We made these modifications for the main purpose of removing hyperparameters. We kept the grouped kernel strategy selection stage similar to TMI-GKP, but only expanded its search scope using the Smooth Beam Greedy search (Figure 4), because prior investigation suggests this stage is sensitive to smoothness-aware mix-up operations.

D Extended Experiments and Results

D.1 Preliminary

D.1.1 Details of Experiment Setups

For all experiments on VGG-16 and ResNet-32/56/110, we aim to provide all pruning methods an around 300 epochs fine-tuning/retraining budget. Experiments conducted on ResNet-50/101 are budgeted with 100 epochs. We allow compared methods to utilize their own training schedule and vanilla SGD optimizer setup.

SR-GKP utilizes an initial lr of 0.01, with a step size of 100 or 30, depending on if it is on the 300 epoch or 100 epoch schedule. BasicBlock ResNets with VGG use a weight decay of 5e-4 but BottleNeck ResNets use a weight decay of 1e-4 . SR-GKP strictly employs a batch size of 64 for all CIFAR-10 experiments, 128 for Tiny-ImageNet experiments, and 256 for ImageNet-1k experiments.

D.1.2 Details of Evaluation Criteria

We manually ensure all pruned models have a similar MACs/Params reduction from the identical baseline models. Then, following Li et al. [2017], we report all model checkpoints - yielded during the pruning procedure - that may reach the target MACs/Params reduction. For performance (accuracy) evaluation, we test each model against FGSM [Goodfellow et al., 2014] and PGD [Madry et al., 2018] under various settings and intensities.

D.1.3 Details of Compared Methods

We provide a method overview in Table 10 to provide our readers with a more comprehensive understanding of such methods.

Table 10: Details of Compared Methods. Note 'C/F/GK' under 'Type' represents Channel/Filter/Grouped Kernel Pruning. Whether a method requires a 'Special Setup' is determined by whether it follows the most vanilla train - prune - fine-tune procedure. 'Fully Pruned Epoch' reflects if given a 300 fine-tune/retrain budget, what would be the first epoch that meets the target MACs/Params reduction? (a.k.a. checkpoints after this epoch are reportable as 'superscore' readings defined in Section 4.2.) 'Zero-Masked?' reflects whether the pruning method can easily yield a compressed model without masking (a.k.a. hard pruning).

Further, following the background introduction of pruning implementation in Appendix C.2.2, we hereby justify why a post-train one-shot hard pruning method - such as SR-GKP - may 'require no extra (in fact, often less) cost than a standard pruning procedure.'

Under the classic train - prune - fine-tune/retrain paradigm (where our method lives), the training cost is determined at two stages: training the unpruned baseline model and fine-tuning the pruned model.

SR-GKP carries no additional training cost as it is a post-train pruning method, so it won't affect the training cost of the first stage. As for fine-tuning, the SR-GKP one-shot pruned model is already compressed and will go through a vanilla SGD update with no extra operation; we also specifically ensured that all compared methods are using an identical (or similar to the best of the method's ability) epoch budget for fine-tuning as stipulated in Section 4.2 - thus, it requires no additional (in fact, often less) cost on the fine-tuning stage as well.

For illustration convenience, here we walk through two exemplary comparisons:

Compared to TMI-GKP [Zhong et al., 2022] - the founding method of Grouped Kernel Pruning a SR-GKP pruned model shares the exact same format with a TMI-GKP pruned model. Both pruned models will go through an identical fine-tuning procedure. In this case, SR-GKP yields no additional training/fine-tuning cost. Additionally, we'd like to direct your attention to Table 8, which indicates SR-GKP has a much faster execution time than TMI-GKP in terms of pruning procedure, thus being a more 'executionally efficient' method overall.

Compared to SFP [He et al., 2018] - a popular filter pruning method that exhibits strong adversarial performance - SFP produces a zero-masked model while SR-GKP's pruned model is hard-pruned. Such zeroed weights will reactivate during its retraining/fine-tuning, where the algorithm decides which filter to zero out between epochs. In this case, the one-shot pruned SR-GKP induces significantly less cost as a) it does not require weight updates to the full model, but only on a compressed one; and b) it does not perform extra pruning operations during the fine-tuning/retraining stage.

D.2 Additional Experiments

D.2.1 ResNet-32/56/110 on CIFAR-10 with pruning rate ≈ 43 . 75% (in addition to Table 2)

Please refer to Table 11 with Figure 6, Table 12 with Figure 7, and Table 13 with Figure 8 for details. We also provide Table 3 in a ranked chart format for an easy digest of the three aforementioned tables.

20

40

60

80

100

Unpruned

CC

DHP

FPGM

L1Norm-A

L1Norm-B

LRF

NPPM

OTOv2 (FS)

OTOv2 (PT)

RAP-ADMM

SFP

TMI-GKP

SR-GKP (Ours)

33.36

32.05

30.18

57.66

31.96

27.15

30.51

30.07

35.20

30.70

32.57

26.80

30.75

35.40

31.32

30.51

30.34

25.80

28.01

27.97

27.54

34.62

32.50

30.99

27.23

25.84

31.35

71.52

70.61

69.25

72.27

70.63

66.36

70.45

70.22

71.59

70.00

70.84

68.14

75.75

71.93

92.97

92.99

91.94

74.88

92.14

90.97

93.13

93.04

91.58

91.45

92.41

92.26

93.01

92.80

Benign

FGSM

ε

= 0.01

FGSM

ε

= 0.1

PGD

ε

= 8/255,

ε

_step = 2/255, max_iter = 3

20

40

60

80

100

Unpruned

CC

DHP

FPGM

Greg-1

Greg-2

HRank

L1Norm-A

L1Norm-B

LRF

NPPM

OTOv2 (FS)

OTOv2 (PT) RAP-ADMM

SFP

TMI-GKP SR-GKP (Ours)

44.85

44.51

44.10

54.22

41.58

25.52

38.13

36.78

43.71

42.74

32.51

43.71

42.22

44.66

33.64

38.90

42.58

45.17

44.09

43.83

45.35

40.46

23.63

31.18

28.20

45.16

48.19

39.30

37.11

44.96

70.42

29.26

39.64

76.40

75.88

75.63

61.08

74.29

65.91

75.50

74.80

75.16

74.30

72.59

75.40

74.78

76.28

70.66

75.75

75.15

94.08

93.95

93.15

61.54

93.02

91.57

93.55

93.93

92.62

92.44

92.27

93.74

93.30

93.60

92.42

94.04

93.24

Benign

FGSM

ε

= 0.01

FGSM

ε

= 0.1

PGD

ε

= 8/255,

ε

_step = 2/255, max_iter = 3

20

40

60

80

100

Unpruned

CC

DHP

FPGM

HRank

L1Norm-A

L1Norm-B

LRF

NPPM

OTOv2 (FS)

OTOv2 (PT)

RAP-ADMM

SFP

TMI-GKP

SR-GKP (Ours)

50.01

49.00

52.57

63.11

48.46

34.74

42.14

42.67

47.05

45.49

35.72

52.82

27.19

39.78

50.85

49.09

47.15

53.62

29.26

47.96

23.94

33.71

30.67

41.92

45.48

54.15

26.64

29.00

48.83

79.14

78.02

78.75

78.36

77.25

71.43

76.56

76.93

76.10

75.83

73.70

79.78

68.86

75.88

78.56

95.00

94.90

94.44

81.43

93.99

91.58

94.16

94.49

92.96

92.75

92.26

94.18

92.53

94.31

94.26

Benign

FGSM

ε

= 0.01

FGSM

ε

= 0.1

PGD

ε

= 8/255,

ε

_step = 2/255, max_iter = 3

D.2.2 ResNet-32/56/110 on CIFAR-10 with pruning rate ≈ 62 . 5% )

Please refer to Table 15, 16, and 17. We also provide Table 14 in a ranked chart format for an easy digest of the three aforementioned tables.

D.2.3 VGG-16 on CIFAR-10

Please refer to Table 18 with Figure 9 for details.

20

40

60

80

100

Unpruned

CC

HRank

L1Norm

TMI-GKP

SR-GKP (Ours)

66.62

66.06

57.03

56.50

62.85

64.99

59.03

55.30

41.81

34.76

48.97

50.51

83.85

83.48

81.53

81.58

83.38

83.73

93.95

94.07

92.88

93.57

94.14

93.94

Benign

FGSM

ε

= 0.01

FGSM

ε

= 0.1

PGD

ε

= 8/255,

ε

_step = 2/255, max_iter = 3

D.2.4 ResNet-56/101 on Tiny-ImageNet

Please refer to Table 19 for details.

Table 19: Full experiments of ResNet-56/101 on Tiny-Imagenet. All pruning methods are performed on the same baseline model. 'Best (a)' represents the performance of a model checkpoint that meets the showcased MACs/Params reduction and performs best against criterion (a).

D.2.5 Comparison with Channel Pruned HARP

HARP by Zhao and Wressnegger [2023] is one of a few adversarially robust structured pruning methods available for comparison, as it has a channel pruning version offered in its appendix (HARP-CP). However, HARP utilizes several different training schedules than other methods, and we therefore cannot compare it with other methods using a unified baseline (as in Table 18). To fulfill the comparison, we opt to align SR-GKP with HARP under different potential training schedule settings offered in HARP literature or our experiment setup. Please refer to Table 20 for details.

We believe it is safe to conclude that SR-GKP showcased clear dominance over the HARP-CP.

Although one network-dataset combination is not a comprehensive evaluation, Figure 13 of HARP suggests HARP-CP is extremely close to RAP-ADMM [Ye et al., 2019] - a method we already compared - in terms of both benign and adversarial acc; and from Figure 6, 7, 8 as well as Table 3, we may conclude that RAP-ADMM performs vastly below SR-GKP on various metrics. We find it interesting that HARP and RAP-ADMM - two methods utilizing adversarial training - still perform worse than SR-GKP on most adversarial tasks, though SR-GKP only sees benign inputs; indicating the effectiveness of SR-GKP. We suspect this is mainly due to the fact HARP and RAP-ADMM are not structured pruning-focused methods.

D.3 Ablation Studies

Please refer to Table 21, Table 22, and Table 23 for ablation studies on hyperparameter α in Equation 2. We denote it as CSB as it is in essence a 'cost-smoothness balancer.' It can be observed that a relatively high CSB - meaning giving more bias to the distance-based cost metrics - may yield better performance.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors base their work on top of Grouped Kernel Pruning (GKP) to alleviate the issue of adversarial robustness when pruning CNNs structurally. The authors propose Smoothly Robust Grouped Kernel Pruning (SR-GKP) that alters how the filter grouping occurs in GKP and how the grouped kernels are selected. The authors claim that this method is one-shot and data-free. Post one-shot pruning, the authors fine-tune the pruned models on the original training set. The authors claim that this method has no extra cost as opposed to other pruning methods to gain adversarial robustness.\n\nSTRENGTHS:\nS1. The problem considered is important\n\nS2. The method is clever and technically sound.\n\nS3. Empirical evidence shows merit in using this approach.\n\nWEAKNESSES:\nW1. The baselines for the ImageNet experiments are weak (Table 3). Some suggestions for incorporating recent baselines: [a], [b], [c] (GitHub repos are available).\n\nW2. Lack of ablation studies w.r.t. $\\alpha$\n\nW3. Lack of adherence to the NeurIPS 2023 template (Figure 2)\n\nW4. Writing needs to be improved.\nSome typos: \n- Line 193. find\n- Line 222 and 243. (Appendix C) [brackets]\n- Line 248. Did you mean tamper?\n- Line 310. (and similar experiments showcased in ??)\n\n\n\n\n\n\n\n\n[a] Only Train Once: A One-Shot Neural Network Training And Pruning Framework. Chen et al. 2021\n\n[b] Neural Pruning via Growing Regularization. Wang et al. 2021\n\n[c] Structural Pruning via Latency-Saliency Knapsack. Shen et al. 2022\n\n[d] DFPC: Data flow driven pruning of coupled channels without data. Narshana et al. 2023\n\nQUESTIONS:\nQ1. As stated on lines 276-277 you state that you fine-tune the model. However, you state that your method is data-free. Why?\n\nQ2. When discarding grouped kernels (Stage 2), how do you decide when to stop pruning a layer?\n\nQ3. How do you decide an appropriate value of $\\alpha$? Does the optimal $\\alpha$ value remain the same across datasets and architectures?\n\nQ4. When pruning a layer (structured), you essentially reduce the output tensor size. This affects the kernels in the following layers (they also need to be pruned). Do you include them to decide whether a grouped kernel is to be pruned?\n\nLIMITATIONS:\nThe method makes progress in alleviating the issue of adversarial robustness. However, the pruned models are still prone to suffer from adversarial inputs, as shown in the empirical results.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the adversarial robustness of modern structured pruning methods in deep neural networks. The paper insists that while naively pruned networks show poor performance under simple adversarial attacks, there is a lack of thorough investigation into the adversarial performance of carefully designed modern structured pruning methods. The authors aim to address this gap and provide remedies named Smoothly Robust Grouped Kernel Pruning (SR-GKP), a one-shot post-training pruning method for improving adversarial robustness without incurring significant additional costs.\n\nSTRENGTHS:\n1. Clarity: Clear writing. Easy to understand paper.\n2. The paper is technically sound.\n\nWEAKNESSES:\n1. The novelty of the method is not high. The method is highly dependent on TMI-GKP [Ref_1]. The method looks very similar to previous TMI-GKP. Even if the paper has a method with smoothness, it still sounds like one of the variants of TMI-GKP under the same umbrella.\n2. The adversarial performance has an incremental improvement compared to original TMI-GKP in Table 2. Both best benign performance and best(a) performance have a small improvement (less than 2%), which is not a significant contribution.\n3. The paper is based on empirical study. To increase the value of the paper, it would be better if the authors can provide a theoretical analysis on why the proposed method is helpful to both benign accuracy and adversarial accuracy (especially, why benign accuracy is increased?).\n4. Typos and mistakes make readers confused. \nTable 2 caption: ResNet-56 on CIFAR-10. All pruning are performed on the same baseline model. “Best (a)” represents the performance of a model checkpoint that meats\nLine 263: So Smoothness(GKbranch) would suggest this particular GKbranch has a greater smoothness value (a.k.a. “less smooth”) then all other GKbranch candidates in considerations.\nLine 314: Figure ?? provides a vivid illustration with LRF — a 2021 method— showing the weakest adversarially robustness across the plotted methods\n\n---\n\n**Post-Rebuttals**\n\nI would like to thank authors for their detailed response and for their effort in improving paper according to reviewer’s suggestions. After reading the wall-clock runtime comparison in Table 7 pointed out by authors in supplementary material, I respectfully agree with the author’s opinion on the novelty of the paper. The authors have adequately addressed most of my concerns regarding clarity except for the theoretical analysis. I have updated my rating to accept for this paper through author-reviewer discussion.\n\nQUESTIONS:\n1. What is the best(a) and best(b) in Table 2? No explanations for this.\n2. Why does the RAP-ADMM performance against FGSM in Figure 5 lower than the unpruned version?\n\nLIMITATIONS:\nYes, the authors have addressed the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper proposes a structured pruning method that introduces kernel smoothness into grouped kernel pruning to obtain adversarial robustness. Without additional overhead (compared to standard pruning methods), the method can obtain considerable accuracy and adversarial robustness simultaneously. The authors compare with popular structured pruning methods, and the results validates the effectiveness of the proposed method.

STRENGTHS:
- The motivation is reasonable: introducing kernel metrics related to adversarial robustness into the pruning process to obtain robustness.

- The authors designed sufficient ablation experiments to validate that the proposed 'Smoothness Snaking' Filter Grouping strategy and 'Smooth Beam Greedy' pruning strategy are more effective than other strategies.

WEAKNESSES:
- The novelty is limited: kernel smoothness and grouped kernel pruning have been studied in some works. The core contribution of the paper seems to be a direct combination of two ideas. 

- Some baselines are omitted in the experiments, such as HARP, which can be extended to structured pruning for robustness.

  [1] Holistic adversarially robust pruning. 

- Lack of performance comparison under varying pruning ratio.

QUESTIONS:
- The author claims the method is one-shot. However, the one-shot pruning typically refers to the pruning method without the need of fine-tuning, such as OTO [2]. The proposed method seems still requiring fine-tuning afterwards.

  [2] Only train once: a one-shot neural network training and pruning framework.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-05

comments: SUMMARY:
The contribution of the paper is two fold. First, it presents a large scale benchmarking study of the adversarial robustness of various filter pruning methods. It finds that such methods often lead to a large degradation in the adversarial robustness of the pruned models, though they preserve benign accuracy. The paper then proposes an alternative form of pruning to solve this issue. The proposed method borrows ideas from grouped kernel pruning and kernel smoothness to prune out groups of kernels while making sure that the model does not overfit to adversarially vulnerable high frequency components of the image. This leads to better adversarial performance than baselines.

STRENGTHS:
1. The empirical evaluation of the paper is solid.
2. The paper proposes a simple hypothesis of pruned models fitting to HFCs to explain their poor robustness, and unit-tests it well. 
3. The proposed method effectively combines two existing works, and provides solid gains.

WEAKNESSES:
1. The writing of the paper can be improved. There are multiple typos, and the paper could benefit from formally defining terms and notations earlier on. A more thorough discussion of the GKP and kernel smoothing techniques would also help.
2. The claim of no additional training cost over standard pruning methods is not verified.

QUESTIONS:
1. Can the authors provide information about the training cost

LIMITATIONS:
The limitations are sufficiently discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07