Sparsity Winning Twice: Better Robust Generalization from More Efficient Training
Authors: Tianlong Chen
Zhenyu Zhang
pengjun wang
Santosh Balachandra
Haoyu Ma
Zehao Wang
Zhangyang Wang
Venue: ICLR 2022 Poster (2022)

Abstract: Recent studies demonstrate the deep networks, even robustified by the state-of-the-art adversarial training (AT), still suffer from large robust generalization gaps, in addition to the much more expensive training costs than standard training. In this paper, we investigate this intriguing problem from a new perspective, i.e., $\textit{injecting appropriate forms of sparsity}$ during adversarial training. We introduce two alternatives for sparse adversarial training: (i) $\textit{static sparsity}$, by leveraging recent results from the lottery ticket hypothesis to identify critical sparse subnetworks arising from the early training; (ii) $\textit{dynamic sparsity}$, by allowing the sparse subnetwork to adaptively adjust its connectivity pattern (while sticking to the same sparsity ratio) throughout training. We find both static and dynamic sparse methods to yield win-win: substantially shrinking the robust generalization gap and alleviating the robust overfitting, meanwhile significantly saving training and inference FLOPs. Extensive experiments validate our proposals with multiple network architectures on diverse datasets, including CIFAR-10/100 and Tiny-ImageNet. For example, our methods reduce robust generalization gap and overfitting by $34.44\%$ and $4.02\%$, with comparable robust/standard accuracy boosts and $87.83\%$/$87.82\%$ training/inference FLOPs savings on CIFAR-100 with ResNet-18. Besides, our approaches can be organically combined with existing regularizers, establishing new state-of-the-art results in AT. All codes are included.

Published as a conference paper at ICLR 2022
SPARSITY WINNING TWICE: BETTER ROBUST GEN- ERALIZATION FROM MORE EFFICIENT TRAINING
Tianlong Chen' Zhenyu Zhang", Pengjun Wang", Santosh Balachandra",
Haoyu Ma". Zehao Wang', Zhangyang Wang' 'University of Texas at Austin, University of Science and Technology of China, *University of California, Irvine
ftianleng chen, santoshb, at wal.aaaawaaeeeaassas. {azy] 9969, wpj520, wangze @ma uste. wat.eedd.eee.......oo..oo.o...e. edu. haoyum3@uci, edu
ABSTRACT
Recent studies demonstrate that deep networks, even robustified by the state-of- the-art adversarial training (AT), still suffer from large robust generalization gaps, in addition to the much more expensive training costs than standard training. In this paper, we investigate this intriguing problem from new perspective, i.e., in- jecting appropriate forms of sparsity during adversarial training. We introduce alternatives for sparse adversarial training: static sparsity, by leveraging recent results from the lottery ticket hypothesis to identify critical sparse sub- networks arising from the early training; (ii) dynamic sparsity, by allowing the sparse subnetwork to adaptively adjust its connectivity pattern (while sticking to the same sparsity ratio) throughout training. We find both static and dynamic sparse methods to yield win-win: substantially shrinking the robust generalization gap and alleviating the robust overfitting, meanwhile significantly saving training and inference FLOPs. Extensive experiments validate our proposals with multi- ple network architectures on diverse datasets, including CIFAR- 10/100 and Tiny- ImageNet. For example, our methods reduce robust generalization gap and overfit- ting by 34.44% and 4.02%, with comparable robust/standard accuracy boosts and 87.8 87.83%/8.88%% training/inference FLOOPs savings CIFAR-100 with ResNet- 18. Besides, our approaches can be organically combined with existing regulariz- ers, establishing new state-of-the-art results in AT. Codes are available in tps: //github. wwwp://iA-rrrro/oaaaaaaaaaeeaeee
Win
ietaaliiaiionn inn
INTRODUCTION
Rohae Train Accurae
0.0.8
0.6
Deep neural networks (DNNs) are notoriously vul- nerable to maliciously crafted adversarial attacks. To conquer this fragility, numerous adversarial defense
0.4
mechanisms are proposed to establish robust neu-
100
150
ral networks (Schmidt et al.) 2018 Sun al. 2019
Nakkiran 2019 Raghunathan et al. 2019 Hu ct al. 2019 Chen et al. 2020c. 2021e; liang al. 2020).
Robut Tox Accuracy
0.5
Among them, adversarial training (AT) based meth-
ods (Madry al.) 2017; Zhang et al. 2019 have maintained the state-of-the-ar robustness. However, the AT training process usually comes with order-of- magnitude higher computational costs than standard training, since multiple attack iterations are needed construct strong adversarial examples (Madry et al. 20186). Moreover, AT' was recently revealed to incur severe robust generalization gaps (Rice al., (2020), between its training and testing accuracies, as shown in Figure I and to require significantly more training samples (Schmidt et al. 2018) to generalize robustly.
Dense
0.80
0.40
0.90
100
Epochs
Figure Robust train test accuracy (Top Bot- tom) CIFAR-10 with ResNet-18 across var- ious sparsity levels from 0% (Dense) to 90%. The dash-dot and solid lines represent the vanilla PGD-AT dense baseline and our sparse propos- als, respectively. As the sparsity increases, the robust generalization gap between training and testing accuracy substantially narrowed.
"Equal Contribution.

---

Published as a conference paper at ICLR 2022
In response to those challenges, /Schmidt (Sahiin et al. 20188) Lee et al. (2020); Song et al. (2019) investigate
the possibility of improving generalization by leveraging advanced data augmentation techniques,
which further amplifies the training cost of AT. Recent studies (Rice al. 2020 Chen et al. 2021e)
found that early stopping, or several amooothness/laaness-aarr regularizations CCen et al.
2021e
Stutz et al. 202T [Singla et al. )02T), can bring effective mitigation.
In this paper, a new perspective has been explored to tackle the above challenges by enforcing ap- propriate sparsity patterns during AT. The connection between robust generalization and sparsity is mainly inspired by two facts. On one hand, sparsity can effectively regularize the learning of over- wver-paraeterized neural networks, hence potentially benefiting both standard and robust general- ization (Balda et al.) (019). As demonstrated in Figure [I with the increase of sparsity levels, the robust generalization gap is indeed substantially shrunk while the robust overfitting is alleviated. On the other hand, one key design philosophy that facilitates this consideration is the lottery ticket hypothesis (LTHH) (Frankle & Carbin 2019). The LTH advocates the existence of highly sparse and separately trainable subnetworks (a. winning tickets), which can be trained from the original initialization to match or even surpass the corresponding dense networks' test accuracies. These facts point out promising direction that utilizing proper sparsity is capable of boosting robust gen- eralization while maintaining competitive standard robust accuracy.
Although sparsity is beneficial, the current methods (Frankle & Carbin 2019 Frankle et al., 2020 Renda et al. 2020) often empirically locate sparse critical subnetworks by Iterative Magnitude Prun- ing (IMP). It demands excessive computational cost even for standard training due to the iterative train-prune-retrain process. Recently, You et al. (2020) demonstrated that these intriguing subnet- works can be identified at the very early training stage using one-shot pruning, which they term as Early Bird (EB) tickets. We show the phenomenon also exists in the adversarial training scheme. More importantly, we take one leap further to reveal that even in adversarial training, EB tickets can be drawn from cheap standard training stage, while still achieving solid robustness. In other words, the Early Bird is also a Robust Bird that yields an attractive win-win of efficiency and robustness we name this finding as Robust Bird (RB) tickets.
Furthermore, we investigate the role of sparsity in a scene where the sparse connections of subnet- works change on the fly. Specifically, we initialize subnetwork with random sparse connectivity and then optimize its weights and sparse typologies simultaneously, while sticking to the fixed small parameter budget. This training pipeline, called Flying Bird (FB), is motivated by the latest sparse training approaches (Evci et al.) (020b) to further reduce robust generalization gap in AT, while ensuring low training costs. Moreover, enhanced algorithm, i.e., Flying Bind+, is proposed to dynamically adjust the network capacity (or sparsity) to pursue superior robust generalization, at few extra prices of training efficiency. Our contributions can be summarized as follows:
We perform a thorough investigation to reveal that introducing appropriate sparsity into AT is appealing win-win, specifically: (1) substantially alleviating the robust general- ization gap; (2) maintaining comparable or even better standard/robust accuracies; and (3) enhancing the AT efficiency by training only compact subnetworks.
We explore two alternatives for sparse adversarial training: (i) the Robust Bird (RB) training that leverages static sparsity, by mining the critical sparse subnetwork at the early training stage, and using only the cheapest standard training; (ii) the Flying Bird (FB) training that allows for dynamic sparsity, which jointly optimizes both network weights and their sparse connectivity during AT, while sticking to the same sparsity level. We also discuss FB variant called Flying Bird+ that adaptively adjusts the sparsity level on demand during AT. with
Extensive experiments are conducted CIFAR-10. CIFAR-100, and diverse network architectures. Specifically, our proposals obtain 80.16% ~ 87.83% train- ing FLOPs and 80.16% ~ 87.83% inference FLOPs savings, shrink robust generalization from 28.00 28.00% ~ 63. 68% to 4.43% ~ 34.44%, and boost the robust accuracy by up to 0.60% and the standard accuracy by up to 0.90%. across multiple datasets and architec- tures. Meanwhile, combining our sparse adversarial training frameworks with existing regularizations establishes the new state-of-the-aar results.
2 RELATED WORK
A Adversarial training and robust generrulizationlovefiiinnng Deep neural networks present vul- nerability to imperceivable adversarial perturbations. To deal with this drawback, numerous defense
2

---

Published as a conference paper at ICLR 2022
approaches have been proposed (Goodfellow et al. [015 Kurakin et al. 2016 Madry et al., 201 8a).
Although many methods (Liao et al. 2018]] 8| Guo et al. [2018a]
2017 Dziugaite et al. 2016 Xu et al.
Dhillon et al. 2018a| Xie et al. 2018 Jiang et al. 2020)
were later found to result from obfuscated
gradients (Athalye et al. 2018), adversarial training (AT)
Madry et al.
2018a). together with some
of its variants (Zhang
2018), remains as one of the 2019 Moshach et al. 2018 Dong al.
most effective yet costly approaches.
A pitfall of AT, i.e., the poor robust generalization, was spotted recently. Schmidt et al. (2018) showed that AT intrinsically demands a larger sample complexity to identify well-generalizable ro- bust solutions. Therefore, data augmentation (Lee et al. 2020; Song et al. 2019) is an effective rem- edy. |tutz et al.] (202T): Singlaa et al. (202I) related robust generalization gap to curvature/fltnees of loss landscapes. They introduced weight perturbing approaches and smooth activation functions to reshape the loss geometry and boost robust generalization ability. Meanwhile, the robust overfit- ting (Rice et a 2020) in AT usually happens with or as a result of inferior generalization. Previous studies (Rice et al. 2020 Chen et al. 2021e) demonstrated that conventional regularization--aase methods (e.g., weight decay and simple data augmentation) can not alleviate robust overfitting. Then, numerous advanced algorithms (Zhang al., 2020 2021b Zhou et al. 2021 Bunk et al., 2021 Chen et al. 2021a] Dong et al. [021]] Zi et al. 2021 Tack et al. 2021; Zhang et al. [202 la) arose the last half year tackle the overfitting, using data manipulation, smoothened training.
and else. Those methods work orthogonally to our proposal as evidenced in Section 4
Another group of related literature lies in the field of sparse robust networks (Guo al. 2018h) These works either treat model compression as a defense mechanism (Wang et al. 2018 Gao al. 2017: Dhillon et al.) 2018b) or pursue robust and efficient sub-models that can be deployed in resource-limitee platforms (Gui et al. 2019: Ye et al.) 2019; Sehwag et al. 2019). Compared to those ffference-focused methods, our goal fundamentally different: injecting sparsity during training to reduce the robust generalization gap while improving training efficiency.
Static pruning and dynamic sparse training. Pruning (LeCun et al., 1990 Han et al. 2015a) serves as a powerful technique to eliminate the weight redundancy in wee-pparameterized DNNs, which aims to obtain storage and computational savings with almost undamaged performance. can roughly divided into two categories based on how to generate sparse patterns: (i) static prun- ing. It removes parameters (Han et al., 2015a LeCun et al. 1990, Han et al., 20135) or sub- structures (Liu al. 2017 Zhou et al. 2016 He et al.] 2017) based on optimized importance scores (Zhang et al. 2018: He et al.] (017)) 7) or some heuristics like weight magnitude (Han et al., 2015ap. gradient (Molchanov et al.] (019). hessian (LeCun et al. 1990) statistics. The discarded elements usually will not participate in the next round of training or pruning. Static pruning can be flexibly applied prior to training, such as SNIP (Lee et al., 2019). GraSP (Wang et al. 2020) and SynFlow (Tanaka et al., 2020); during training (Zhang et al. 2018 He et al. 2017)) and post train- ing (Han et al. 2005aa) for different trade-off between training cost and pruned models' quality. (ii) dynamic sparse training. It updates model parameters and sparse connectivities at the same time,
starting from a randomly sparsified subnetwork (Molchanov et al.
2017). During the training, the removed elements have chances to be grown back if they potentially benefit to predictions. Among [2019] Mostafa & Wang, 2019
the huge family of sparse training (Mocanu et al. |0166] Evci et al.
Liu et al. a Dettmers & Zettlemoyer 2019 Jayakumar et al. 2021 Raihan & Aamodt 2020).
the recent methods Evci et al. (2020a) Liu et al. (0021) lead to the ofate-of-the-art performance.
A special case of static pruning, Lottery tickets hypothesis (LTH) (Frankle & Carbin 2019). demon- strates the existence of sparse subnetworks in DNNs, which are capable of training in isolation and reach a comparable performance of their dense counterpart. The LTH indicates the great potential to
train a sparse network from scratch without sacrificing expressiveness and has recently drawn lots of
attention from diverse fields (Chen et al.) 2020b a. 2021g b 2022 Ding et al. [022 Gan al.
2021 beyond image recognition (Zhang et al. 2021d Frankle et al. 2020 Redman et al. 2021).
3
METHODOLOGY
3.1 PRELIMINARIES
Adversarial training (AT). As one of the widely adopted defense mechanisms, adversarial train- ing (Madry et al.) 2018b) effectively tackles the vulnerability to maliciously crafted adversarial
samples. As formulated in Equation [ AT (specifically PGD-AT) replaces the original empirical risk minimization into min-max optimization problem:
3

---

Published as a conference paper at ICLR 2022
Stundard Training
PGD Adversarial Training
Noo- robust Weights
Robust Weights
Pruning
Robusi Bird
Flying Bind
(+)
Ideniiyy Robust Bird
PGD-AI for Robust Bird
Inital Sporse Model
Pruning Grow
Pruning & Grow
Jointly Identify and PGD-AI for Flying Bird
Spurse
Overliting Underfiting
Overfiting Underfitting X
Pruning Less Grow More
Pruning & Grow Lees
Jointly Identiyy
Flying Dynomically Adjust Capacity Optimization
Overview of our proposed training frameworks including Robust Bird (RB), Flying Bird (FB), and Flying Bird (FB+). The length of cycles roughly indicates the number of training epochs.
Figure 2:
min E (E ()))
min (a.)) max L( 8;0).u) |6
9
where f(x; 0) is network with parameters 0. Input data r and its associated label from training set D are used to first generate adversarial perturbations ó and then minimize the empirical classification loss L. To meet the imperceptible requirement, the Ep norm of 5 is constrained by a small constant e. Projected Gradient Descent (PGD). i.e., 8++1 = projp [8 + a sgn V(f(((a + 8' ; f). y))l is usually utilized to produce the adversarial perturbations with step size a. which works in iterative manner leveraging the local first order information about the network (Madry et al. 2018b).
Sparse subnetworks. Following the routine notations in Frankle & Carbin 2019). f(a; @ 0) donates a sparse subnetwork with binary pruning mask m E 11 where @ is the element- wise product. Intuitively, is a copy of dense network f [x; 0) with portion of fixed zero weights.
3.2 ROBUST BIRD FOR ADVERSARIAL TRAINING
Introducing Robust Bird. The primary goal of Robust Bird is to find high-quality sparse subnet- work efficiently. shown in Figure 2 it locates subnetworks quickly by detecting critical network structures arising in the early training, which later can be robustified with much less computation.
Specifically, for each epoch during training, Robust Bird creates a sparsity mask m, by "masking out" the p% towest-magnitude weights; then, Robust Bird tracks the corresponding mask dynamics. The key observation behind Robust Bird that the sparsity mask m, does not change drastically beyond the early epochs of training (You al. 2020) because high-level network connectivity pat- terns are learned during the initial stages (Achille et al., 2019). This indicates that (i) winning tickets emerge at very early training stage, and (ii) that they can be identified efficiently.
Robust Bird exploits this observation by comparing the Hamming distance between sparsity masks found in consecutive epochs. For each epoch, the last I sparsity masks stored. If all the stored masks are sufficiently close to each other, then the sparsity masks are not changing drastically over time and network connectivity patterns have emerged; thus, Robust Bird ticket (RB ticket) is drawn. A detailed algorithmic implementation is provided in Algorithm of Appendix Al This is the RB ticket used in the second stage of adversarial training.
4

---

Published as a conference paper at ICLR 2022
Random Pruning
Flying Bird+
Dense
Figure 3: Visualization of loss contours and training trajectories. We compare the dense network, randomly pruned sparse networks, and flying bind+ 90% sparsity from ResNet-18 robustified on CIFAR-10.
Rationale of Robust Bird. Recent studies (Zhag et al. 2021c) present theoretical analyses that identified sparse winning tickets enlarge the convex region near the good local minima, leading to improved generalization. Our work also shows related investigation in Figure A9 that, compared with dense models and random pruned subnetworks, RB tickets found by the standard training have much flatter loss landscapes, serving high-quality starting point for further robustification. This occurs because flatness of the loss surface is often believed to indicate the standard generalization. Similarly, advocated by Wu al. ( Hein & Andriushchenko (2077) flatter adversarial loss landscape also effectively shrinks the robustness generalization gap. This "flatness preference" of adversarial robustness has been revealed by numerous empirical defense mechanisms, including Hessian/curvature- based regularization (Moosavi-Deztoon et al., 2019), learned weight and logits smoothening (Chen al. (2021e), gradient magnitude penalty (Wang & Zhang 22019) smoothening with random noise (Liu et al.) 2018), or entropy regularization (Jagatap et al. 2020).
These observations make the main cornerstone for our proposal and provide possible interpretations to the surprising finding that the RB tickets pruned from a non-robust model can be used for obtain- ing well-geneealiizbble robust models the followed robustification. Furthermore, unlike previous costly flatness regularizers MMoosavi-Deztood et al. 2019), our methods not only offer a flatter starting point but also obtain substantial computational savings due to the reduced model size.
3.3 FLYING BIRD FOR ADVERSARIAL TRAINING
Introducing Flying Bird(+). Since sparse subnetworks from static pruning are unable regret for removed elements, they may be too aggressive to capture the pivotal structural patterns. Thus, we introduce Flying Bird (FB) to conduct thorough exploration of dynamic sparsity, which allows pruned parameters to be grown back and engages in the next round of training pruning, demon- strated in Figure Specifically, it starts from sparse subnetwork f(x; m O0) with a random binary mask m, and then jointly optimize model parameters and sparse connectivities simultaneously. In other words, the subnetwork's ' typologies are "on the fly", decided dynamically based on current training status. Specifically, we update Flying Bird's ' sparse connectivity every Át epochs of adver- sarial training, which consists of two continually applied operations: pruning and growing. For the pruning step, p% of model weights with the lowest magnitude will be eliminated, while 9% weights with the largest gradient will be added back in the growth step. Note that newly added connections are not activated the last sparse topology, and are initialized to zero since it establishes better per- formance as indicated in (Evci et al. 2020a Liu et al. 2021b). Flying Bird maintains the sparsity ratio unchanged during the full training by keeping both pruning and growing ratio p%,, 9% equal 1% that decays with a cosine annealing schedule.
We further propose Flying Bird+, an enhanced variant of FB, capable of adaptively adjusting the sparsity and learning the right parameterization level "on demand" during training, as shown in Figure To be specific, we first record the robust generalization gap and robust validation loss at each training epoch. An increasing generalization gap of the later training stage indicates risk of overfitting, while a plateau validation loss implies underfitting. Hence, we then analyze the fitting status according the upward/downward trend of those measurements. If most epochs (e.g., more than out the past 5 epochs in our case) tend to see enlarged robust generalization gaps, we raise the pruning ratio p% to further trim down the network capacity. Similarly, if the majority of epochs present unchanged validation loss, we will increase the growing ratio 4% to enrich the subnetwork capacity. Detailed procedures summarized in Algorithm of Appendix AT
Rationale of Flying Bird(+). As demonstrated in Evei et al. (2020a), allowing new connections to grow yields improved fiexibility in navigating the loss surfaces, which creates the opportunity to
5

---

Published as a conference paper at ICLR 2022
escape bad local minima and search for the optimal sparse connectivity Liu et al. (0021b) Flying Bird follows a similar design philosophy that excludes least important connections (Han et al, 2013a) while activating new connections with the highest potential to decrease the training loss fastest. Recent works (Wu et al.) 2020c. Liu et al. 2019) have also found enabling network (re- )growth turn poor local minima into saddle point that facilitates further loss decrease. Flying Bird+ empowers the flexibility further by adaptive sparsity level control.
The flatness of loss geometry provides another view to dissect the robust generalization gain Chen et al. 202 Stutz et al. 202T; (Snglaa et al. (02T). Figure comparees the loss landscapes and train- ing trajectories of dense, randomly pruned subnetworks, and Flying Brid+ robustified on CIFAR- 10. We observe that Flying Bird+ converges to wider loss valley with improved flatness. which usually suggests superior robust generalization (Wu et al., 2020a; (Hein & Andriushchenko (017). Last but not the least, our approaches also significantly trim down both the training memory overhead and the computational complexity, enjoying extra bonus of efficient training and inference.
4 EXPERIMENT RESULTS
Datasets and architectures. Our experiments consider two popular architectures, ResNeet-1 (He) et al. 2016), VGG-16 16 (Simonyan & Zisserman (014)) on three representative datasets, CIFAR-TO, CIFAR- 110 (Krizhevsky & Hinton (2009) and Imngeet (Deng et al.] 2009). We randomly split one-tenth of the training samples as the validation dataset, and the performance is reported on the official testing dataset.
Training and evaluation details. We implement our experiments with the original PGD-based adversarial trainig (Madry al. 2018b), in which train the network against foo adversary with maximum perturbations e of 8/255. 10-steps PGD for training and 20-steps PGD for evaluation are chosen with step size of 2 2/255, following Madry et al. all.(221 2b); Chen et al. (2021e) In addition, we also use Auto-Attack (Croce & Hein 2020) and CW Attack (Carlini & 2017) for more rigorous evaluation. More details are provided in Appendix [2] For each experiment, we train the network for 200 epochs with an SGD optimizer, whose momentum and weight decay are kept to 0.9 and 10-4, respectively. The learning rate starts from 0.1 that decays by 10 times at 100,150 epoch and the batch size is 128, which follows Rice et al. (2020).
For Robust Bird, the threshold 7 of mask distance set as 0.1. In Flying Birds(+), we calculate the layer-wise sparsity by Ideal Gas Quotas (IGQ) (Vysogorets & Kempe 202T) and then apply random pruning to initialize the sparse masks. FB updates the sparse connectivity per 2000 iterations of AI, with an update ratio k that starts from 50% and decays by cosine annealing. More details are referred to Appendix A2 Hyperparameters are either tuned by grid search or following Liu et al. (001bb)
Evaluation metries. In general, we care about both the accuracy and efficiency of obtained sparse networks. To assess the accuracy, we consider both Robust Testing Accuracy (RA) and Standard Testing Accuracy (SA) which are computed on the perturbed and the original test sets, together with Robust Generalization Gap (RGG) (i.e., the gap of RA between train and test sets). Meantime, we report the floating point operations (FLOPs) of the whole training process and single image inference to measure the efficiency.
4.1 ROBUST BIRD IS A GOOD BIRD
In this section, we evaluate the effectiveness of static sparsity from diverse representative pruning approaches, including: (i) Random Pruning (RP), by randomly eliminating model parameters to the desired sparsity; (ii) One-shot Magnitude Pruning (OMP), which globally removes a certain ratio of lowest-manittud weights; (iii) Pruming at Initialization algorithms. Three advanced methods, i... SNIP (Lee et al., 2019) GraSP (Wang et al. (2000) and SynFlow (Tanaka al.] (020)), are considered, which identify the subnetworks at initialization respect to certain criterion of gradient flow. (iv) Ideal Gas Quotas (IGS) (Vysogorets & Kempe 2021). It adopts random pruning based on pre-calculated layer-wise sparsity which draws intuitive analogies from physics. (v) Robust Bird (RB), which can be regarded as an early stopped OMP. (vi) Small Dense. It is an important sanity check via considering smaller dense networks with the same parameter counts the ones of sparse networks. Comprehensive results of these subnetworks at 80% and 90% sparsity are reported in Table I where the chosen sparsity follows routine options (Evei et al. 2020a Liu et al. 202

---

Published as a conference paper at ICLR 2022
Table 1: Performance showing the appearance of poor robust generalization/rouust overfitting, and the effec- tiveness of our sparse proposals with various comparisons to other sparsification methods on CFAIR- 10 with ResNet-18. The difference between best and final robust accuracy indicates degradation in performance during training. We pick the best checkpoint by the best robust accuracy on the validation set. Bold numbers indicate superior perfornance, and & displays shrunk robust generalization gap compared to dense models. Note that model picking criterion and the presentation style are consistent for all tables.
Robust Accuracy
Standard Accuracy
Training
Inference
Robust
Sparsity(%) (%)
Settings
Best
Final
Diff.
Best
Final
Dift.
(LOP%
FLON (x:17)
Generalization
Buseline
51.10
43.61
7.49
81.15
83.38
2.23
772.41
260.07
38.82
Small Dense
49.04
41.18
4.86
76.61
80.77
-4 -.13
09.51
23.41
21.68
1 117.4
Rundom Pruning
49.32
43.97
5.35
77.75
81.27
3.52
154.40
51.99
25.70
13.12
OMP
50.16
45.02
5.14
79.80
82.39
-2.59
966.63
65.39
28.38
SNIP
50.46
46.44
4.02
80.13
83.20
-3.07
241.85
81.43
24.24 13.58
GraSP
50.16
15.31
1.85
78.38
12
-1.44
187.11
63.00
26.28
12.54
SynHlow
51.17
46.91
4.26
79.08
83.19
-4.11
256.09
86.23
24.66
11..0
IGQ
51.12
46.74
4.38
79.73
26
3.53
239.39
80.60
25 41
11.41
Robust Bird
50.18
46.30
4.08
78.46
-3.96
209.54
23.37
15:45
Flying Bird
51.62
46.37
5.25
80.55
83.17
- 2.62
239.38
80.60
28. 90.
9..92
Flying Bird+
51.70
17.51
4.19
80.71
83.16
22412
120.04
10.42
23.89
14.50
Small Dense
46.81
45.48
1.33
77.13
78.54
-1.11
24.31
8.19
13. 86 24.06
Random Pruning
47.09
11.97
2.12
75.25
78.77
3.32
77.16
25.98
15.11
23.71
OMP
49.31
46.11
3.20
77.99
$1.00
- 3300
877.76
335.47
19.05 10:27
SNIP
49.49
47.85
1.64
77.74
81.92
-4.18
154.35
51.97
16.20 22.62
GraSP
48.56
46.80
1.76
79.02
81.39
-2.77
113.38
38.18
66.80 22.02
90
SynFlow
50.08
48.02
2.06
81.15
81.56
-0.41
156.74
52.77
14.68 24.14
IGQ
49.74
48.05
1.69
81.06
81.81
-0.78
141.10
47.51
15. 15.955. 22.87
Robust Bird
49.09
46.56
2.53
77.96
80.93.
-2.77
133.42
39.01
16.62 222000
Flying Bird
50.97
48.10
2.87
79.62
82.93
-3.31
141.10
47.51
20.07 18.75
Flying Bird+
50.88
49.27
1.61
79.95
82.65
2.70
66.67
22.45
15.16 23.06
As shown in Table 1 we first observe the occurrence of poor robust generalization with 38.82% RA gap and robust overfitting with 7.49% RA degradation, when training the dense network (Baseline). Fortunately. coincided with our claims, injecting appropriate sparsity effectively tackle the issue. For instance, RB greatly shrinks the RGG by 15.45%//2.2009 at 80/90% sparsity, while also mitigates robust overfitting by 2.53% ~ 4.08%. Furthermore, comparing all static pruning methods, we find that (1) Small Dense and RP behave the worst, which suggests the identified sparse typologies play important roles rather than reduced network capacity only; (2) RB shows clear advantages to OMP in terms of all measurements, especially for 78.32% 84.80% training FLOPs savings. It validates our RB proposal that a few epochs of standard training are enough to learn a high-quality sparse structure for further robustification, and thus there is need to complete the full training in the tickets finding stage like traditional OMP. (3) SynFlow and IGQ approaches have the best RA and SA, while RB obtains the superior robust generalization among static pruning approaches.
Finally, we explore the influence of training regimes during the RB ticket finding on CIFAR-100 with 18.. (ebee/AAjdemonstratte that RB tickets perform best when found with the cheapest standard training. Specifically, at 90% and 95% sparsity, SGD RB tickets outperform both Fast AT (Wong et al. 2020) and PGD-10 RB tickets with up to 1.27% higher RA and 1.86% narrower RGG. Figure A offers possible explanation for this phenomenon: the SGD training scheme more quickly develops high-level network connections, during the early epochs of training (Achille al., 2 As a result, RB Tickets pruned from the model trained with SGD achieve superior quality.
4.2 FLYING BIRD IS A BETTER BIRD
In this section, discuss the advantages of dynamic sparsity and show that our Flying Bird(+) is a superior bird. Table [ examines the effectiveness of FB(+) on CIFAR-10 with ResNet-18, and several consistent observations can be drawn: O FB(+) achieve 9.92% 23.66% RGG reduction, 2.24% ~ 5.88% decrease for robust overfitting, compared with the dense network. And FB+ at 80% sparsity even pushes the RA 0.60% higher. @ Although the smaller dense network shows the leading performance w.r.t improving robust generalization, the robustness has been largely sacrificed, with up to 4.29% RA degradation, suggesting that only reducing models' parameter counts is insufficient to keep satisfactory SA/RA. ① FB and FB+ achieve superior performance of RA for both the best and final checkpoints across all methods, including RB. ! Regardless of small dense and random pruning due to their poor robustness, FB+ reaches the most impressive robust generalization (rank #1 or #2) with the least training and inference costs. Precisely, FB+ obtains 84.46% ~ 91.37% training FLOPS and 84.46% 93.36% inference FL.PPs saving, i.e., Flying Bird+ is SUPER light-
7

---

Published as a conference paper at ICLR 2022
Table 2: Performance showing the effectiveness of our proposed approaches across different datasets with
ResNet-18. The subnetworks at 80% sparsity selected here.
Robust Accuracy
Standard Accuracy
Training
Inference
Robust
Dataset
Settings
Best
Final
Dift.
Best
Final
Diff.
FLOBS (x10")
FLOPS (x10
Generalization
Baseline
51.10
43.61
7.49
81.15
83.38
-222
772.41
260.U7
38.
Robust Bird
50.18
46.10
4.08
78.46
82.42
-3.96
209.54
64.64
23.37
15.45
CIBAR-10
Flying Bird
51.62
46.37
5.25
80.55
83.17
-2.62
239.38
80,60
28,90 L9.52
Flying Bird+
51.70
47.51
4.19
80.74
83.16
-242
120.04
40.42
23.89 14:90
Baseline
26.93
19.62
7.31
52.03
53.91
-1.88
772.41
260.07
54.56
Robust Bird
25.54
20.82
4.72
48.79
53.33
-.54
189.80
58.00
25.46 29.10
CIBAR-100
Flying Bird
26.641
22.00
4.61
53.57
55.41 11
-1844
237.12
79.84
27 27.10
Hlying Bird+
26.66
23.37
3.29
52.29
53.23
-2.44
100.90
33.97
20.12 $1311
Baseline
20.84
15.76
5.08
43.57
46.64
-3.07
6179.30
1040.29
36.84
Robust Bird
19.58
16.45
3.13
43.70
46.30
-2.60
1410.44
215.15
15.22 21.62
Tiny-ImageNet
Flying Bird
20.34
19.00
1.34
45.95
46.86 86
0.91
1884.01
317.17
14.93 21.91
Flying Hirc++
20.36
19.11
1.25
45.67
46.73
-1.06
1225.80
206.36
13.24 25.00
Table 3: Performance showing the offectiveness our proposed approaches with other architectures, i.c.,
VGG-16 16 on CIFAR-10/100. The subnetworks 80% sparsity are selected here.
Robust Accuracy
Standard Accuracy
FLOPS
Robust
Architecture
Dataset
Settings
Final
Final
Training
Inference
Generalization
Baseline
18.33
12.73
5.60
76,81
79.73
-2.89
574.69
193.50
28.00
Robust Bird
47.69
41.66
6.03
75.32
78.58
-3.26
163.95
51.48
23.57 4.48
VGG-16
CIFAR-10
Flying Bird
48.43
44.65
3.78
77.53
79.72
-2.19
173.56
58.44
21.01 (6.50)
Flying Bind+
48.25
45.24
3.01
77.48
79.55
-2.07
94.63
31.86
17.75 L 10.26
Baseline
22.76
18.06
4.70
-0.77
574.69
193.50
63.18
Robust Bird
23.46
17.48
5.98
46.33
47.59
-1.26
165.77
51.42
48.19 14.99
VGG-16
CIFAR-100
Flying 22.75
17.96
4.79
46.61
47.36
00.75
172.14
57.96
48.11 11,07
Flying Bird+
22.92
19.02
3.90
47.01
48.11
-1.10
69.93
23.54
34.63
23..55
Superior performance across datasets and architectures. We further evaluate the performance of FB(+) across various datasets (CIFAR-10, CIFAR-100 and Tiny-ImageNet) and architectures (ResNet-18 18 and VGG-16). Table 2 and display that both static and dynamic sparsity of our pro- posals serve effective remedies for improving robust generalization and mitigating robust overfitting, with 4.43% 15.45%, 14. 14.99% ~ 34.44% and 21.62% 23.60% RGG reduction across different architectures on CIFAR-10, CIFAR- and Tiny-ImageNet, respectively. Moveover, both RB and FB(+) gain significant efficiency, with up 87.83% training and inference FLOPS savings.
Superior performance across improved attacks. Additionally, we verify both RB and FB(+) under improved attacks, i.e., Auto-Attack (Croce & Hein (020) and CW Attack (Carlini & Wag- ner 2017). As shown in Table A8 our approaches shrink the robust generalization gap by up to 76% on CIFAR-10/ and largely mitigate robust overfitting. This piece of evidence shows our proposal's 's effectiveness sustained across diverse attacks.
Combining FB+ with existing start-of-the-art (SOTA) mitigation. Previous works (Chen et al. 2021e] Zhang et al. 2021a Wu et al. 2020b) point out that smoothening regularizations (e.g.. KD (Hinton et al. 2015) and SWA (Izmailov et al. 2018)) help robust generaliza- tion and lead to SOTA robust accuracies. We combine them with our FB+ and collect the robust accuracy CIFAR-10 with ResNet- 18 in Figure 4 The extra robustness gains from FB+ imply that they makes complementary contributions.
Figure 4: Combination of
FB+ and previous SOTAs.
Excluding obfuscated gradients. A common "counterfeit" of robustness improvements is less effective adversarial examples resulted from obfuscated gradients (Athalye et al.) 2018). Table A7 demonstrates the maintained enhanced robustness under unseen transfer attacks, which excludes the possibility of gradient masking. More are referred to Section[3
4.3 ABLATION STUDY AND VISUAIIIZATION
Different sparse initialization and update frequency. As two major components in the dynamic sparsity exploration (Evci et al. 2020a), we conduct thorough ablation studies in Table 4 and [ We found the performance of Flying Bird+ is more sensitive different sparse initialization; using SNIP to produce initial layer-wist wise sparsity and updating the connections per 2000 iterations serves the superior configuration for FB+.
8

---

Published as a conference paper at ICLR 2022
Table 4: Ablation of different sparse initialization in Table S: Ablation different update frequency in Flying Bird+. Subnetwroks at 80% initial sparsity are Flying Bird+. Subnetworks at 80% initial sparsity are chosen on CIFAR-I0 with ResNet-18.
chosen CIFAR-IO with ResNet-18. 88
Iniiaiiaatio
Accurcy
Standand Accurccy
Uptase Progurncy
Rebusl Accorsy
Staddae Acconey
Rebrst
Bosi
Firsl
Dill.
Bost
Firei
Diit
Generdivation
(Reaatos)
Finil
Generalatiio
Unifeess
60.00
46.00
2.15
78.32
50.22
-2.00
16.51
103
50:33
(1:01
1:30
01:
61 19
-0.71
13.36
50.57
73 15
50.37
79:76
40:17
79.60
SyaFoow XQQ
00.56
T8.31
62:17
3000
50.30
18.43
140
79.11
60.58
-5.17
17.95
50.58
48.27
79.56
82.05
-2.70
15.10
10030
50.05
16:03
2.01
79.25
62.30
-5.25
17.61
Prusing
Robust
Flying
Flying
Figure 5: Loss landscape visualization of rohusitified dense network and sparse networks (90% sparsity) from different sparsification approaches CIFAR-10 10 with ResNet-18.
Final checkpoint loss landscapes. From visualizations in Figure 5 FB and FB+ converge to much flatter loss valleys, which evidences their effectiveness in closing robust generalization gaps.
Attention and saliency maps. To visually inspect the benefits of our proposal. here we provide attention and saliency maps generated by Grad-GAM (Selvaraju al., (017) and tools in (Smilkov et al., 2017). Comparing the dense model to our "talented birds" (e.g., FB+), Figure ( shows that our approaches have enhanced concentration on main objects, and are capable of capturing more local feature information, aligning better with human perception.
Adversarial Samples
Dense
Flying
Bird+
Flying
Bird
Robust
Bird
SNIP
Random
Pruning
Heatmap
Saliency Map
Figure 6: (Left) Visualization attention heatmaps on adversarial images based on Grad-Cam (Selaraauu
et al. 2017). (Right) Saliency map visualization on adversarial samples (Smilkov et al. 2017)
CONCLUSION
We show the adversarial training of dense DNNs incurs a severe robust generalization gap, which can effectively and efficiently resolved by injecting appropriate sparsity. Our proposed Robust Bird and Flying Bird(+) with static and dynamic sparsity, significantly mitigate the robust gener- alization gap while retaining competitive standard/robust accuracy, besides substantially reduced computation. Our future works plan to investigate channel- and block-wise sparse structures.

---

Published as a conference paper at ICLR 2022
REFERENCES
Alessandro Achille, Matteo Rovere, and Stefano Soatto. Critical learning periods in deep net- works. In International Conference on Learning Representations, 2019. URL https://
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. arXiv preprint arXiv: 100,000000000200,, 2018.
Emilio Rafael Balda, Arash Behboodi, Niklas Koep, and Rudolf Mathar. Adversarial risk bounds for neural networks through sparsity based compression. arXiv preprint /906.00/0000009000 2019.
Jason Bunk, Srinjoy Chattopadhyay, BS Manjunath, and Shivkumar Chandrasckaran. Adversarially optimized mixup for robust classification. arXiv preprint arXiv:2103.11589 2021.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017
ieee symposium on security and privacy (sp), pp. 39-57. IEFF, 2017.
Chen Chen, Jingfeng Zhang, Xilie Xu, Tianlei Hu, Gang Niu, Gang Chen, and Masashi Sugiyama. Guided interpolation for adversarial training. arXiv preprint arXiv:2102.07327, 2021a.
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Michael Carbin, and Zhangyang Wang. The lottery tickets hypothesis for supervised and self-supervised pre-training in computer vision models. arXiv preprint arXiv:2012.06908 2020a.
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The lottery ticket hypothesis for pre-trained bert networks. arXiv preprint arXiv:2007. 12223, 2020b.
Tianlong Chen, Sijia Liu, Shiyu Chang, Yu Cheng, Lisa Amini, and Zhangyang Wang. Adversarial robustness: From self-supervised pre-training to fine-tuning. Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 699-708, 2020c.
Tianlong Chen, Yu Cheng, Zhe Gan, Jingjing Liu, and Zhangyang Wang. Ultra-data-effiienn gan training: Drawing lottery ticket first, then training it toughly. arXiv preprint arXiv:1103.00397, 2021b.
Tianlong Chen, Yongduo Sui, Xuxi Chen, Aston Zhang, and Zhangyang Wang. A unified lottery ticket hypothesis for graph neural networks, 2021c.
Tianlong Chen, Zhenyu Zhang, Sijia Liu, Shiyu Chang, and Zhangyang Wang. Long live the lottery:
The existence of winning tickets in lifelong learning- In International Conference on Learning Representations, 2021d. URI.htt httpt::ss///peeevviieeee
net http:/orrmmmm.ddttssssiniii
Tianlong Chen, Zhenyu Zhang, Sijia Liu, Shiyu Chang. and Zhangyang Wang. Robust overfitting may be mitigated by properly learned smoothening. In International Conference on Learning Representations, 2021e. URL
Tianlong Chen, Xuxi Chen, Xiaolong Ma, Yanzhi Wang, and Zhangyang Wang. Coarsening the
granularity: Towards structurally sparse lottery tickets. arXiv preprint 2rXiv:2002.04736, 2022.
Xuxi Chen, Tianlong Chen, Zhenyu Zhang, and Zhangyang Wang. You are caught stealing my win- ning lottery ticket! making lottery ticket claim its ownership. Advances in Neural Information Processing Systems, 34, 2021f.
Xuxi Chen, Zhenyu Zhang, Yongduo Sui, and Tianlong Chen. Gans can play lottery tickets too. In International Conference on Learning Representations, 2021g. URL.httss://openrewe
net futt/Eorum?icom 1AMMhe 9 9jER
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-fi free attacks. In International Conference on Machine Learning, pp. 2206- 2216. PMLR, 2020.
10

---

Published as a conference paper at ICLR 2022
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A scage-scale hi- erarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255, Icee, 2009.
Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv preprini urXiv:1907.04844 2019.
Guneet S. Dhillon, Kamyar Azizzadenesheli, Jeremy D. Bernstein, Jean Kossaifi, Aran Khanna, Zachary C. Lipton, and Animashree Anandkumar. Stochastic activation pruning for robust adversarial defense. In International Conference on Learning Representations, 20. URL
https:// //tpr//eepeeeeeeeeeeeeeeeeee net nett/oorumid---1104400
Guneet S Dhillon, Kamyar Azizzadenesheli, Zachary C Lipton, Jeremy Bernstein, Jean Kossaifi, Aran Khanna, and Anima Anandkumar. Stochastic activation pruning for robust adversarial de- fense. arXiv preprint arXhv:1803.0144; 2018b.
Shaojin Ding, Tianlong Chen, and Zhangyang Wang. Audio lottery: Speech recognition made ligtteighh,eiiht, noise-robust, and transferable. In International Conference on Learning Repre- sentations, 2022. URL https: httpps://peereeview n forur (te.rummmiii9n6-nA--. 6AJkVYB
Yinpeng Dong, Fangzhou Liao, Tianyu Pang. Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boost- ing adversarial attacks with momentum. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, 2018.
Yinpeng Dong, Ke Xu, Xiao Yang, Tianyu Pang, Zhijie Deng. Hang Su, and Jun Zhu. Exploring memorization adversarial training. arXiv preprint arXiv:2106.01666, 2021.
Gintare Karolina Dziugaite, Zoubin Ghahramani, and Daniel M Roy. A study of the effect of jpg compression on adversarial images. arXiv preprint arXiv: 00tr:6008.0085,, 2016.
Logan Engstrom, Andrew lyas, and Anish Athalye. Evaluating and understanding the robustness of adversarial logit pairing. arXiv preprint arXiv:/807.10222, 2018
Utku Evci, Fabian Pedregosa, Aidan Gomez, and Erich Elsen. The difficulty of training sparse neural networks. arXiv preprint arXiv: 10732, 2019.
Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pp. 2943-2952. PMLR, 2020a.
Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In Hal Daumé III and Aarti Singh (eds.). Proceedings the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning mmrr.
Research, pp. 2943-2952. PMLR, 13-18 Jul 2020b. URL http::/ppoceedee
html
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019. URL https::/ openreview net. 2orrumidd
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In ICML, 2020.
Yonggan Fu, Qixuan Yu, Yang Zhang, Shang Wu, Xu Ouyang, David Daniel Cox, and Yingyan Lin. Drawing robust scratch tickets: Subnetworks with inborn robustness are found within randomly initialized networks. In Thirty-Fii Conference Neural Information Processing Systems, 2021.
Zhe Gan, Yen-Chun Chen, Linjie Li, Tianlong Chen, Yu Cheng, Shuohang Wang, and Jingjing Liu. Playing lottery tickets with vision and language. arXiv preprint 2tti::2004.113222 2021.
Ji Gao, Beilun Wang, Zeming Lin, Weilin Xu, and Yanjun Qi. Deepcloak: Masking deep neural network models for robustness against adversarial samples. arXiv preprint 1rXiv:1702.06777 2017.

---

Published as a conference paper at ICLR 2022
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and hamnessing adversarial examples. In International Conference on Learning Representations, 2015. URL http://
arxiv. awtaaasss111 6572
Shupeng Gui, Haotao N Wang, Haichuan Yang, Chen Yu, Zhangyang Wang, and Ji Liu. Model compression with adversarial robustness: A unified optimization framework. Advances in Neural Information Processing Systems, 32:1285-1296 2019.
Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens van der Maaten. Countering adversarial images using input transformations. In International Conference Learning Representations, 2018a. URL httlttttt//:::eereeeet wwtp:/iwwwwweeeeeeeeeee.me.eee
Yiwen Guo, Chao Zhang, Changshui Zhang, and Yurong Chen. Sparse dnns with improved adver- sarial robustness. arXiv preprint 1810:0881 2018b.
Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding, arXiv preprint arXiv: 15 150.000/49 2015a.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in newral information processing systems, pp. 1135-1143, 2015b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pp. 770-778, 2016.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. Proceedings of the IEEE International Conference on Computer Vision, pp. 1389-1397, 2017.
Matthias Hein and Maksym Andriushchenko. Formal guarantees the robustness of classifier against adversarial manipulation. In Advances in Neural Information Processing Systems, pp. 2266-2276 2017.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv:1503.0253: 2015.
Ting-Kue Hu, Tianlong Chen, Haotao Wang, and Zhangyang Wang. Triple wins: Boosting aceu- racy, robustness and efficiency together by enabling input-adaptive inference. In International Conference on Learning Representations, 2019.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wil- son. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv: 05407, 2018.
Gauri Jagatap. Animesh Basak Chowdhury, Siddharth Garg, and Chinmay Hegde. Adversarially robust learning via entropic regularization. arXiv preprint arXiv:2008. 12338, 2020.
Siddhant M Jayakumar, Razvan Pascanu, Jack W Rae, Simon Osindero, and Erich Elsen. Top-kast: Top-k always sparse training. arXiv preprint arXiv:2106.03517, 2021.
Ziyu Jiang, Tianlong Chen, Ting Chen, and Zhangyang Wang. Robust pre-training by adversarial contrastive learning. arXiv preprint arXiv:2010.13337 2020.
A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. 2016.
arXiv preprint
Yann LeCun, John $ Denker, and Sara A Solla. Optimal brain damage. In Advances in neural information processing systems, pp. 598-605, 1990.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip H. S. Torr. Snip: Single-shot network pruning based connection sensitivity, 2019.
12

---

Published as a conference paper at ICLR 2022
Sachyung Lee, Hyungyu Lee, and Sungroh Yoon. Adversarial vertex mixup: Toward better adver- sarially robust generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 272-281, 281, 2020.
Bai Li, Shiqi Wang, Yunhan Jia, Yantao Lu, Zhenyu Zhong, Lawrence Carin, and Suman Jana. Towards practical lottery ticket hypothesis for adversarial training. arXiv preprint arX5v:2003.05733, 2020.
F. Liao, M. Liang, Y. Dong, T. Pang, X. Hu, and J. Zhu. Defense against adversarial attacks using high-level representation guided denoiser. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2018.
Qiang Liu, Lemeng Wu, and Dilin Wang. Splitting steepest descent for growing neural architectures. arXiv preprint arXiv:1910.02366. 2019.
Shiwei Liu, Decebal Constantin Mocanu, Yulong Pei, and Mykola Pechenizkiy, Selfish sparse rnn training. arXiv preprint arXiv:2101.09045: 2021a.
Shiwei Liu, Lu Yin, Decebal Constantin Mocanu, and Mykola Pechenizkiy. Do we actually need dense wve-pparameterization? in-time over-parameterizzaion in sparse training. arXiv preprint arXiv:2102.02889 2021b.
Xuanqing Liu, Minhao Cheng, Huan Zhang, and Cho-Jui Hsieh. Towards robust neural networks via random self-ensemble. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 369-385, 2018.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. In Proceedings of the IEEE international conference on computer vision, pp. 7736-2744, 2017.
Alcksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint 1rXiv:1706.06008 2017.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. To- wards deep learning models resistant adversarial attacks. In International Conference on Learn- ing Representations, 20. URL /tRLhttts////peereeee/ net.:/ewww....e.e.......e...ee.e http://pppaaaeeeeeeeeeee.ocmaoaa wtt::/orrmmdddiaaaaaaaa
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. To- wards deep learning models resistant to adversarial attacks, In International Conference on Learn- ing Representations, 2018b. URL /ttp::s////peeeeeeieeee./ http:ppwww.....eeeeeee.eooeoaaaa
Decebal Constantin Mocanu, Elena Mocanu, Phuong Nguyen, Madeleine Gibescu, and Antonio Liotta. A topological insight into restricted boltzmann machines. Machine Learning, 104(2); 243-270, 2016.
Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural networks. In International Conference on Machine Learning, pp. 1498-2507. PMLR, 2017.
Pavlo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1164--1272, 2019.
Seyed-Mohsen Moosavi-Dezfooli Alhussein Fawzi, Jonathan Uesato, and Pascal Frossard. Ro- bustness via curvature regularization, and vice versa. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9078-9086, 2019.
Marius Mosbach, Maksym Andriushchenko, Thomas Trost, Matthias Hein, and Dietrich Klakow. Logit pairing methods can fool gradient- gradient-based attacks. arXiv preprint arXiv:1810.12044 2018.
Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In International Conference on Machine Learning, pp. 4646-465. PMLR, 2019.
13

---

Published as a conference paper at ICLR 2022
Preetum Nakkiran. Adversarial robustness may be at odds with simplicity. arXiv preprint arXiv:1901.00532, 2019.
Ozan Ózdenizei and Robert Legenstein. Training adversarially robust sparse networks via bayesian connectivity sampling. In International Conference Machine Learning, pp. 8314 8324. PMLR, 2021.
Tianyu Pang. Xiao Yang. Yinpeng Dong, Hang Su, and Jun Zhu. Bag tricks for adversarial training. In International Conference Learning Representations, 2021. URL https://
Aditi Raghunathan, Sang Michael Xie, Fanny Yang, John C Duchi, and Percy Liang. Adversarial training can hurt generalization. In ICMLW, 2019.
Md Aamir Raihan and Tor M Aamodt. arXiv:2001.01969 2020.
Sparse weight activation training. arXiv preprint
William T Redman, Tianlong Chen, Akshunna Dogra, and Zhangyang Wang. Universality of deep neural network lottery tickets: A renormalization group perspective. arXiv preprint arXiv:21 2021.
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. In International Conference Learning Representations, 2020.
Leslie Rice, Eric Wong, and Zico Kolter. Overfitting in adversarially robust deep learning. In
Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Ma- chine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 8093-8104. URL.
PMLR, 13-18 Jul 2020.
ml
html
Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. Ad- versarially robust generalization requires more data. In NewrIPS, pp. 5014-5026, 2018.
Vikash Sehwag, Shiqi Wang, Prateek Mittal, and Suman Jana. Towards compact and robust deep neural networks. arXiv preprint 19X.v:9906.06110, 2019.
Vikash Sehwag. Shiqi Wang, Prateek Mittal, and Suman Jana. Hydra: Pruning adversarially robust neural networks. arXiv preprint arXiv:2002. 10509, 2020.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 618-626, 2017.
Ali Shafahi, Mahyar Najibi, Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! arXiv preprint arXiv:1904.1 12843, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1rXivv:149..11 2014.
Vasu Singla, Sahil Singla, David Jacobs, and Soheil Feizi Low curvature activations reduce overfit- ting in adversarial training. arXiv preprint arXiv:2102.07861, 2021.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint /ttiv:770..03888 2017.
Chuanbiao Song, Kun He, Jiadong Lin, Liwei Wang. and John E Hoperoft. Robust local features for improving the generalization of adversarial training. arXiv preprint arXiv: 10147, 2019,
David Stutz, Matthias Hein, and Bernt Schiele. Relating adversarially robust generalization to flat minima. arXiv preprint 2Xix:2/04.04444 2021.
Ke Sun, Zhanxing Zhu, and Zhouchen Lin. Towards understanding adversarial examples systemat- ically: Exploring data size, task and model factors. arXiv preprint /90iv: /900. 2019.
14

---

Published as a conference paper at ICLR 2022
Jihoon Tack, Sihyun Yu, Jongheon Jeong, Minseon Kim, Sung Ju Hwang, and Jinwoo Shin. Con- sistency regularization for adversarial robustness. arXiv preprint arXiv:2103.04623, 2021.
Hidenori Tanaka, Daniel Kunin, Daniel LK Yamins, and Surya Ganguli. Pruning neural networks
without any data by iteratively conserving synaptic flow. arXiv preprint arXiv:2006. 05467,, 2020.
Artem Vysogorets and Julia Kempe. Connectivity matters: Neural network pruning through the lens of effective sparsity. arXiv preprint arXiv:2107.02336. 2021.
Chaoqi Wang. Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow, 2020.
Jianyu Wang and Haichao Zhang. Bilateral adversarial training: Towards fast training of more robust models against ad adversarial attacks, In Proceedings of the IEEE International Conference on Computer Vision, pp. 6629-6638. 2019.
Siyue Wang, Xiao Wang, Shaokai Yc, Pu Zhao, and Xue Lin. Defending dnn adversarial attacks with pruning and logits augmentation. In 2018 IEEE Global Conference on Signal and Information Processing (GlobalSIP), pp. 1 1144-118, IEEE, 2018.
Eric Wong, Leslie Rice, and J. Zico Kolter. Fast is better than free: Revisiting adversarial training. In International Conference on Learning Representations, 2020. UKL.itt.pss:/ppenrvviic.
Dongxian Wu, Yisen Wang, and Shu-tao Xia. Revisiting loss landscape for adversarial robustness. arXiv preprint arXiv:2004.05884 2020a.
Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust gener- alization. In NewrlPS, 2020b.
Lemeng Wu, Mao Ye, Qi Lei, Jason D Lee, and Qiang Liu. Steepest descent neural architecture opti-
mization: Escaping local optimum with signed neural splitting. arXiv preprint arXiv:2003.10392 2020c.
Cihang Xie, Jianyu Wang, Zhishuai Zhang, Zhou Ren, and Alan Yuille. Mitigating adversarial
effects through randomization. In International Conference on Learning Representations, 2018. URL http:httss://ppeneeeeee
ftt/m/orrmmm/// Sd= 9yuq OZ)
Weilin Xu, David Evans, and Yanjun Qi. Feature squeezing: Detecting adversarial examples in deep neural networks. arXiv preprint arXiv: 155, 2017.
Dingqing Yang, Amin Ghasemazar, Xiaowei Ren, Maximilian Golub, Guy Lemieux, and Mieszko Lis. Procrustes: a dataflow and accelerator for sparse deep neural network training. In 2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), pp. 711-724. IEEE, 2020.
Shaokai Ye. Kaidi Xu, Sijia Liu, Hao Cheng, Jan-Henrik Lambrechts, Huan Zhang, Aojun Zhou, Kaisheng Ma, Yanzhi Wang, and Xue Lin. Adversarial robustness vs. model compression, or both? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 111- 120, 2019.
Haoran You, Chaojian Li, Pengfei Xu, Yonggan Fu, Yue Wang, Xiaohan Chen, Richard G. Baraniuk, Zhangyang Wang, and Yingyan Lin. Drawing early-bird tickets: Toward more efficient training of deep networks. In International Conference on Learning Representations, 2020. URL. https: //openreview. .et. wwttt/foummiiiee BUx
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. International Conference on Machine Learning. pp. 7472-7482. PMLR, 2019.
Jingfeng Zhang, Xilie Xu, Bo Han, Gang Niu, Lizhen Cui, Masashi Sugiyama. and Mohan Kankan- halli. Attacks which do not kill training make adversarial learning stronger. In International conference on machine learning, pp. 11278-1 1287. PMLR, 2020.
15

---

Published as a conference paper at ICLR 2022
Jingfeng Zhang, Xilie Xu, Bo Han, Tongliang Liu, Gang Niu, Lizhen Cui, and Masashi Sugiyama. Noilin: Do noisy labels always hurt adversarial training? arXiv preprint arXiv:2105.14677 2021a.
Jingfeng Zhang, Jianing Zhu, Gang Niu, Bo Han, Masashi Sugiyama, and Mohan Kankanhalli. Geometry-aware instance-reweighted adversarial training. In International Conference on Learn- ing Representations, 2021b.
Shuai Zhang. Meng Wang, Sijia Liu, Pin- Yu Chen, and Jinjun Xiong. Why lottery ticket wins? theoretical perspective of sample complexity sparse neural networks, 2021c. URL https:
Tianyun Zhang, Kaiqi Zhang, Shaokai Ye, Jian Tang, Wujie Wen, Xue Lin, Makan Fardad, and Yanzhi Wang. Adam-admm: A unified, systematic framework of structured weight pruning for dnns. arXiv preprint 1807.1180..1199 2018.
Zhenyu Zhang, Xuxi Chen, Tianlong Chen, and Zhangyang Wang. Efficient lottery ticket finding: Less data more. International Conference on Machine Learning, pp. 12380-12390. PMLR, 2021d.
Dawei Zhou, Nannan Wang, Xinbo Gao, Bo Han, Jun Yu, Xiaoyu Wang, and Tongliang Liu. Im- proving white-box robustness of pre-processing defenses via joint adversarial training. arXiv preprint arXiv: 106. 106.05536. 2021.
Hao Zhou, Jose M Alvarez, and Fatih Porikli. Less more: Towards compact cnns. In European Conference on Computer Vision, pp. 662-677. Springer, 2016.
Bojia Zi, Shihao Zhao, Xingjun Ma, and Yu-Gang Jiang. Revisiting adversarial robustness distilla-
tion: Robust soft labels make student better. arXiv preprint arXiv:2108.0790 079669 2021.
16

---

Published as a conference paper at ICLR 2022
A1 MORE TECHNIQUE DETAILS
Algorithms of Robust Bird and Flying Bird(+). Here we present the detailed procedure to iden- tify robust bird and flying bird(+), summarized in algorithm [ and Note that for the increasing frequency on Line 10 and 11 in algorithm 2 we compare the measurements stored in the queue between two consequent epochs and calculate the frequency of increasing.
Algorithm 1: Finding Robust Bird
Input: f(;;;00 w. initialization On. target sparsity 8%, FIFO queue Q with length 1, threshold 7 Output: Robust bird f(x; m @ 01)
while 7 < (max do
Update network parameters 0r t- 0,-1 via standard training
Apply static pruning towards target sparsity 5% and obtain the sparse mask my Calculate the Hamming distance 6h (m,, mc-1), append result to Q
f t+1
(max(() < r then
ta -
Rewind f(x; my- @ 0(-))- f(x; my- O 00)
Training / (2; my @ 0n) via PGD-AT for epochs
return (a; my- @ (T)
10
end 11
12 end
Algorithm 2: Finding Flying Bird(+)
Input: Initialization parameters 0o, sparse masks of sparsity s%. FIFO queue Qp andQ, with length 1, pruning and growth increasing ratio 6p and 6g. update threshold c, optimize interval At, parameter update ratio k%, ratio update starting point tstart Output: Flying bird(+) (x; m @ 0T)
while T do
Update network parameters 0, + 0,-1 via PGD-AT;
Record training statistics
Add robust generalization gap between train and validation set to Qp
Add robust validation loss to Q,
Update sparse masks
if (t mod At) == 0 then
--- -0ttiionnnll for Flying Birdt---1
Update pruning and growth ratio p%, 9%
ift fstart and increasing frequency of Q₂ 2 c: p (1+ 5p) x k else p k
11
if > fstart and increasing frequency of Qg 2 (1 6g)
x k else g = k
-----pttonne
for Flying Bi Birdt---1
13
Prane p% parameters with smallest weight magnitude
Grow 9% parameters with largest gradient
14
15
Update sparse mask m accordingly
16
end
17 end
A2
MORE IMPLEMENTATION DETAILS
A2.1 OTHER COMMON DETAILS
We select two checkpoints during training: best, which has the best RA values on the validation set, and final, i.e., the last checkpoint. And we report both RA and SA of these two checkpoints on test sets. Apart from the robust generalization gap, we also show the extent of robust overfitting numerically by the difference of RA between best and final. Furthermore, calculate the FLOPSs
A17

---

Published as a conference paper at ICLR 2022
at both training and inference stages to evaluate the prices of obtaining and exploiting the subnet- works respectively, in which we approximate the FLOPs of the back-propggation to be twice that of forwarding propagation (Yag al. 2020
A2.2 MORE DETAILS ABOUT ROBUST BIRD
For the experiments of RB tickets finding, we comprehensively study three training regimes: stan- dard training with stochastic gradient descent (SGD), adversarial training with PGD-10 AT (Madry) (ct al., 20188), and Fast AT (Wong et al. 2020). Following Pang et al. (202I), we train the network with SGD optimizer of 0.9 momentum and 5 10 weight decay. We use batch size of 128. For the experiments of PGD-10 AT, we adopt the l PGD attack with a maximum perturbation 8/255 and step size 2/255. And the learning rate starts from 0.1, then decays by ten times at 50, 150 epoch. As for fast AT, we use a cyclic schedule with a maximum learning rate equals 0.2.
A2.3 MORE DETAILS ABOUT FLYING BIRD(+)
For the experiments of Flying Bird+, the increasing ratio of pruning and growth 6p. 6g kept default to 0.4% and 0.05%, respectively.
A3 MORE EXPERIMENT RESULTS
A3.1 MORE RESULTS ABOUT ROBUST BIRD
Accuracy during RB Tickets Finding Figure|A7 the curve standard test accuracy during the training phase of RB ticket finding. We can observe the SGD training scheme develops high- level network connections much faster than the others, which provides possible explanation for the superior quality of RB tickets from SGD.
RB Ticket Finding Performance
80
Standard Accuracy %
70
60
50
PGD-10
40
SGD
30
FAST AT
5
10
15
20
25
30
Epoch
Figure A7: Standard accuracy (SA) of PGG-100, SGD, and Fast AT during the RB ticket finding phase.
Mask Similarity Visualization. Figure A visualizes the dynamic similarity scores for each epoch among masks found via SGD, Fast AT, and PGD-10. Specifically, the similarity scores (You et al.) 2020) reflect the Hamming distance between a pair of masks. We notice that masks found by SGD and PGD-10 share more common structures. A possible reason is that Fast AT usually adopts cyclic learning rate schedule, while SGD and PGD use multi-step decay schedule.
Different training regimes for finding RB tickets. We denote the subnetworks identified by stan- dard training with SGD, adversarial training with Fast AT (Wong et al. 2020) and adversarial train-
A18

---

Published as a conference paper at ICLR 2022
20
40 60 80 100
20
40
60
80
100
0
20
40
60
80
100
PGD-10 Mask
20
SGD Mask
20
SGD Mask
20
40
40
40
60
80
80
80
100
100
Fast AT Mask
Fast AT Mask
PGD-10 Mask
0.50
0.55
0.60
0.65
0.70
0.75
Similarity scores by epoch among masks found via Fast AT, SGD, and PGD-10. A brighter color denotes higher similarity.
Figure A8:
Table A6: Comparison results of different training regimes for RB ticket finding CIFAR-100 with
ResNet- 18. The subnetworks at 90% and 95% are selected here.
Roubst Accuarcy
Standard Accuarcy
Robust
Sparsity(%)
Settings
Best
Final
Diff.
Best
Final
Diff.
Generalization
0
Baseline
26.93
19.62
7.31
52.03
53.91
1.88
54.56
SGD tickets
25.83
23.40
2.43
49.35
53.51
4.16
18.37, 36.19
90
Fast AT tickets
25.15
22.88
2.27
51.00
51.75
-0.75
20.23] 34.33
PGD-10 tickets
25.34
22.96
2.38
52.01
53.27
-126
20.03; 34.53
SGD tickets
24.77
24.12
0.65
49.88
50.89
-1.01
9.18| 45.38
95
Fast AT tickets
23.50
22.46
1.04
41.67
43.19
1.52
9.53] 45.03
PGD-10 tickets
24.44
23.77
0.67
49.30
50.65
-1335
9.86, 44.70
ing with PGD-10 AT as SGD tickets, Fast AT tickets, and PGD-10 tickets, respectively. Table A6 demonstrate the SGD tickets has the best performance.
Loss Landscape Visualization We visualize the loss landscape of the dense network, random pruned subnetwork. and robust bird tickets 30% sparsity in Figure A9 Compared with the dense model and random pruned subnetwork, RB tickets found by the standard training shows much flatter loss landscapes, which provide high-quality starting point for further robustification.
A3.22 MORE RESULTS ABOUT FLYING BIRD(+)
Excluding Obfuscated Gradients. To exclude this possibility of gradient masking, we show that methods maintain improved robustness under unseen transfer attacks. As shown in Table A7 the left part represents the testing accuracy of perturbed test samples from an unseen robust model, and the right part shows the transfer testing performance separately robustified ResNet-50 with PGD-10 on CIFAR-100).
unseen robust model (here we use
Performance under Improved Attacks. We report the performance of both RB and FB(+) under Auto-Attack (Croce & Hein 2020) and CW Attack (Carlini & Wagner 2017). For Auto-Attack, we keep the default setting with 255
$ And for CW Attack we perform search step on with
an initial constant of 0.1. And we use 100 iterations for each search step with the learning rate of 0.01. As shown in Table both RB and FB(+) outperform the dense counterpart in terms of robust generalization. And FB+ achieves superior performance.
More Datasets and Architectures We report results of different sparsification methods across diverse datasets and architectures at Table A9b A10 All and [112] from which we observe approaches capable of improving robust generalization and mitigating robust overfitting.
A19

---

Published as a conference paper at ICLR 2022
Dense Models
Random Pruning (30%)
RB Tickets (30%)
0.10
0.10
-0.10
0.00
-0.10
0.00
-0.10
0.00
-0.05
-0.05
-0.05
0.00
-0.05
0.00
0.05
0.05
0.05
-0.10
0.10
0.10
-0.10
0.00
-0.10
0.00
-0.10
0.00
0.00
0.00
-0.05
0.00
-0.05
-0.10
0.05
-0.10
0.05
Figure A9: Loss landscapes visualizations (Engstrom et al. 2018 Chen et al. 2021e of the dense model (unpruned), random pruned subnetwork at 30% sparsity, and Robust Bird (RB) tickets at 30% sparsity found by the standard training. The ResNet-18 backbone with the same original initialization CIFAR-10 is adopted here. Results demonstrate that RB tickets offer smoother and flatter starting point for further robustification in the second stage.
Table A7: Transfer attack performance from/on an unseen non-robust model, where attacks are generated
by/applied to the non-robust model. The robust generalization gap also calculated based transfer attack accuracies between train test sets. We use ResNet-18 on CIFAR-10/100 and sub-networks at 80% sparsity.
Transfer Attack from Unseen Model
Transfer Attack on Unseen Model
Dataset
Settings
Accuracy
Robust
Accuracy
Robust
Best
Final
Diff.
Generalization
Best
Final
Diff.
Generalization
Bascline
79.68
82.03
-2.3
16.43
70.48
79.85
-9.37
11.84
Robust Bird
77.33
81.04
-3.71
12.18
73.17
77.03
-3.86
11.49
CIFAR-10
Flying
79.13
82.17
-3.04
13.49
71.59
77.19
-5.60
Flying Bird+
79.47
81.90
-2.43
11.85
70.43
76.00
-5.57
11.42
Bascline
50.51
52.15
-1.64
45.91
48.67
54.48
-5.81
36.98
Robust Bird
47.25
51.74
-4.49
28.80
47.47
50.90
-3.43
35.82
CIFAR-100
Flying Bird
51.80
53.52
-1.72
31.98
45.56
50.61
-5.05
35.39
Flying Bird+
50.72
53.56
-284
25.09
47.04
49.43
-239
35.09
Distributions of Adopted Sparse Initialization. We report the layer-wise sparsity of different initial sparse masks. As shown in Figure A10 we observe that subnetworks generally have better performance when the top layers remain most of the parameters.
Training Curve of Flying Bird+. Figure A1l shows the training curve of Flying Bird+, in which the red dotted lines represent the time for increasing the pruning ratio and the green dotted lines for growth ratio. The detailed training curve demonstrates the flexibility of flying bird+ for dynamically adjusting the sparsity levels.
A4 EXTRA RESULTS AND DISCUSSION
We sincerely appreciate all anonymous reviewers' and area chairs' constructive discussions for im- proving this paper. Extra results and discussions presented in this section.
A20

---

Published as a conference paper at ICLR 2022
Table A8: Evaluation under improved attacks (i.c, Auto-Attack and CW-Attack on CIFAR-10/100 with ResNet-18 18 at 80% sparsity. The robust generalization gap is computed under improved attacks.
Auto-Attack
CW-Attack
Dataset
Settings
Accuracy
Robust
Accuracy
Robust
Best
Final
Diff.
Generalization
Best
Final
Diff.
Generalization
Baseline
47.41
41.59
5.82
35.30
75.76
66.13
9.63
30.39
Robust Bird
45.90
42.45
3.45
21.58 L 13.72
73.95
73.52
0.43
17.67 12.72
CIFAR-10
Flying Bird
47.55
43.37
3.98
26.55 8.75
75.30
72.08
3.22
21.77 4 8.02
Flying Bird+
47.06
44.09
3.17
21.73 13.57
76.00
73.83
2.17
17.77 à 12.62
Baseline
23.16
17.68
5.48
49.73
45.83
36.21
9.62
57.52
Robust Bird
21.29
18.00
3.29
21.72
28.00
43.30
42.39
0.91
30.82
26.70
CIFAR-100
Flying Bird
22.74
19.44
3.30
25.18
46.23
42.36
3.87
35.50
Flying Bird+
22.90
20.31
2.59
19.05 L.3068
45.86
43.90
1.96
26.76
Table A9: More results of different sparcification methods on CIFAR-10 with ResNet-18.
Robust Accuracy
Standard Accuracy
Robust
Sparsity($)
Settings
Best
Final
Diff.
Best
Final
Diff.
Generalization
Baseline
51.10
43.61
7.49
81.15
83.38
2.23
38.82
Small Dense
45.99
44.55
1.44
74.26
75.64
1.38
7.87
4. 1,3955
Random Pruning
45.64
44.18
1.46
75.20 20
75.20
0.00
7.96
30.86
OMP
47.08
46.23
0.85
78.77
79.36
0.59
12.01
262681
SNIP
48.18
46.72
1.46
78.55
79.21
-0.66
9.58
29.24
GraSP
48.58
47.15
1.43
78.95
79.44
-0.49
10.37
28.45
95
SynFlow
48.93
48.22
0.71
78.70
78.90
-0.20
8.25
[ 30.57
IGQ
48.82
47.56
1.26
79.44
79.76
0.32
9.33
29.49
Robust Bird
47.53
46.48
1.05
78.33
78.78
-0.45
9.20
Flying Bird
49.62
48.46
1.16
78.12
81.43
3.31
13.32
25.52
Flying Bird+
49.37
48.84
0.53
80.33
80.28
0.05
9.27
29.55
UniSeem
GraSP
Sparsity
SNIP
SyuFlow
BRR
capvi
layer (oneee
layer 0.coer2
Isyer cxevl
layer L.s0v2
1aye2 (uoerv
laye2 0.00v22
loyer2 n
laye2..1 Locevi
1aye2.1 Low022
laver. 1
layes Dovee
layerk dbrrocut.n
Iayes3 sooev
layed, Lome
layert 0
layasd cveee
layort dhortouu
layert 1
layert, 2
Layer Name
Figure A10: Layer-wise sparisty of different initial sparse masks with ResNet-18
A4.1 MORE RESULTS OF DIFFERENT SPARSITY
We report more results of subnetworks with 40/60 /60 sparsity on CIFAR-10/100 with ResNet-18 and VGG-16. As shown in Table A13 A14 A15 and AT6 our flying bird(+) achieves consistent improvement than baseline unpruned networks, in terms of 2.45 ~ 19.81% narrower robust gener- alization gaps with comparable RA and SA performance.
A4.2 MORE RESULTS ON WIDERESNET
We further evaluate our flying bird(+) with WideResNNet334110 CIFAR-10 10 and report the results on Table A17 We can observe that compared with the dense network, our methods significantly shrink the robust generalization gap by up 14.1% and maintain comparable RA/SA performance.
A21

---

Published as a conference paper at ICLR 2022
Table A10: More results of different sparcification methods CIFAR-10 with VGG-16.
Robust Accuracy
Standard Accuracy
Robust
Sparsity($))
Settings
Best
Final
Diff.
Best
Final
Diff.
Generalization
0
Baseline
48.33
42.73
5.60
76.84
79.73
2.89
28.00
Random Pruning
46.14
40.33
5.81
74.42
76.68
-2.6
21.01 J 6.99
OMP
47.90
43.19
4.71
76.60
80.02
3.42
24.97
3.003
SNIP
48.03
43.17
4.86
76.68
80.08
-3.40
24.71
3.29
GraSP
47.91
42.34
5.57
75.74
78.87
3.13
23.65
4.35
80
SynFlow
48.47
45.32
3.15
77.62
79.09
-1.47
20.17
7.83
IGQ
48.57
44.25
4.32
77.51
80.01
2.50
22.79
5.21
Robust Bird
47.69
41.66
6.03
75.32
78.58
-3.26
23.57
4.43
Flying Bird
48.43
44.65
3.78
77.53
79.72
-2.9
21.01
6.99
Flying Bird+
48.25
45.24
3.01
77.48
79.55
-2.07
17.75
10.25
Random Pruning
44.33
40.33
4.00
71.27
74.46
-3.19
15.48
12.52
OMP
47.84
43.34
4.50
75.60
79.10
3350
18.29
9.7.1
SNIP
47.76
44.27
3.49
75.92
79.62
-3.70
17.85
10.15
GraSP
45.96
42.12
3.84
75.19
77.03
1.84
15.04
12.96
90
SynFlow
47.54
45.79
1.75
78.43
78.70
-0227
14.40
13.60
IGQ
47.79
45.12
2.67
74.87
79.19
4.32
16.06
11.94
Robust Bird
47.09
44.13
2.96
75.53
78.36
-283
16.57
11.43
Flying Bird
48.45
45.55
2.90
75.82
79.21
3.39
16.56
11.44
Flying Bird+
48.39
46.26
2.13
78.73
79.12
-033
12.47
15.53
Table All: More results of different sparcification methods on CIFAR-100 with ResNet-18.
Robust Accuracy
Standard Accuracy
Robust
Sparsity(%)
Settings
Best
Final
Diff.
Best
Final
Diff.
Generalization
Baseline
26.93
19.62
7.31
52.03
53.91
-1.88
54.56
Small Dense
24.40
21.83
2.57
51.87
51.64
0.23
21.93
32.63
Random Pruning
25.92
20.83
5.09
48.16
51.31
-3.15
34.04
| 20252
OMP
25.12
20.18
4.94
50.08
52.81
-2733
28.57
26.00
SNIP
26.61
23.55
3.06
49.47
54.79
-5.32
23.69
1 3.3087
GraSP
25.37
20.79
4.58
50.27
53.29
-3.02
28.03
26.53
80
SynFlow
26.31
23.52
2.79
48.33
54.49
6.16
20.29
3..227
IGQ
26.87
23.07
3.80
49.80
54.39
-459
27.04
27.52
Robust Bird
25.54
20.82
4.72
48.79
53.33
4.54
25.46
29.10
Flying Bird
26.64
22.00
4.64
53.57
55.41
-1.84
27.46
27
Flying Bird+
26.66
23.37
3.29
52.29
55.23
2.94
20.12
34.44
Small Dense
23.61
22.81
0.80
48.45 44
48.63
-0.19
11.18
13.38
Random Pruning
24.06
21.45
2.61
47.06
49.73
-2.67
18.04
3.3652
OMP
24.45
21.38
3.07
48.02
51.26
3.24
17.11
33..45
SNIP
26.10
24.46
1.64
52.35
52.88
-0.53
11.54
13.02
GraSP
24.83
22.74
2.09
51.09
52.55
1.46
14.55
40.01
90
SynFlow
25.45
24.62
0.83
51.03
51.96
-0.93
10.38
14.18
IGQ
26.22
24.87
1.35
52.37
53.16
0.79
13.90
40.66
Robust Bird
24.65
22.96
1.69
46.16
51.87
-5.71
16.14
38-42
Flying Bird
26.14
23.57
2.57
50.53
54.78
4.25
16.73
37.83
Flying Bird+
26.26
24.16
2.10
51.16
53.97
-2.81
11.44
43.12
A4.3 COMPARISON WITH EFFICIENT ADVERSARIAL TRAINING METHODS
To elaborate more about training efficiency, we compare our methods with two efficient training methods. Shafahi et al. (2019) proposed Free Adversarial Training that improves training efficiency by reusing the gradient information, which is orthogonal to our approaches and can be easily com- bined with methods pursue efficiency by replacing the PGD-10 training with Free AT.
A22

---

Published as a conference paper at ICLR 2022
Table A12: More results of different sparcification methods CIFAR-100 with VGG-16.
Robust Accuracy
Standard Accuracy
Robust
Sparsity($))
Settings
Best
Final
Diff.
Best
Final
Diff.
Generalization
0
Baseline
22.76
18.06
4.70
46.11
46.88
-0.77
63.18
Random Pruning
22.38
15.76
6.62
41.79
44.85
-3.06
51.15
12.03
OMP
22.98
16.32
6.66
45.45
45.96
0.51
53.59
9.55
SNIP
23.34
17.83
5.51
46.58
48.55
-1.97
40.42
22.76
GraSP
23.05
16.50
6.55
43.01
46.84
3.83
49.71
13:47
80
SynFlow
23.02
17.67
5.35
45.55
47.33
-178
41.70
21.48
IGQ
23.60
17.44
6.16
45.77
47.43
1.66
48.18
15.00
Robust Bird
23.46
17.48
5.98
46.33
47.59
-126
48
15:00
Flying Bird
22.75
17.96
4.79
46.61
47.36
-0.75
48.11
15.07
Flying Bird+
22.92
19.02
3.90
47.01
48.11
- 1.10
34.63
28.55
Random Pruning
21.48
16.33
5.15
43.10
44.93
-188
31.34
31.84
OMP
22.18
17.38
4.80
44.81
45.63
0.82
38.91
24.27
SNIP
22.92
20.30
2.62
48.50
49.05
-0.5
20.02
43.16
GraSP
22.17
17.60
4.57
44.54
47.00
-246
29.76
33.42
90
SynFlow
22.58
18.88
3.70
43.62
46.73
-3.11
24.96
38.22
IGQ
22.55
18.56
3.99
44.96
48.08
33.1
27.91
33.227
Robust Bird
22.80
19.19
3.61
45.78
48.61
-2.83
26.46
36.77
Flying Bird
23.59
18.86
4.73
46.64
48.45
-.81
34.05
29.13
Flying Bird+
23.31
20.34
2.97
45.51
48.13
2.62
22. 16
41.02
Sparsity: 80%
Sparsity: 90%
70
Truin RA
Train RA
Tost RA
60
Test RA
Accuracy (%)
Accuracy (%)
50-
50
40
40
30
30
20
0
25
50
75
100
125
150
175
200
25
75
100
125
150
175
200
Epochs
Epochs
Figure A11: Training curve of Flying Bird+ at 80%(Left) and (0%ght))) sparsity on CIFAR-10 with ResNet-18. 18. The Red and Green dotted lines indicate the time for increasing the pruning and growth ratio, respectively.
Table A13: Comparison results of the unpruned dense network and our flying birds more sparsity levels. Experiments conducted on CIFAR-10 10 with ResNet-18 18 under PGD-10 adversarial training.
Robust Accuracy
Standard Accuracy
Robust
Sparsity% %
Settings
Best
Final
Diff.
Best
Final
Diff.
Generalization
0
Baseline
51.10
43.61
7.49
81.15
83.38
2.23
38.82
40
Flying Bird+
51.25
43.45
7.80
81.51
82.94
-1.43
34.38 4.44
Flying Bird
51.20
43.58
7.62
81.27
83.35
-2.08
35.65
33.17
60
Flying Bird+
51.23
44.95
6.28
81.35
83.19
-1.84
29.89
8.93
Additionally, Li al. (2020) uses magnitude pruning to locate sparse structures, which is similar to OMP reported in Table I except they use smaller learning rate. Our methods achieve better perfor- mance and efficiency than OMP. Specifically, with 80% sparsity, narrower robust generalization gap and 1.54% higher RA yet only requires 87.58% less training FLOPs. Also, methods can be easily combined with Fast AT for further training efficiency.
flying bird+ reaches 4.49%
A23

---

Published as a conference paper at ICLR 2022
Table A 14: Comparison results of the unprumed dense network and our flying birds at more sparsity levels. Experiments conducted on CIFAR-100 with ResNet-18 under PGD-10 adversarial training.
Robust Accuracy
Standard Accuracy
Robust
Sparsity% %
Settings
Best
Final
Diff.
Best
Final
Diff.
Generalization
0
Baseline
26.93
19.62
7.31
52.03
53.91
-1.8
54.56
Flying Bird
26.63
19.80
6.83
53.44
54.46
-1.02
48.66
5.590
40
Flying Bird+
27.35
20.48
6.87
52.34
54.76
-2.42
40.31
114.25
Flying Bird
26.95
20.60
6.35
51.77
54.71
2.94
42.13
112.43
60
Flying Bird+
26.95
21.38
5.57
51.77
55.32
-3.55
34.75
(19.81
Table A15: Comparison results of the unpruned dense network and our flying birds more sparsity levels. Experiments are conducted on CIFAR-10 10 with VGG-16 under PGD-10 adversarial training.
Robust Accuracy
Standard Accuracy
Robust
Sparsity %
Settings
Best
Final
Diff.
Best
Final
Diff.
Generalization
0
Baseline
48.33
42.73
5.60
76.84
79.73
2.89
28.00
Flying Bird
48.03
42.86
5.17
76.28
79.66
-3.38
25.40
2.660
40
Flying Bird+
49.13
43.56
5.57
77.03
79.92
-2.89
23.19 44.81
Flying Bird
48.06
43.69
4.37
78.31
80.11
-1.8
25.55 2.44
60
Flying Bird+
48.41
44.64
3.77
76.45
80.03
-3.58
21.63 6.37
Table A16: Comparison results of the unpruned dense network and flying birds sparsity levels. Experiments are conducted on CIFAR-100 with VGG-16 under PGD-10 adversarial training.
Robust Accuracy
Standard Accuracy
Robust
Sparsity %
Settings
Best
Final
Diff.
Best
Final
Diff.
Generalization
0
Baseline
22.76
18.06
4.70
46.11
46.88
-0.77
63.18
Flying Bird
23.22
18,20
5.02
45.20
46.95
-1.75
59.19
3.99
40
Flying Bird+
23.21
17.90
5.31
45.20
47.13
-1.93
49.40
13.778
Flying Bird
23.53
18.14
5.99
46.03
46.90
-0.87
51.77 111.41
60
Flying Bird+
23.61
17.91
5.70
46.17
47.59
-1.42
49.78 4 133.40
Table A17: Comparison results of the unpruned dense network and our flying birds on CIFAR-10 with WideResNNt-34-10.
Robust Accuracy
Standard Accuracy
Robust Generalization
Sparsity%
Settings
Best
Final
Diff.
Best
Final
Diff.
0
Baseline
54.73
46.83
7.90
84.08
85.84
- -1.76
52.60
Flying Bird
55.34
46.79
8.55
83.76
85.93
-2.17
49.41 13.19
80
Flying Bird+
55.34
46.82
8.52
83.76
85.97
-2.21
46.73 15.87
Flying Bird
54.27
46.16
8.11
85.44
86.01
-0.57
45.41
1
90
Flying Bird+
54.24
46.91
7.33
85.52
85.93
-0.41
39.46
133114
A4.4 COMPARISON WITH OTHER PRUNING AND SPARSE TRAINING METHODS
Compared with the recent work Özdenizei & Legenstein 2021 our flying bird(+) is different at both levels of goal and methodologies. Firstly, Ozdenizei & Legenstein (2021) pursues supe- rior adversarial robust testing accuracy for sparsely connected networks. While we aim to investi- gate the relationship between sparsity and robust generalization, and demonstrate that introducing appropriate sparsity (e.g., L.T--based static sparsity dynamic sparsity) into adversarial training
A24

---

Published as a conference paper at ICLR 2022
substantially alleviates the robust generalization gap and maintains comparable even better stan- dard/robust accuracies. Secondly, Ozdenizci & Legenstein (2021) samples network connectivity flying bird first removes the
from learned posterior to form sparse subnetwork. However, parameters with the lowest magnitude, which ensures a small term of the first-order Taylor approx- imation of the loss and thus limits the impact the output of networks (Evci al.) (020a). And then, it allows new connectivity with the largest gradient to grow to reduce the loss quickly (Evci al., 2020a). Furthermore, we propose an enhanced variant of Flying Bird, i.e., Flying Bird+, which not only learns the sparse topologies but also is capable of adaptively adjusting the network capacity to determine the right parameterization level "on-demand" during training, while Ozdenizei & Legenstein (2021) stick to a fixed parameter budget.
Another work, HYDRA (Sehwag al. 2020) also has several differences from our robust birds. Specifically, HYDRA starts from robust pre-trained dense network, which requires at least hun- dreds of epochs for adversarial training. However, our robust bird's pre-training only needs a few epochs of standard training. Therefore, /Sehwag et al. (2020) has significantly higher computational costs, compared to ours. Then, |Shwaa et al. (2020) adopt TRADES (Zhang et al. 2019) for ad- versarial training, which also requires auxiliary inputs of clean images, while our methods follow the classical adversarial training (Madry et al. 2018b) and only take adversarial perturbed samples input. Moreover, for CIFAR-10 experiments, /Sehwag al. (2020) uses 500k additional pseudo- labeled images from the Timy-ImageNet dataset with a robust semi-supervised training approach. However, all our methods and experiments do not leverage any external data.
Furthermore, one concurrent work (Fu et al. 2021 demonstrates that there exist subnetworks with inborn robustness. Such randomly initialized networks have matching even superior robust accu- racy of adversarially trained networks with similar parameter counts. It's interesting to utilize this finding for further improvement of robust generalization, and we will investigate it in future works.
A25


Reviews:
[{"review_id": "ZlyFfFz4yxq", "reviewer": "Reviewer_EaMs", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nIt is well known that sparsity helps generalizing: from example, already Han 2015 in Fig 5 shows that pruning (simple L2 regularization + thresholding) helps the network generalizing over unseen data. Similarly, the same article shows that refining the surviving parameters after pruning yields better performance, and according to my personal experience it is also true that allowing parameters to enter and exit the pruning pool, i e allowing the pruning mask to evolve, improves performance under multiple points. So, in general those \"findings\" claimed by the article are not that novel according to the existing literature.\nConcerning the experiments, the authors compare with a number of different reference strategies for pruning ratios of 80% and 90%. While I do not put into question the tradeoff between sparsity and performance, the performance corresponding to the sparsity ratio selected by the authors is so low and far beyond typical useful accuracy numbers for unpruned architecture that I cannot avoid questioning the meaningfulness of the reported results. In other words, it is inconclusive for the reader to see  that the proposed method performs x% better than the closest reference when you are in the 40/50% accuracy range for cifar 10.  I would suggest the authors to compare for a sparsity so that the  performance is  closer to more typical values for unpruned architectures.\nIn this context, it is not clear how a reference strategy based on simple L2 regularization without pruning would perform in term of generalization ability and that would be an interesting extra reference to consider.\n\n\n\n[Han 2015] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems, pp. 1135–1143,\n2015b", "summary_of_the_paper": "This paper deals with the problem of training a neural network so that it generalizes well over data unseen at training time. Namely, they address the particular case where a network is trained over an adversarial scheme. This paper proposes two methods for learning a sparse architecture called robust and flying bird. These methods aim at identifying sparse subnetworks arising during early training stages, so to get a pruning mask that eventually yields a sparse architecture (RobustBird). FlyingBird improves over Robust Bird in teh sense that the learning mask can be dynamically adjusted over time, i.e. pruned params may be recovered later on. The authors then experiment at training multiple architectures over different datasets in the experimental section, showing better generalization abilities and lower computational complexity (MACs) for their proposed methods Robust and Flying Birds. the authors conclude that sparsity help networks to generalize better, and as a byproduct it slashes computational complexity.\n", "summary_of_the_review": "This paper addresses a relevant issue, however the main conclusions they draw didn't exactly make me jump on my seat, and while the proposed method may have some originality, the experimental results leave quite a question open about their significance given the considered sparsity/performance tradeoff point the authors consider. I would recommend the authors to revisit their claims under the light of the existing literature dealing with the relationship between sparsity and generalization ability, and to present their experimental results at a higher accuracy target.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "P253FOUCuJU", "reviewer": "Reviewer_dw7x", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "(+) The idea is very interesting and promising. It is reasonable to find sparsity helps both reduce overfitting and improve training efficiency. For adversarial training, that could lead to very significant cost reduction. \n  \n(+) The authors’ study on how to inject sparsity is very comprehensive. They considered two alternatives for sparse adversarial training: a static Robust Bird (RB) training, and a dynamic Flying Bird (FB) training. The former identifies critical mask structure at early training stage, while the latter continues to optimize the mask throughout the entire training. \n\n(+) It is great the authors show that their proposed methods can be combined to boost previous SOTAs. That definitely amplifies their work’s value. \n\n(+/-) Experiments are a bit limited, using only two models (VGG and Res18). However, the aspects being evaluated as well as the ablations are very thorough, and I kinda agree the results mostly suffice to validate their points. \n\n(-) However, the biggest issue I have with experiments is: they did not compare with latest efficient adversarial training methods, such as “Adversarial training for free!”, NeurIPS 2019; and “Towards practical lottery ticket hypothesis for adversarial training” in arXiv 2020. The latter one is also based on sparsity and has certain overlap with Robust Bird. The authors did compare with Fast AT, but that was placed in the Appendix only.\n\n(-) Furthermore, even standard adversarial training or adversarial pruning methods could have been made cheaper, by either early-stopping with less epochs, or using fast/free-AT to replace their more expensive AT sub-modules. The authors need to discuss whether those methods can become their competitive baselines, in a convincing way, not ignoring it. \n\n(-) It is unclear to me whether the final robustness gain comes from the good mask structure or just sparsity itself. In particular, random pruning in Table 1 performs better than I would expect, inviting the aforementioned question. \n\n(-) More sparsity levels should have been tried.\n", "summary_of_the_paper": "This paper studies an important topic of improving adversarial training with different sparsity forms, and the authors made positive discoveries that injecting sparsity properly would make a win-win between efficiency and generalization. \n\nAll findings are not too surprising, but the authors’ effort of presenting a particularly comprehensive empirical study is acknowledged. \n", "summary_of_the_review": "My biggest reservation currently lies in lacking comparison with SOTA efficient adversarial training methods. The value of this work cannot be solely established without that comparison. Overall, I’m currently rating as borderline and would decide my final rating based on the authors’ rebuttal.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "L4Fyu-F8nGV", "reviewer": "Reviewer_75p4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n-\tThe motivation for introducing sparsity to adversarial training for both generalization and efficiency gains are valuable and novel. The paper investigates two sparsity forms: static and dynamic.\n-\tFor the static form, the paper extends the early-bird idea by You et. al. 2020, and show the phenomenon also exists in the adversarial training scheme. So in general the novelty of this paper is only fair, except a surprise finding that even in adversarial training, EB tickets can still be drawn from a cheap standard pre-training stage.\n-\tFor the dynamic form, the authors presented a vanilla version plus an advanced variant capable of adaptively adjusting the sparsity levels. \n-\tExperiments are solid and convincing. Besides PGD, the authors also used Auto-Attack (Croce & Hein, 2020) and CW Attack (Carlini & Wagner, 2017) for a more rigorous evaluation. The authors also carefully excluded obfuscated gradients using transferred unseen attacks\n-\tAttention visualization is another interesting and novel angle to compare different pruning methods.\n-\tPaper is structured well and easy to follow. I especially like that rationale is always clearly presented in company with the algorithms\n-\tAll codes are included, and the reproducibility looks good to me.\n\nWeakness:\n-\tI would appreciate if the authors could elaborate more on why the sparse mask found from a non-robust model could be reused to training a robust model. If that is true, I wonder whether or not there indeed exists any tight coupling between sparse mask structure and robustness.\n-\tTable 1 only have two sparsity levels: 80% and 90%. Why only this two, are they specifically cherry-picked? It would be better if the authors could demonstrate some more sparsity levels.\n-\tTable 1 should also have compared with Early Bird (You et al. (2020) ) and existing dynamic sparse training methods (Evci et al. (2020a); Liu et al. (2021b))\n-\tThis paper does not provide any theoretical analysis on why the proposed strategy should work for efficient adversarial training. It is unclear which factor is the main performance booster: sparsity regularizing the adversarial overfitting, or sparsity for efficient “lottery”-style training. Despite this work being mainly empirical, some theory probing would have improved it.\n-\tTypos: “much more flatter” -> much flatter, and more. Proofreading is required.\n", "summary_of_the_paper": "The authors proposed to leverage static and dynamic sparsity in efficient robust training. The proposed methods can significantly mitigate the robust generalization gap while retaining competitive performance (standard/robust accuracy) with substantially reduced computation budgets.", "summary_of_the_review": "This is an interesting work with solid execution. More experimental clarification and analysis would be welcome.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "63bvTgpjojV", "reviewer": "Reviewer_a6jd", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "It is known that good sparsity can help prevent overfitting as well as reduce inference costs. The main barrier of making sparsity practical for (adversarial) training is the good sparsity pattern itself can be expensive to retrieve. This paper lays out a series of options to mitigate that barrier for adversarial training (AT).\n\nThe authors first demonstrate that good sparse subnetworks can be identified at the very early AT training stage with one-shot pruning, and the remaining stage could focus on training that very compact subnetwork. While similar observations were already drawn by You et. al. 2020 in the standard training, a notably progress made by the authors is that the mask can be located from just the cheap standard training, and it will still incur almost no performance loss when being re-used towards adversarial re-training.  I find it quite intriguing and meaningful.\n\nThe authors continue to investigate the role of sparsity when the sparse connections of subnetworks are on the fly. It allows more flexibility for the network to lower the training loss more, by tweaking not only weights but also topology. The authors further relaxed it to dynamically adjust the network capacity/sparsity to pursue superior robust performance, at a minor sacrifice of training efficiency.\n\nThe authors reported a variety of experiments using two backbones and three datasets. Their proposed methods are found to reduce robust generalization gap and overfitting by 34.44% and 4.02%, with comparable robust/standard accuracy boosts and 87.83%/87.82% training/inference FLOPs savings on CIFAR-100 with ResNet18; and similar competitive performance on other settings. Their proposed approaches can be combined with existing regularizers to yield new state-of-the-art results. The authors also carefully examined their robustness gains against adaptive and transferred attacks. \n\n\n\nHere I have only two nitpicks. First, it would be better if the authors can give some theoretical insights why their strategy works, even with simplified assumptions. Currently the rationales offered are a bit vague. Second, while the FLOPS reduction is impressive, no actual hardware measurements were reported like in the Early Bird paper. Discussing the practical hardware benefits of the proposed strategy would enhance this work’s impact.\n\n=================== post rebuttal ====================\n\nI have read the rebuttal and the rebuttal addresses my concerns.", "summary_of_the_paper": "Recent studies demonstrate adversarial training suffers from severe overfitting besides getting very expensive. This paper proposes to handle the two problems organically altogether, with the tool of sparse training. \n\nThe authors show that injecting appropriate sparsity forms in training could substantially shrink the robust generalization gap and alleviate the robust overfitting, meanwhile significantly saving training and inference FLOPs.\n", "summary_of_the_review": "I am inclined to recommend acceptance, owing to the paper’s novel angle and solid experiment evaluations. I have no major critique.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper focuses on leveraging static and dynamic sparsity in efficient robust training. The proposed methods can significantly mitigate the robust generalization gap while retaining competitive performance (standard/robust accuracy) with substantially reduced computation budgets. The philosophy behind sounds quite interesting to me, namely, sparsity allevating overfitting and improving training efficiency simultaneously. This philosophy leads to two novel algorithms design, i.e., static Robust Bird training, and dynamic Flying Bird training.

The clarity and novelty are clearly above the bar of ICLR. While the reviewers had some concerns on the significance, the authors did a particularly good job in their rebuttal. Thus, most of us have agreed to accept this paper for publication! Please include the additional experimental results in the next version.
Decision: Accept (Poster)