Bag of Tricks for Adversarial Training
Authors: Tianyu Pang
Xiao Yang
Yinpeng Dong
Hang Su
Jun Zhu
Venue: ICLR 2021 Poster (2021)
Keywords: Adversarial Training
Robustness
Adversarial Examples

Abstract: Adversarial training (AT) is one of the most effective strategies for promoting model robustness. However, recent benchmarks show that most of the proposed improvements on AT are less effective than simply early stopping the training procedure. This counter-intuitive fact motivates us to investigate the implementation details of tens of AT methods. Surprisingly, we find that the basic settings (e.g., weight decay, training schedule, etc.) used in these methods are highly inconsistent. In this work, we provide comprehensive evaluations on CIFAR-10, focusing on the effects of mostly overlooked training tricks and hyperparameters for adversarially trained models. Our empirical observations suggest that adversarial robustness is much more sensitive to some basic training settings than we thought. For example, a slightly different value of weight decay can reduce the model robust accuracy by more than $7\%$, which is probable to override the potential promotion induced by the proposed methods. We conclude a baseline training setting and re-implement previous defenses to achieve new state-of-the-art results. These facts also appeal to more concerns on the overlooked confounders when benchmarking defenses.

Published as a conference paper at ICLR 2021
BAG OF TRICKS FOR ADVERSARIAL TRAINING
Tianyu Pang, Xiao Yang, Yinpeng Dong, Hang Su, Jun Zhu
Department of Computer Science & Technology, Institute for AI, BNRist Center Tsinghua-Bose Bosch Joint ML Center, THBI Lab, Tsinghua University, Beijing, 100084 China
1 potyi7,yangxiee yangxiao19, dyp17) emails, wtwiiissssaaiii www.guu..edd.....m Isuhangss, deez ja Metsinghu., on
ABSTRACT
Adversarial training (AT) is one of the most effective strategies for promoting model robustness. However, recent benchmarks show that most of the proposed improvements on AT are less effective than simply early stopping the training pro- cedure. This counter-intuitive fact motivates us to investigate the implementation details of tens of AT methods. Surprisingly, we find that the basic settings (e.g., weight decay, training schedule, etc.) used these methods are highly inconsistent. In this work, we provide comprehensive evaluations on CIFAR-10, focusing on the effects of mostly overlooked training tricks and hyperparameters for adversarially trained models. Our empirical observations suggest that adversarial robustness is much more sensitive to some basic training settings than we thought. For example, slightly different value of weight decay can reduce the model robust accuracy by more than 7%, which is probable to override the potential promotion induced by the proposed methods. We conclude baseline training setting and re-implement previous defenses to achieve new state-of-the-art results'. These facts also appeal to more concerns on the overlooked confounders when benchmarking defenses.
INTRODUCTION
Adversarial training (AT) has been one of the most effective defense strategies against adversarial attacks (Biggio al., 2013; Szegedy et al., 2014; Goodfellow et al., 2015). Based on the primary AT frameworks like PGD-AT (Madry et al., 2018), many improvements have been proposed from different perspectives, and demonstrate promising results (detailed in Sec. 2). However, the recent benchmarks (Croce& & Hein, 2020b; Chen & Gu, 2020) find that simply early stopping the training procedure of PGD-AT (Rice et al., 2020) can attain the gains from almost all the previously proposed improvements, including the state-of-ther thrt TRADES (Zhang et al., 2019b).
This fact somewhat striking since TRADES also executes early stopping (one epoch after decaying the learning rate) in their code implementation. Besides, the reported robustness of PGD-AT in Rice et al. (2020) is much higher than in Madry et al. (2018), even without early-stopping. This paradox motivates us to check the implementation details of these seminal works. We find that TRADES uses weight decay of 2 x 10- Gaussian PGD initialization as 6, ~ (O, ad), and eval mode of batch normalization (BN) when crafting adversarial examples, while Rice et al. (2020) use weight decay of 5 x 10- uniform PGD initialization as óg ~ (((-c,c), and train mode of BN to generate adversarial examples. In our experiments on CIFAR- 10 (e.g.. Table 8), the two slightly different settings can differ the robust accuracy by ~ 5%, which is significant according to the reported benchmarks.
To have comprehensive study, further investigate the implementation details of tens of papers working on the AT methods, some of which are summarized in Table 1. We find that even using the same model architectures, the basic hyperparameter settings (e.g., weight decay, learning rate schedule, etc.) used in these papers are highly inconsistent and customized, which could affect the model performance and may override the gains from the methods themselves. Under this situation, if we directly benchmark these methods using their released code or checkpoints, some actually effective improvements would be under-estimated due to the improper hyperparameter settings.
Our contributions. We evaluate the effects of wide range of basic training tricks (e.g., warmup, early stopping, weight decay, batch size, BN mode, etc.) on the adversarially trained models. Our empirical results suggest that improper training settings can largely degenerate the model performance,
"Corresponding author.
Codee is available https: httpttuub.ccmm/233///gggggg

---

Published as a conference paper at ICLR 2021
Table 1:
Hyperparameter settings and tricks used implement different AT methods on CIFAR-10 We convert the training steps into epochs, and provide code links for reference in Table 11. Compared the model architectures, the listed settings are easy to be neglected and paid less attention to unify.
Total epoch
Batch
Weight decay
Early stop
Warm-up
Method
Lr.
(I.r. decay)
size
(train attack)
(I.r. pertub.)
Madry et al. (2018)
0.1
200 (100, 150)
128
2 x 10
No /No
No/ No
Cai et al. (2018)
0.1
300 (150, 250)
200
5 x 10
No / No
No/ Yes
Zhang et al. (2019b)
0.1
76 (75) 120 (60, 100)
128
2 x 10
Yes No
No / No
Wang et al. (2019)
0.01
128
1 1 10
No / Yes
No / No
Qin et al. (2019)
0.1
110 (100, 105)
256
2 x 10
No / No
No / Yes
Mao et al. (2019)
0.1
80 (50, 60) 100 (cosine anneal)
50
2 x 10
No / No
No / No
Carmon et al. (2019)
0.1
256
5 X 10
No / No
No / No
Alayrac et al. (2019)
0.2
64 43, 51)
128
5 x 10
No/ / No
No / No
Shafahi al. (2019b)
200 (100, 150)
128
2 x 10
No/ / No
No / No
Zhang et al. (2019a)
0.05
105 (79, 90, 100)
256
5 X 10
No / No
No / No
Zhang & Wang (2019)
0.1
200 (60, 90)
60
2 X 10
No / No
No / No
Atzmon et al. (2019)
0.01
100 (50)
32
1 X 10
No / No
No / No
Wong al. (2020)
0-0.2
30 (one eycle)
128
5 X 10
No / No
Yes No
Rice et al. (2020)
0.1
200 (100, 150)
128
5 X 10
Yes No
No / No
Ding et al. (2020)
0.3
128 (51, 77, 102)
128
2 10
No / No
No / No
Pang et al. (2020a)
0.01
200 (100, 150)
50
1 1 10
No / No
No / No
Zhang et al. (2020)
0.1
120 (60, 90, 110)
128
2 10
No / Yes
No No
Huang et al. (2020)
0.1
200 (cosine anneal)
256
5 10
No / No
Yes No
Cheng et al. (2020)
0.1
200 (80, 140, 180)
128
5 10
No / No
No / No
Lee et al. (2020)
0.1
200 (100, 150)
128
2 10
No / No
No / No
Xu et al. (2020)
0.1
120 (60, 90)
256
1 10
No / No
No / No
while this degeneration may be mistakenly ascribed to the methods themselves. We provide a baseline recipe for PGD-AT CIFAR- 10 as an example, and demonstrate the generality of the recipe training other frameworks like TRADES. As seen in Table 16, the retrained TRADES achieve new state-of-the-art performance on the AutoAttack benchmark (Croce & Hein, 2020b).
Although our empirical conclusions may not generalize to other datasets or tasks, we reveal the facts that adversarially trained models could be sensitive certain training settings, which are usually neglected in previous work. These results also encourage the community to re-implement the previously proposed defenses with fine-tuned training settings better explore their potentials.
2 RELATED WORK
In this section, we introduce related work on the adversarial defenses and recent benchmarks. We detail on the adversarial attacks in Appendix A.I.
2.1 ADVERSARIAL DEFENSES
To alleviate the adversarial vuinerability of deep learning models, many defense strategies have been proposed, but most of them can eventually be evaded by adaptive attacks (Carlini & Wagner, 2017b; Athalye al., 2018). Other more theoretically guaranteed routines include training provably robust networks (Dvijotham et al., 2018a;b; Hein & Andriushchenko, 2017; Wong & Kolter, 2018) and obtaining certified models via randomized smoothing (Cohen et al., 2019). While these methods are promising, they currently do not match the state-of-ther -hre--r-t-e-rttt robustness under empirical evaluations.
The idea of adversarial training (AT) stems from the seminal work of Goodfellow al. (2015), while other AT frameworks like PGD-AT (Madry et al., 2018) and TRADES (Zhang et al., 2019b) occupied the winner solutions in the adversarial competitions (Kurakin et al., 2018; Brendel et al., 2020). Based on these primary AT frameworks, many improvements have been proposed via encoding the mechanisms inspired from other domains, including ensemble learning (Tramèr et al., 2018; Pang et al., 2019), metric learning (Mao et al., 2019; Li et al., 2019; Pang al., 2020c), generative modeling (Jiang et al., 2018; Pang et al., 2018b; Wang & Yu. 2019; Deng et al., 2020), semi- supervised learning (Carmon et al., 2019; Alayrac et al., 2019; Zhai et al., 2019), and self-supervised
2

---

Published as a conference paper at ICLR 2021
Table 2: Test accuracy (%) under different early stopping and warmup on CIFAR-10. The model is ResNet-18 (results on WRN-34-10 10 is in Table 14). For early stopping attack iter., we denote, e.g., 40 70 as the epochs increase the tolerance step by one (Zhang et al., 2020). For warmup, the learning rate and the maximal perturbation linearly increase from zero to preset values in 10/ 15 / 20 epochs.
Early stopping attack iter.
Warmup on L...
Warmup on perturb.
Base
40 / 70
40/ 100
60/ / 100
10
15
20
10
15
20
Clean
82.52
86.52
86.56
85.67
82.45
82.64
82.31
82.64
82.75
82.78
PGD-10
53.58
52.65
53.22
52.90
53.43
53.29
53.35
53.65
53.27
53.62
AA
48.51
46.6
46.04
45,96
48.26
48.12
48.37
48.44
48.17
48.48
learning (Hendrycks et al., 2019; Chen al., 2020a;b; Naseer al., 2020). On the other hand, due to the high computational cost of AT, many efforts are devoted to accelerating the training procedure via reusing the computations (Shafahi et al., 2019b; Zhang et al., 2019a), adaptive adversarial steps (Wang et al., 2019; Zhang et al., 2020) or one-step training (Wong et al., 2020; Liu et al., 2020; Vivek B & Venkatesh Babu, 2020). The following works try to solve the side effects (e.g., catastrophic overfitting) caused by these fast AT methods (Andriushchenko & Flammarion, 2020; Li et al., 2020).
2.2 ADVERSARIAL BENCHMARKS
Due to the large number of proposed defenses, several benchmarks have been developed to rank the adversarial robustness of existing methods. Dong et al. (2020) perform large-scale experiments to generate robustness curves, which are used for evaluating typical defenses. Croce & Hein (2020b) propose AutoAttack, which is an ensemble of four selected attacks. They apply AutoAttack on tens of previous defenses and provide a comprehensive leader board. Chen & Gu (2020) propose the black-box RayS attack, and establish a similar leader board for defenses. In this paper, we mainly apply PGD attack and AutoAttack as two common ways to evaluate the models.
Except for the adversarial robustness, there are other efforts that introduce augmented datasets for accessing the robustness against general corruptions perturbations. Mu & Gilmer (2019) introduce MNIST-C with a suite of 15 corruptions applied to the MNIST test set, while Hendrycks & Dictterich (2019) introduce ImageNet-C and ImageNet-P with common corruptions and perturbations images. Evaluating robustness on these datasets can reflect the generality of the proposed defenses, and avoid overfitting to certain attacking patterns (Engstrom et al., 2019; Tramèr & Boneh, 2019).
natural
3 BAG OF TRICKS
Our overarching goal is to investigate how the usually overlooked implementation details affect the performance of the adversarially trained models. Our experiments done on CIFAR-10 (Krizhevsky & Hinton, 2009) under the for threat model of maximal perturbation e = 8/255, without accessibility to additional data. We evaluate the models under 10-steps PGD) attack (PGD-10) (Madry al., 2018) and Attck (AA) (Croce& & Hein, 2020b). For the PGD attack, we apply untargeted mode using ground truth labels, step size of 2/255, and 5 restarts for evaluation no restart for training. For the AutoAttack?, we apply the standard version, with no restart for AutoPGD and FAB, compared to 5 restarts for plus version. We consider some basic training tricks and perform ablation studies on each of them, based on the default training setting as described below:
Default setting. Following Rice et al. (2020), in the default setting, we apply the primary PGD-AT framework and the hyperparameters including batch size 128; SGD momentum optimizer with the initial learning rate of 0.1; weight decay 10-4; ReL.U activation function and no label smoothing: train mode for batch normalization when crafting adversarial examples. All the models are trained for 110 epochs with the learning rate decaying by a factor of 0.1 at 100 and 105 epochs, respectively. We report the results on the checkpoint with the best PGD-10 accuracy.
Note that our empirical observations and conclusions may not always generalize to other datasets or AT frameworks, but we emphasize the importance of using consistent implementation details (not only the same model architectures) to enable fair comparisons among different AT methods.
3

---

Published as a conference paper at ICLR 2021
Table 3: Test accuracy (%) under different batch size and learning rate (l.r.) on CIFAR-10. The basic Lr. is 0.1, while the scaled Lr. is, e.g., 0.2 for batch size 256, and 0.05 for batch size 64.
Table Test aceuracy (%) under differ- ent degrees of label smoothing (LS) on CIFAR- 10. More evaluation results under, egg, PGD-1000 can be found in Table 17.
ResNet-18
ResNet-18
Batch
Basic lr..
Scaled L.r.
LS
Clean
PGD-10
AA
RayS
size
Clean
PGD-10
Clean
PGD-10
0
82.52
53.58
48.51
53,34
64
80.08
51.31
82.44
52.48
0.1
82.69
54.04
48.76
53.71
128
82.52
53,58
0.2
82.73
54.22
49.20
53.66
256
83.33
52.20
82.24
52.52
0.3
82.51
54.34
49.24
53.59
512
83,40
50.69
82.16
53.36
0.4
82.39
54.13
48.83
53.40
WRN-34-10
WRN-34-10
Batch size
Basic lr..
Scaled l.r.
LS
Clean
PGD-10
AA
RayS
Clean
PGD-10
Clean
PGD-10
0
86.07
56.60
52.19
60.07
64
84.20
54.69
85.40
54.86
0.1
85.96
56.88
52.74
59.99
128
86.07
56.60
0.2
86.09
57.31
53.00
60.28
256
86.21
52.90
85.89
56.09
0.3
85.99
57.55
52.70
61.00
512
86.29
50.17
86.47
55.49
0.4
86.19
57.63
52.71
60.64
Table 5: Test accuracy (%) using different optimizers on CIFAR-10. The model is ResNet- (results on WRN-34-10 is in Table 15). The initial learming rate for Adam and AdamW is 0.0001.
Mom
Nesterov
Adam
A
SGD-GC
SGD-GCC
Clean
82.52
82.83
83.20
81.68
82.77
82.93
PGD-10
53.58
53.78
48.87
46.58
53.62
53,40
AA
48.51
48.22
44.04
42.39
48.33
48.51
3.1 EARLY STOPPING AND WARMUP
Early stopping training epoch. The trick of early stopping w.r.t. the training epoch was first applied the implementation of TRADES (Zhang al., 2019b), where the learning rate decays at the 75th epoch and the training stopped at the 76th epoch. Later Rice et al. (2020) provide comprehensive study on the overfitting phenomenon in AT, and advocate early stopping the training epoch as a general strategy for preventing adversarial overfitting, which could be triggered according the PGD accuracy on a split validation set. Due to its effectiveness, we regard this trick as a default choice.
Early stopping adversarial intensity. Another level of early stopping happens the adversarial intensity, e.g., early stopping PGD steps when crafting adversarial examples for training. This trick was first applied by the runner-up of the defense track in NeurIPS 2018 adversarial vision chal- lenge (Brendel et al., 2020). Later efforts are devoted to formalizing this early stopping mechanism with different trigger rules (Wang et al., 2019; Zhang et al., 2020). Balaji al. (2019) early stop the adversarial perturbation, which has a similar effect on the adversarial intensity. In the left part of Table 2, we evaluate the method proposed by Zhang al. (2020) due to its simplicity. As seen, this kind of early stopping can improve the performance on clean data while keeping comparable accuracy under PGD-10. However, the performance under the stronger AutoAttack is degraded.
Warmup w.r.t. learning rate. Warmup w.r.t. learning rate is general trick for training deep learning models (Goodfellow al., 20.6). In the adversarial setting, Wong et al. (2020) show that the one cycle learning rate schedule is one of the critical ingredients for the success of FastAT. Thus, evaluate the effect of this trick for the piecewise learning rate schedule and PGD-AT framework. We linearly increase the learning rate from zero to the preset value in the first 10 15 20 epochs. As shown in the middle part of Table 2, the effect of warming up learning rate is marginal.
Warmup w.rt. adversarial intensity. In the AT procedure, warmup can also be executed w.r.t. the adversarial intensity. Cai et al. (2018) propose the curriculum AT process to gradually increase the adversarial intensity and monitor the overfitting trend. Qin al. (2019) increase the maximal
4

---

Published as a conference paper at ICLR 2021
Table 6: Test accuracy (%) under different non-linear activation function on CIFAR-10. The model is ResNet-18. We apply the hyperparameters recommended by Xie et al. (2020) on ImageNet for the activation function. Here the notation indicates using weight decay of 5 x 10-5, where applying
weight decay of x 10 4 with these activations will lead to much worse model performance.
ReLU
Leaky.
ELU
CELU
SELU
GELU
Softplus
Tanh
Clean
82.52
82.11
82.17
81.37
78.88
80.42
82.80
80.13
PGD-10
53.58
53.25
52.08
51.37
49.53
52.21
54.30
49.12
(a)
57
86
PGD-10 accuracy (%)
56
Clean accuracy (%)
55
84
54
53
x
52
51
X ResNet-18 WRN-34-10
ResNet-18 WRN-34-10
50
10
10
7
Weight decay
Weight decay
(b)
90
60
Clean accuracy (%)
85.
PGD-10 accuracy (%))
55
75-
50
m/w
70
45
65-
111
60
10
10
20
60
100 120
20
40
60
50
100
120
Epochs
Epochs
Figure 1: (a) Test accuracy w.r.t. different values of weight decay. The reported checkpoints correspond to the best PGD-10 accuracy (Rice et al., 2020). We test on two model architectures, and highlight (with red circles) three most commonly used weight decays in previous work; (b) Curves of test accuracy w.r.t training epochs, where the model is WRN-34-10. We set weight decay be 1 x 10 4, 2 10 14-4, and 5 10 -, respectively. We observe that smaller weight decay can learn faster but also more tend to overfit w.r.t. the robust accuracy. In Fig. 4, we early decay the learning rate before the models overfitting, but weight decay of 5 10-1 still achieve better robustness.
perturbation from zero to 8/255 in the first 15 epochs. In the right part of Table we linearly increase the maximal perturbation in the first 10 / 15 20 epochs, while the effect is still limited.
3.2 TRAINING HYPERPARAMETERS
Batch size. On the large-scale datasets like ImageNet (Deng al., 2009), has been recognized that the mini-batch size is an important factor influencing the model performance (Goyal et al., 2017 where larger batch size traverses the dataset faster but requires more memory usage. In the adversarial setting, Xie et al. (2019) use a batch size of 4096 to train a robust model on ImageNet, which achieves state-of-the-an performance under adversarial attacks. dataset, the mini-batch sizes are usually chosen between 128 and 256, as shown in Table 1. To evaluate the effect, we test on two model architectures and four values of batch size in Table 3. Since the number of training epochs is fixed to 110, we also consider applying the linear scaling rule introduced in Goyal et al. (2017), i.e., when the mini-batch size multiplied by k. multiply the learning rate k. We treat the batch size of 128 and the learning rate of 0.1 as a basic setting to obtain the factor k. We can observe that the batch size of 128 works well on CIFAR-18 10, while the linear scaling rule benefit the cases with other batch sizes.
the defenses reported on the CIFAR-10

---

Published as a conference paper at ICLR 2021
Table 7: Test accuracy (%) under different BN modes on CIFAR-10. We evaluate across several model architectures, since the BN layers have different positions in different models.
BN
Model architecture
mode
ResNet-18
SENet-18
DenseNet-121
GoogleNet
DPN26
WRN-34-10
train
82.52
82.20
85,38
83.97
83.67
86.07
Clean
eval
83.48
84.11
86.33
85.26
84.56
87.38
+0.96
+1.91
+0.95
+1.29
+0.89
+1.31
train 53.58
54.01
56.22
53.76
53.88
56.60
PGD-10
eval
53.64
53.90
56.11
53.77
53.41
56.04
+0.06
-0.11
-0.11
+0.01
-0.47
-0.56
train
48.51
48.72
51.58
48.73
48.50
52.19
AA
eval
48.75
48.95
51.24
48.83
48.30
51.93
+0.24
+0.23
-0.34
+0.10
-0.20
-0.26
GoogleNet
PGD-10 1ID-10mm accuracy (%)
DenseNet-121
DenseNet-201
DPN-26
DPN-92
RegNetX (200MF)
RegNetX (400MF)
ResNet-18
ResNet--0
ResNeXt-29
SENet-18
82
83
84
85
86
Clean accuracy (%)
Figure 2: Clean accuracy vs. PGD-10 accuracy for different model architectures. The circle sizes are proportional to the number of parameters that specified in Table 12.
Label smoothing (LS). Shafahi et al. (2019a) propose to utilize LS to mimic adversarial training. Pang et (2019) also find that imposing LS on the ensemble prediction can alleviate the adversar- ial transferability among individual members. Unfortunately, combing LS with standard training cannot prevent the models from evaded by adaptive attacks (Tramer al., 2020) or larger iteration further evaluate the effect
steps (Summers & Dinneen, 2018). Beyond previous observations, of LS on adversarial training. As shown Table and Table 17, mild LS can improve 0.5 ~ 1% robust accuracy under the strong attacks evaluated, including AutoAttack and PGD-1000, without affecting the clean performance. This can be regarded as the effect induced by calibrating the confi- dence (Stutz al., 2020) of adversarially trained models (80% 85% accuracy clean data). In contrast, excessive LS could degrade the robustness (e.g., LS = 0.3 vs. LS 0.4 on ResNet-18), which is consistent with the recent observations in Jiang et (2020) (they LS 0.5). However, since LS is known for its potential gradient masking effect, we advocate careful evaluations when applying this trick on the proposed defenses, following the suggestions in Carlini al. (2019).
Most of the AT methods apply SGD with momentum as the optimizer. The momentum factor is usually set to be 0.9 with zero dampening. other cases, Carmon et al. (2019) apply SGD with Nesterov, and Rice et al. (2020) apply Adam for cyclic learning rate schedule. We test some commonly used optimizers Table as well as the decoupled AdamW (Loshchilov Hutter, 2019) and the recently proposed gradient centralization trick SGD-GC SGD-GCC (Yong al., 2020). can find that SGD-based optimizers (e.g., Mom, Nesterov, SGD-GC SGD-GCC) have similar performance, while Adam AdamW performs worse for piecewise learning rate schedule.
Optimizer.
Weight decay, As observed in Table 1, three different values of weight decay are used in previous defenses, including 10 4 10 and 5 10-4, While 5 10 is fairly widely used value for weight decay in deep learning. the prevalence of the value x 10-1 should stem from Madry al. (2018) in the adversarial setting. In Fig. (a)), report the best accuracy under different values of weight d We can see that the gap of robust accuracy can be significant due to slightly
Note that Rice et al. (2020) also investigate the effect of different weight decay (i.c., & regularization), hut value range of (5 x 10*) where k (-4, -3, -2, 1,
they focus

---

Published as conference paper at ICLR 2021
Table 8: The default hyperparameters include batch size 128 and SGD momentum optimizer. The AT
framework is PGD-AT. We highlight the setting used by the implementation in Rice et al. (2020).
Label
Weight
Activation
BN
Accuracy
Architecture
smooth
decay
function
mode
Clean
PGD-10
AA
0
10
ReLU
train
85.87
49.45
46.43
0
10
ReLU train
86.14
52.08
48.72
0
10
ReLU train
86.07
56.60
52.19
O
5 x 10
ReLU eval
87.38
56.04
51.93
WRN-34-10
O
X 10
Softplus
train
86.60
56.44
52.70
0.1
5 10
Softplus
train
86.42
57.22
53.01
0.1
5 10-4
Softplus
eval
86.34
56.38
52.21
0.2
5 10
Softplus
train
86.10
56.55
52.91
0.2
5 10
Softplus
eval
86.98
56.21
52.10
0
x 10
ReLU train
86.21
49.74
47.58
0
2 X 10
ReLU train
86.73
51.39
49.03
X 10
ReLU train
86.97
57.57
53.26
WRN-34-20
0
5 10
ReLU
eval
87.62
57.04
53.14
0
5 10
Softplus train
85.80
57.84
53.64
0.1
5 10
Softplus
train
85.69
57.86
53.66
0.1
5 10
Softplus
eval
87.86
57.33
53.23
0.2
5 x 10
Softplus train 84.82
57.93
53.39
0.2
5 X 10
Softplus eval
87.58
57.19
53,26
1x10
Weight decay
5x10
Region of correct predictions
Region wrong prodictions
Figure 3: Random normal cross-sections of the decision boundary for PGD-AT with different weight decay. The model architecture is WRN-34-10. Following the examples in Moosavi-Dezfooli et al. (2019), we craft PGD-10 perturbation as the normal direction and r be random direction, under the foo constraint of 8/255. The values of x-axis and y-axis represent the multiplied scale factors.
different values weight decay (e.g., up to ~ 7% for 10 vs. x 10-4). Besides, Fig. 1(b) plot the learning curves of test accuracy w.r.t. training epochs. Note that smaller values of weight decay make the model learn faster in the initial phase, but the overfitting phenomenon also appears earlier. In Fig. 3, visualize the cross sections of the decision boundary. We values of weight decay (e.g., x 10-4) can enlarge margins from decision boundary and improve robustness. Nevertheless, shown in the left columns, this effect is less significant on promoting clean accuracy. As result, weight decay is a critical and usually neglected ingredient that largely influences the robust accuracy of adversarially trained models. In contrast, the clean accuracy much less sensitive to weight decay, for both adversarially and standardly (shown in Fig. 5) trained models.
that proper
Activation function. Most the previous AT methods apply ReLU the non-linear activation function in their models, while Xie et al. (2020) empirically demonstrate that smooth activation functions can better improve model robustness on ImageNet. Following their settings, we test if similar conclusion holds CIFAR-10 10. By comparing the results on ReLU and Softplus in Table 6 (for PGD-AT) and Table 13 (for TRADES), we confirm that smooth activation indeed benefits model robustness for ResNet-18. However, as shown in Table (for PGD-AT) and Table 9 (for TRADES), this benefit is less significant on larger models like WRN. Thus we deduce that smaller model capacity benefit more from the smoothness of activation function. Besides, shown in Table 6, models trained on CIFAR-10 10 seem to prefer activation function (xx) with zero truncation, i.c. (xx) 2 0. Those with negative return values like ELU, LeakyRel Tanh have worse performance than ReL.U.
7

---

Published as a conference paper at ICLR 2021
Table 9: Test accuracy (%). The AT framework TRADES. We highlight the setting used by the original implementation in Zhang et al. (2019b). As listed in Table 16, our retrained TRADES models can achieve state-of the-art performance the AutoAttack benchmark.
Threat model: so constraint, = 0.031
Architecture
Weight decay
BN mode
Activation
Clean
PGD-10
AA
2 x 10
train
ReLU
83.86
54.96
51.52
x 10
eval
ReLU
85.17
55.10
51.85
WRN-34-10
x 10
train
ReLU
84.17
57.34
53.51
10
eval
ReLU
85.34
58.54
54.64
10
eval Softplus
84.66
58.05
54.20
10-4
eval
ReLU
86.93
57.93
54.42
WRN-34-20
10-4
eval Softplus
85.43
57.94
54.32
Threal model: fee constraint, 8 8/555
Architecture
Weight decay
BN mode
Activation
Clean
PGD-10
AA
x 10
train eval ReLU
ReLU
84.50
54.60
50.94
2 x 10-4
85.17
54.58
51.54
WRN-34-10 10
X 10-4
train
ReLU
84.04
57.41
53.83
5 X 10-4
eval
ReLU
85.48
57.45
53.80
5 x 10
eval
Softplus
84.24
57.59
53.88
2 x 10
train
ReLU
84.50
53.86
51.18
2 x 10
eval
ReLU
85.48
53.21
50.59
WRN-34-20
5 x 10-4
train
ReI.U
85.87
57.40
54.22
x 10
eval ReLU
86.43
57.91
54.39
x 10-4
eval Softplus
85.51
57.50
54.21
Model architecture. Su et al. (2018) provide a comprehensive study on the robustness of standardly trained models, using different model architectures. the adversarially trained models, has been generally recognized that larger model capacity can usually lead to better robustness (Madry et al., 2018). Recently, Guo et al. (2020) blend in the technique AutoMI. to explore robust architectures. In Fig. 2, we perform similar experiments more hand-crafted model architectures. The selected models have comparable numbers of parameters. We can observe that DenseNet can achieve both the best clean and robust accuracy, while being memory-efficient (but may require longer inference time). This is consistent with the observation Guo et al. (2020) that residual connections can benefit the AT procedure. Interestingly, Wu al. (2020) demonstrate that residual connections allow easier generation of highly transferable adversarial examples, while in our case this weakness for the standardly trained models may turn out to strengthen the adversarially trained models.
Batch normalization (BN) mode. When crafting adversarial examples in the training procedure, Zhang al. (2019b) use eval mode for BN, while Rice al. (2020) and Madry al. (2018) use train mode for BN. Since the parameters in the BN layers are not updated in this progress, the difference between these two modes is mainly on the recorded moving average BN mean and variance used in the test phase. As pointed out in Xie & Yuille (2020), properly dealing with BN layers is critical to obtain a well-performed adversarially trained model. Thus in Table 7, we employ the train or eval mode of BN for crafting adversaria examples during training, and report the results different model architectures to dig out general rules. As seen, using eval mode for BN can increase clean accuracy, while keeping comparable robustness. We also advocate for the eval mode, because if apply train mode for multi-step PGD attack, the BN mean and variance will be recorded for every intermediate step, which could blur the adversarial distribution used by BN layers during inference.
Takeaways:
(i) Slightly different values of weight decay could largely affect the robustness of trained models;
(ii) Moderate label smoothing and linear scaling rule on l.r. for different batch sizes are beneficial:
(iii) Applying eval BN mode to craft training adversarial examples can avoid blurring the distribution:
(iv) Early stopping the adversarial steps or perturbation may degenerate worst-case robustness;
(v) Smooth activation benefits more when the model capacity is not enough for adversarial training.
8

---

Published as a conference paper at ICLR 2021
Table 10: Test accuracy (%). The considered AT frameworks are FastAT and FreeAT. The model
architecture is WRN-34-10. Detailed settings used for these defenses are described in Sec. 3.5.
Label smooth
Weight
BN
Accuracy PGD-10
Defense
decay
mode
Clean
AA
0
x 10
train
82.19
47.47
42.99
FastAT (Wong al., 2020)
0
10
train
82.93
48.48
44.06
0
x 10
eval
84.00
48.16
43.66
0.1
5 x 10-4
train 82.83
48.76
44.50
0
x 10
train 87.42
47.66
44.24
FreeAT
0
10
train
88.17
48.90
45.66
(Shafahi et al., 2019b)
0
5 X 10
eval
88.26
48.50
45.49
0.1
X 10
train
88.07
49.26
45.91
3.3 COMBINATION OF TRICKS
In the above, we separately evaluate the effect of each training trick in the AT procedure. Now we investigate combining the selected useful tricks, which involve label smoothing, weight decay, activation function and BN mode. As demonstrated in Table 8, the improvements are not ideally additive by combining different tricks, while label smoothing and smooth activation function are helpful, but not significant, especially when apply model architectures with larger capacity.
We also find that the high performance the models trained by Rice et al. (2020) partially comes from its reasonable training settings, compared previous work. Based on these, provide trick list for training robust models on CIFAR-10 for reference.
Baseline setting (CIFAR-10):
Batch size 128; SGD momentum optimizer; weight decay 5 10-4; eval mode BN for generating adversarial examples; warmups are not necessary; moderate label smoothing (0.1 ~ 0.2) and smooth activation function could be beneficial; model architecture with residual connections.
3.4 SE-MPPEMEENATION OF TRADES
As sanity check, we re-implement TRADES to see if our conclusions derived from PGD-AT can generalize and provide the results in Table 9. We can observe that after simply changing the weight decay from 2 x 10-4 to 5 x 10-4 the clean accuracy of TRADES improves by ~ 1% and the AA accuracy improves by 4%. which make the trained model surpass the previously state-of-the- art models reported by the AutoAttack benchmark, as listed in Table 16. This fact highlights the importance of employing a standardized training setting for fair comparisons of different AT methods.
3.5 EVALUATIONS ON OTHER AT FRAMEWORKS
To examine the universality of our observations PGD-AT and TRADES, further evaluate other AT frameworks, including FastAT (Wong et al., 2020) and FreeAT (Shafahi et al., 2019b). We base the FastAT code to implement the methods. Specifically, for FastAT, we use cyclic learning rate schedule with 4min = 0 and mmx 0.2, training for 15 epochs. For FreeAT, we also use cyclic learning rate schedule with Imin
and l 0.04, training for 24 epochs with mini-batch replays be 4. The results are provided in Table 10. We can find that our observations generalize well to other AT frameworks, which verifies that the proposed baseline setting could be decent default choice for adversarial training CIFAR-10.
CONCLUSION
In this work, take step in examining how the usually neglected implementation details impact the performance of adversarially trained models. Our empirical results suggest that compared to clean accuracy, robustness is more sensitive some seemingly unimportant differences in training settings. Thus when building AT methods, we should more carefully fine-tune the training settings (on validation sets), or follow certain long-tested setup in the adversarial setting.

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGEMENTS
This work was supported by the National Key Research and Development Program of China (Nos. 2020AAA0104304, 2017YFA0700004), NSFC Projects (Nos, 61620106010, 62076147, U19B2034, U19A2081), Beijing Academy of Artificial Intelligence (BAAI), Tsinghua-Huawei Joint Research Program, grant from Tsinghua Institute for Guo Qiang, Tiangong Institute for Intelligent Computing. and the NVIDIA NVAIL Program with GPU/DGX Acceleration. Tianyu Pang was supported by MSRA Fellowship and Baidu Scholarship.
REFERENCES
Cean-Baptiste Alayrac, Jonathan Uesato, Po-Sen Huang, Alhussein Fawzi, Robert Stanforth, and Pushmeet Kohli. Are labels required for improving adversarial robustness? In Advances in Neural Information Processing Systems (NeurIPS), pp. 12192-12202, 2019.
Maksym Andriushchenko and Nicolas Flammarion. Understanding and improving fast adversarial training. In Advances neural information processing systems (NeurIPS), 2020.
Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square attack:
query-efficient black-box adversarial attack via random search. In European Conference Computer Vision (ECCV), 2020.
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. In International Conference on Machine Learning (ICML), 2018.
Matan Atzmon, Niv Haim, Lior Yariv, Ofer Israclov, Haggai Maron, and Yaron Lipman. Controlling neural level sets. In Advances in Neural Information Processing Systems (NeurIPS), pp. 4034-2043 2019.
Yogesh Balaji, Tom Goldstein, and Judy Hoffman. Instance adaptive adversarial training: Improved accuracy tradeoffs in neural nets. arXiv preprint
2019
Battista Biggio, Igino Corona, Davide Maiorea, Blaine Nelson, Nedim Šmdii Pavel Laskov, Giorgio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. In Joint European Conference on Machine Learning and Knowledge Discovery Databases, pp. 387-402. Springer, 2013.
Wieland Brendel, Jonas Rauber, and Matthias Bethge. Decision- based adversarial attacks: Reliable attacks against black-box machine learning models. In International Conference on Learning Representations (ICLR), 2018
Wieland Brendel, Jonas Rauber, Alexey Kurakin, Nicolas Papernot, Behar Veliqi, Sharada P Mohanty, Florian Laurent, Marcel Salathé, Matthias Bethge, Yaodong Yu, et al. Adversarial vision challenge. In The NeurlPS' 18 Competition, pp. 129-153. Springer, 2020.
Qi-Zhi Cai, Chang Liu, and Dawn Song. Curriculum adversarial training. In International Joint Conference on Artificial Intelligence (IJCAT), pp. 3740-3747, 2018.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In IEEE Symposium on Security and Privacy (S&P), 2017a.
Nicholas Carlini and David Wagner. Adversarial examples not easily detected: Bypassing ten detection methods. ACM Workshop on Artificial Intelligence and Security (AlSec), 2017b.
Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, and Alexey Kurakin. On evaluating adversarial robustness. arXiv preprint arXiv: /90200 2019.
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, Percy Liang, and John C Duchi. Unlabeled data improves adversarial robustness. In Advances Neural Information Processing Systems (NeurIPS), 2019.
10

---

Published as a conference paper at ICLR 2021
Jinghui Chen and Quanquan Gu. Rays: A ray searching method for hard-label adversarial attack. arXiv preprint prXiv:2006.12799 2020.
Kejiang Chen, Yuefeng Chen, Hang Zhou, Xiaofeng Mao, Yuhong Li, Yuan He, Hui Xue, Weiming Zhang, and Nenghai Yu. Self-supervised adversarial training. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 22222. IEEE, 2020a.
Pin- Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. Zoo: Zeroth order optimization based black-box attacks deep neural networks without training substitute models. In ACM Workshop on Artificial Intelligence and Security (ATSec). ACM, 2017a
Pin-Yu Yu Chen, Yash Sharma, Huan Zhang. Jinfeng Yi, and Cho-Jui Hsieh. Ead: elastic-net attacks to deep neural networks via adversarial examples. In AAAI Conference on Artificial Intelligence (AAAI), 2.18.
Tianlong Chen, Sijia Liu, Shiyu Chang, Yu Cheng, Lisa Amini, and Zhangyang Wang. Adversarial robustness: From self-supervised pre-training to fine-tuning. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 699-708, 2020b.
Yunpeng Chen, Jianan Li, Huaxin Xiao, Xiaojie Jin. Shuicheng Yan, and Jiashi Feng. Dual path networks. In Advances in Neural Information Processing Systems (NeurIPS), pp. 4467-4475, 2017b.
Minhao Cheng, Thong Le, Pin-Yu Chen, Jinfeng Yi, Huan Zhang, and Cho-Jui Hsich. Query-efficient hard-label black-box attack: An optimization-based approach. In International Conference on Learning Representations (ICLR), 2019a.
Minhao Cheng, Qi Lei, Pun-Yu Chen, Inderjit Dhillon, and Cho-Jui Hsich. Cat: Customized adversarial training for improved robustness. arXiv preprint arXiv:2002.06789, 2020.
Shuyu Cheng, Yinpeng Dong, Tianyu Pang. Hang Su, and Jun Zhu. Improving black-box adversarial attacks with a transfer- based prior. In Advances in Neural Information Processing Systems (NeurIPS), 2019b.
Jeremy M Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. International Conference on Machine Learning (ICML), 2019.
Francesco Croce and Matthias Hein. Minimally distorted adversarial examples with fast adaptive boundary attack. International Conference on Machine Learning (ICML), 2020a.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International Conference on Machine Learning (ICML), 2020b.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
Zhijie Deng, Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. Adversarial distributional training for robust deep learning. arXiv preprint arXiv:2002.05999 2020.
Gavin Weiguang Ding, Yash Sharma, Kry Yik Chau Lui, and Ruitong Huang. Mma training: Direct input space margin maximization through adversarial training. In International Conference on Learning Representations (ICLR), 2020.
Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial attacks with momentum. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2018.
Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. Evading defenses to transferable adversarial invariant attacks. In IEEE Conference on Computer Vision and Pattern
examples by Recognition (CVPR), 2019.
11

---

Published as a conference paper at ICLR 2021
Yinpeng Dong, Qi-An Fu, Xiao Yang, Tianyu Pang. Hang Su, Zihao Xiao, and Jun Zhu. Benchmark- ing adversarial robustness on image classification. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Krishnamurthy Dvijotham, Sven Gowal, Robert Stanforth, Relja Arandjelovic, Brendan O' D'Donoghue, Jonathan Uesato, and Pushmeet Kohli. Training verified learners with learned verifiers. arXiv preprint arXiv: 10265, 2018a.
Krishnamurthy Dvijotham, Robert Stanforth, Sven Gowal, Timothy Mann, and Pushmeet Kohli. dual approach to scalable verification of deep networks. In Anmual Conference on Uncertainty in Artificial Intelligence (UAI), 2018b.
Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry. rotation and translation suffice: Fooling enns with simple transformations. In International Conference on Machine Learning (ICML), 2019.
Reuben Feinman, Ryan R Curtin, Saurabh Shintre, and Andrew B Gardner. Detecting adversarial samples from artifacts, arXiv preprint arXiv:1703.00411 2017.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http:
//www. www.ddeepppee... Laaeepllarre rn ngboook org.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations (ICLR), 2015.
Sven Gowal, Chongli Qin, Jonathan Uesato, Timothy Mann, and Pushmeet Kohli. Uncovering the limits of adversarial training against norm-bounded adversarial examples. arXiv preprint arXiv:2010.03593, 2020.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint arXiv:1706.02677. 2017.
Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens Van Der Maaten. Countering adversarial images using input transformations. In International Conference on Learning Representations (ICLR), 2018.
Minghao Guo, Yuzhe Yang, Rui Xu, Ziwei Liu, and Dahua Lin. When nas meets robustness: In search of robust architectures against adversarial attacks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 631-640, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European Conference on Computer Vision (ECCV), pp. 630-645. Springer, 2016.
Matthias Hein and Maksym Andriushchenko. Formal guarantees on the robustness of a classifier against adversarial manipulation. In Advances in Neural Information Processing Systems (NeurIPS), pp. 2266-2276, 2017.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and perturbations. In International Conference on Learning Representations (ICLR), 2019.
Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre-training can improve model robustness and uncertainty. In International Conference on Machine Learning (ICML), 2019.
Jie Hu, Li Shen, and Gang Sun. Squeeze-and--excitatio networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7132-7141, 2018.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 4700-4708, 2017.
Lang Huang. Chao Zhang, and Hongyang Zhang. Self-adaptive training: beyond empirical risk minimization. arXiv preprint arXiv:2002.1033 2020.
12

---

Published as a conference paper at ICLR 2021
Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-box adversarial attacks with limited queries and information. In International Conference on Machine Learning (ICML), 2018.
Haoming Jiang. Zhehui Chen, Yuyang Shi, Bo Dai, and Tuo Zhao. Learning to defense by learning to attack. arXiv preprint 181 2018.
Linxi Jiang, Xingjun Ma, Zejia Weng, James Bailey, and Yu-Gang Jiang. Imbalanced gradients: A new cause of overestimated adversarial robustness. arXiv preprint (xXi::2006.13726 2020.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In The International Conference on Learning Representations (ICLR) Workshops, 2017.
Alexey Kurakin, lan Goodfellow, Samy Bengio, Yinpeng Dong, Fangzhou Liao, Ming Liang, Tianyu Pang, Jun Zhu, Xiaolin Hu, Cihang Xie, et al. Adversarial attacks and defences competition. arXiv preprint Xrv: 00097, 2018.
Sachyung Lee, Hyungyu Lee, and Sungroh Yoon. Adversarial vertex mixup: Toward better adver- sarially robust generalization. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 272-281, 2020.
Bai Li, Shiqi Wang, Suman Jana, and Lawrence Carin. Towards understanding fast adversarial training. arXiv preprint arXiv:2006. 000::000600089, 2020.
Pengcheng Li, Jinfeng Yi, Bowen Zhou, and Lijun Zhang. Improving the robustness of deep neural networks via adversarial training with triplet loss. In International Joint Conference on Artificial Intelligence (IJCAI), 2019.
Guanxiong Liu, Issa Khalil, and Abdallah Khreishah. Using single-step adversarial training to defend itcrative adversarial examples. arXiv preprint arXiv:2002.09632, 2020.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- on Learning Representations (ICLR), 2019.
Xingjun Ma, Bo Li, Yisen Wang, Sarah M Erfani, Sudanthi Wijewickrema, Michael E Houle, Grant Schoenebeck, Dawn Song, and James Bailey. Characterizing adversarial subspaces using local intrinsic dimensionality. arXiv preprint 100100/800.02113 2018
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations (ICLR), 2018.
Chengzhi Mao, Ziyuan Zhong, Junfeng Yang, Carl Vondrick, and Baishakhi Ray. Metric learning for adversarial robustness. In Advances in Neural Information Processing Systems (NeurIPS), pp. 478-489, 2019.
Jan Hendrik Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff. On detecting adversarial perturbations. In International Conference on Learning Representations (ICLR), 2017.
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Jonathan Uesato, and Pascal Frossard. Robust- via curvature regularization, and vice versa. In IEEE Conference on Computer Vision and Patter Recognition (CVPR), 2019.
Norman Mu and Justin Gilmer. Mnist-c: A robustness benchmark for computer vision. arXiv preprint arXiv: 02.337 2019.
Muzammal Naseer, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Fatih Porikli. A self- supervised approach for adversarial robustness. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 262-271, 2020.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks easily fooled: High confidence predictions for unrecognizable images. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 427-436 2015.
13

---

Published as a conference paper at ICLR 2021
Tianyu Pang, Chao Du, Yinpeng Dong, and Jun Zhu. Towards robust detection of adversarial examples. In Advances in Neural Information Processing Systems (NeurIPS), pp. 4579 4589, 2018a.
Tanyu Pang, Chao Du, and Jun Zhu. Max-mahalanobis linear discriminant analysis networks. In International Conference Machine Learning (ICML), 2018b.
Tanyu Pang, Kun Xu, Chao Du, Ning Chen, and Jun Zhu. Improving adversarial robustness via promoting ensemble diversity. In International Conference on Machine Learning (ICML., 2019.
Tianyu Pang, Kun Xu, Yinpeng Dong, Chao Du, Ning Chen, and Jun Zhu. Rethinking softmax cross- entropy loss for adversarial robustness. In International Conference on Learning Representations (ICLR), 2020a.
Tianyu Pang, Kun Xu, and Jun Zhu. Mixup inference: Better exploiting mixup to defend adversarial attacks. In International Conference on Learning Representations (ICLR), 2020b.
Tianyu Pang, Xiao Yang, Yinpeng Dong, Kun Xu, Hang Su, and Jun Zhu. Boosting adversarial training with hypersphere embedding. In Advances in Neural Information Processing Systems (NeurIPS), 2020c.
Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, 7. Berkay Celik, and Ananthram Swami. The limitations of deep learning in adversarial settings. In IEEE European Symposium on Security and Privacy (EuroS&P), P), pp. 372-387. IEEE, 2016.
Chongli Qin, James Martens, Sven Gowal, Dilip Krishnan, Krishnamurthy Dvijotham, Alhussein Fawzi, Soham De, Robert Stanforth, and Pushmeet Kohli. Adversarial robustness through local linearization. In Advances in Neural Information Processing Systems (NeurIPS), pp. 13824-13833, 2019.
Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 104428-10436, 2020.
Edward Raff, Jared Sylvester, Steven Forsyth, and Mark McLean. Barrage of random transforms for adversarially robust defense. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 6528-6537, 2019.
Leslie Rice, Eric Wong, and J Zico Kolter. Overfitting in adversarially robust deep learning. In International Conference Machine Learning (ICML), 2020.
Ali Shafahi, Amin Ghiasi, Furong Huang, and Tom Goldstein. Label smoothing and logit squeezing:
A replacement for adversarial training? arXiv preprint 1rXiv:1910.111 2019a.
Ali Shafahi, Mahyar Najibi, Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! In Advances in Neural Information Processing Systems (NeurIPS), 2019b.
Dawn Song, Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Florian Tramer, Atul Prakash, and Tadayoshi Kohno. Physical adversarial examples for object detectors. In USENIX Workshop Offensive Technologies, 2018a.
Yang Song, Taesup Kim, Sebastian Nowozin, Stefano Ermon, and Nate Kushman, Pixeldefend: Lever- aging generative models to understand and defend against adversarial examples. In International Conference on Learning Representations (ICLR), 2018b
David Stutz, Matthias Hein, and Bernt Schicle. confidence-calibraa adversarial training: Generaliz- ing to unseen attacks. In International Conference on Machine Learning (ICML), 2020.
Dong Su. Huan Zhang, Hongge Chen, Jinfeng Yi, Pin- Yun-Yu& Chen, and Yupeng Gao. Is robustness the cost of accuracy? a comprehensive study on the robustness 18 deep image classification models. In European Conference on Computer Vision (ECCV), 2018.
14

---

Published as a conference paper at ICLR 2021
Cecilia Summers and Michael Dinneen. Logit regularization methods for adversarial robustness. ICLR submission, 2018. https: /pwer///eeeeeeeeeeeeeeeee .nt// / joctX.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, lan Goodfellow, and Rob Fergus. Intriguing properties of neural networks. International Conference on Learning Representations (ICLR), 2014.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1-9, 2015.
Pedro Tabacof and Eduardo Valle. Exploring the space of adversarial images. In 2016 International Joint Conference on Neural Networks (IJCNN). 426-433, IEEE, 2016.
Florian Tramèr and Dan Boneh. Adversarial training and robustness for multiple perturbations. In Advances in Neural Information Processing Systems (NeurIPS), 5858-5868, 2019.
Florian Tramèr, Alexey Kurakin, Nicolas Papemot, Dan Boneh, and Patrick McDanicl. Ensemble ad- versarial training: Attacks and defenses. In International Conference on Learning Representations (ICLR), 2018
Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses. Advances in Neural Information Processing Systems (NeurIPS), 2020.
Jonathan Uesato, Brendan O' O'Donoguue, Aaron van den Oord, and Pushmeet Kohli. Adversarial risk and the dangers of evaluating against weak attacks. In International Conference Machine Learning (ICML), 2018.
S Vivek B and R Venkatesh Babu. Single-step adversarial training with dropout scheduling. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Huaxia Wang and Chun-Nam Yu. A direct approach to robust deep learning using adversarial networks. In International Conference on Learning Representations (ICLR), 2019.
Yisen Wang, Xingjun Ma, James Bailey, Jinfeng Yi, Bowen Zhou, and Quanquan Gu. On the convergence and robustness of adversarial training. In International Conference on Machine Learning (ICML), pp. 6586-6595, 2019.
Eric Wong and Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International Conference on Machine Learning (ICML), pp. 5283-5292, 2018.
Eric Wong, Leslie Rice, and Zico Kolter. Fast is better than free: Revisiting adversarial training. In International Conference on Learning Representations (ICLR), 2020.
Dongxian Wu, Yisen Wang, Shu-Tao Xia, James Bailey, and Xingjun Ma. Skip connections matter: On the transferability of adversarial examples generated with resnets. In International Conference on Learning Representations (ICLR), 2020.
Cihang Xie and Alan Yuille. Intriguing properties of adversarial training at scale. In International Conference on Learning Representations (ICLR), 2020.
Cihang Xie, Jianyu Wang, Zhishuai Zhang, Zhou Ren, and Alan Yuille. Mitigating adversarial effects through randomization. International Conference on Learning Representations (ICLR). 2018.
Cihang Xie, Yuxin Wu. Laurens der Maaten, Alan Yuille, and Kaiming He. Feature denoising for improving adversarial robustness. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Cihang Xie, Mingxing Tan, Boqing Gong, Alan Yuille, and Quoc Le. Smooth adversarial training. arXiv preprint 2006v:2006. /4536, 2020.
15

---

Published as a conference paper at ICLR 2021
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1492-1500, 2017.
Zheng Xu, Ali Shafahi, and Tom Goldstein. Exploring model robustness with adaptive networks and improved adversarial training. arXiv preprini (xXi:20006000330 2020.
Hongwei Yong, Jianqiang Huang, Xiansheng Hua, and Lei Zhang. Gradient centralization: A new optimization technique for deep neural networks. European Conference on Computer Vision (ECCV), 2020.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In The British Machine Vision Conference (BMVC), 2016.
Runtian Zhai, Tianle Cai, Di He, Chen Dan, Kun He. John Hopcroft, and Liwei Wang. Adversarially robust generalization just requires more unlabeled data, arXiv preprint arXiv:1906.00 00555, 2019.
Dinghuai Zhang, Tianyuan Zhang, Yiping Lu, Zhanxing Zhu, and Bin Dong. You only propagate once: Accelerating adversarial training via maximal principle. In Advances in Neural Information Processing Systems (NeurIPS), 2019a.
Haichao Zhang and Jianyu Wang. Defense against adversarial attacks using feature scattering- based adversarial training. In Advances in Neural Information Processing Systems (NeurIPS), pp. 1829-1839, 2019.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. International Conference on Machine Learning (ICML), 2019b.
Jingfeng Zhang, Xilie Xu, Bo Han, Gang Niu, Lizhen Cui, Masashi Sugiyama, and Mohan Kankan- halli. Attacks which do not kill training make adversarial learning stronger. International Conference Machine Learning (ICML), 2020.
16

---

Published as a conference paper at ICLR 2021
A TECHNICAL DETAILS
In this section we introduce more related backgrounds and technical details for reference.
A.1 ADVERSARIAL ATTACKS
Since the seminal L-BFGS and FGSM attacks (Szegedy et al., 2014; Goodfellow al., 2015), large amount of attacking methods on generating adversarial examples have been introduced. In the white-box setting, gradient-based methods are popular and powerful, which span in the lo threat model (Nguyen et al., 2015; Madry et al., 2018), &2 threat model (Carlini & Wagner, 2017a), & threat model (Chen et al., 2018), and lo threat model (Papernot et al., 2016). In the black- box setting, the attack strategies are much more diverse. These include transfer-basee based attacks (Dong et al., 2018; 2019; Cheng al., 2019b), quasi-gradient attacks (Chen et al., 2017a; Uesato al., 2018; llyas et al., 2018), and decision-bbsee attacks (Brendel et al., 2018; Cheng et al., 2019a). Adversarial attacks can be also realized in the physical world (Kurakin et al., 2017; Song et al., 2018a). Below we formulate the PGD attack and AutoAttack that we used in our evaluations.
PGD attack. One of the most commonly studied adversarial attack the projected gradient descent (PGD) method (Madry et al., 2018). Let be randomly perturbed sample in the neighborhood of the clean input x, then PGD iteratively crafts the adversarial example as
clip,,, (x:-1 + sign( V=
(1)
where clip,,e (.) is the clipping function and L is the adversarial objective. The accuracy under PGD attack has been standard metric to evaluate the model robustness.
autoAttack Croce & Hein (2020b) first propose the Auto-PGI (APGD) algorithm, where the main idea is to automatically tune the adversarial step sizes according the optimization trend. As the adversarial objective, except for the traditional cross-entropy (CE) loss, they develop new difference of logits ratio (DLR) loss as
DLR(F,)) y) By max 24
2x, 2Tz
where 2 is the logits and æ is the ordering which sorts the components of 2. Finally, the authors propose to group APGDgE and APGDDLR with FAB (Croce & Hein, 2020a) and square attack (An- driushchenko et al., 2020) to form the AutoAttack (AA).
A.2 REFERENCE CODES
In Table 11, we provide the code links for the referred defenses. The summarized training settings are either described in their papers or manually retrieved by us in their code implementations.
Table 11: We summarize the code links for the referred defense methods in Table 1.
Method
Code link
Madry et al. (2018)
github. wattubboooo/MMaaLLaaaaiiiii challenge
Cai et al. (2018)
github.com//uneeee
Zhang al. (2019b)
Wang et al. (2019)
guthub.com/VisnnWWangyyyaan adv_training
Mao al. (2019)
Carmon et al. (2019)
Alayrac al. (2019)
adversariall training
Shafahi et al. (2019b)
github.
ctttub..oom/asaaffah//rre adv train
Zhang al. (2019a)
github. github.com//l 10000///1110000018000/0/0 Oollyooooaa
Zhang & Wang (2019)
Atzmon al. (2019)
Wong et al. (2020)
Rice et al. (2020)
Ding et al. (2020)
cuttub..com/Borcalll////nnn tariininttrrinnnnnae
Pang et al. (2020a)
guthhb.com/P2333MMaasMhaaaaa
Traiiaan
Zhang al. (2020)
Huang et al. (2020)
Lee et al. (2020)
hittbb.cm/Sseeyyug--Le/diiaiaa ___chllenneeeeeeee
17

---

Published as a conference paper at ICLR 2021
A.3 MODEL ARCHITECTURES
We select some typical hand-crafted model architectures as the objects of study, involving DenseNet (Huang et al., 2017). GoogleNet (Szegedy al., 2015), (PreAct) ResNet (He al., 2016), SENet (Hu et al., 2018), WRN (Zagoruyko & Komodakis, 2016), DPN (Chen et al., 2017b), ResNeXt (Xie et al., 2017), and RegNetX (Radosavovic al., 2020). The models are implemented by https:://// hitpbb....uu. thub. com/ kuangliu wwwpu/uppttchhatteeeeeeeeeeeeeee cifar.
Table 12: Number of parameters for different model architectures.
Architecture
of param.
Architecture
# of param.
Architecture
# of param.
DenseNet-122
28.29 M
DPN26
46.47 M
GoogleNet
24.81 M
DenseNet-201
73.55 M
DPN92
137.50 M
ResNeXt-29
36.65 M
RegNetX (200MF)
9.42 M
ResNet-18
44.70 M
SENct-18
45.09 M
RegNetX (400MF)
19.34 M
ResNet-50
94.28 M
WRN-34-10
193.20 M
A.4 INFERENCE-PHASE ADVERSARIAL DEFENSES
Except for enhancing the models in the training phase, there are other methods that intend to improve robustness in the inference phase. These attempts include performing local linear transformation like adding Gaussian noise (Tabacof Valle, 2016). different operations of image processing (Guo et al., 2018; Xie et al., 2018; Raff et al., 2019) or specified inference principle (Pang et al., 2020b). the other hand, detection- higher-level intervention. Although detection is suboptimal strategy compared to classification, it can avoid over-confident wrong decisions. These efforts include training auxiliary classifiers to detect adversarial inputs (Metzen et al., 2017), designing detection statistics (Feinman al., 2017; Ma et al., 2018; Pang et al., 2018 or basing on additional probabilistic models (Song et al., 2018b).
methods aim to filter out adversarial examples and resort to
A.5 CONCURRENT WORK
Gowal et al. (2020) also provide comprehensive study on different training tricks of AT, and push forward the wate-of-the-aae performance of adversarially trained models on MNIST, CIFAR- 10 and CIFAR- 100. While they analyze some properties that we also analyze in this paper (such as training batch size, label smoothing, weight decay, activation functions), they also complement our analyses with experiments on, e.g., weight moving average and data quality. Both of our works reveal the importance of training details in the process of AT, and contribute to establishing more justified perspectives for evaluating AT methods.
18

---

Published as a conference paper at ICLR 2021
B ADDITIONAL RESULTS
In this section, we provide additional results further support the conclusions in the main text.
B.I EARLY DECAYS LEARNING RATE
As shown in Fig. 1, smaller values of weight decay make the training faster but also more tend to overfit. So in Fig. 4, we early decay the learning rate at 40 and 45 epochs, rather than 100 and 105 epochs. We can see that the models can achieve the same clean accuracy, but the weight decay of 5 x 10 can still achieve better robustness. Besides, in Fig. 5, we use different values of weight decay for standard training, where the models can also achieve similar clean accuracy. These results demonstrate that adversarial robustness is more difficult target than clean performance, and is more sensitive to the training hyperparameters, both for standardly and adversarially trained models.
Clean accurucy (%)
8.5
ww mmmmyr
PGDD-1 10 accuracy
mm Mahhym
75
45.
x 10 (ir dacepear 40 spaci)
x 10 decays it opoch) spoct)
60
10 "tr deaaps specis)
10
decoycar 100 spock)
10 de decass a 100 epochi)
20
60
80
100
120
40
60
80
100
120
Epochs
Epochs
Figure 4: Curves of test accuracy w.r.t. training epochs, where the model is WRN-34-10. Here early decay the learning rate at 40 and 45 epochs for the cases of weight decay 1 x 10 and x 10-1, just before they overfitting. We
that the models can achieve the same clean accuracy weight decay 5 x 10 but still worse robustness.
Clean uvcuracy (%)
90
85
10
75
10
70
100
150 200
Epochs
Figure 5: Curves of test accuracy w.r.t. training epochs. The model architecture is WRN-34-10. 10, and is standardly trained on CIFAR-10. We can observe that the final performance of each model comparable, which means that clean accuracy is less sensitive to different values of weight decay. This observation also holds for the adversarially trained models as shown in Fig. 1.
B.2 THE EFFECT OF SMOOTH ACTIVATION FUNCTION
In Table we test the effect of Softplus and BN mode on ResNet-15 18.
Table 13: Test accuracy (%) of TRADES. We compare with the results in Table to check the effect of smooth activation function TRADES, well the compatibility of it with eval BN mode.
Threat model: to constraint, c
8/255
Architecture
Weight decay
BN mode
Activation
Clean
PGD-10
AA
5 10-1
train ReLU
80.23
53.60
48.96
5 x 10-
train
Softplus
81.26
54.58
50.35
18
5 x 10-4
eval
ReLU
81.45
53.51
49.06
5 10 1
eval
Softplus
82.37
54.37
50.51
19

---

Published as a conference paper at ICLR 2021
B.3 RESULTS OF EARLY STOPPING, WARMUP, AND OPTIMIZERS ON WRN-34-10
In Table 14 and Table 15, we provide the results on WRN-34-10.
Table Test accuracy (%) under different early stopping and warmup on CIFAR-IO. The model is WRN-34-10. For early stopping attack iterations, we denote, e.g. 40 70 as the epochs to increase the tolerance step by one (Zhang et al., 2020). For warmup, the learning rate (I.r.) and the maximal perturbation (perturb.) linearly increase from
the preset value in the first 10/ 15 20 epochs.
Early stopping attack iter.
Warmup on Lr.
Warmup on perturb.
Base
40 / 70
40/ 100
100
10
15
20
15
20
Clean
86.07
88.29
88.25
88.81
86.35
86.63
86.41
86.66
86.43
86.73
PGD-10
56.60
56.06
55.49
56.41
56.31
56.60
56.28
56.25
56.37
55.65
AA
52.19
50.19
49.44
49.81
51.96
52.13
51.75
51.88
52.06
51.70
Table 15: Test accuracy (%) using different optimizers CIFAR-10. The model is WRN-34-10. The initial learning rate for Adam and Adam W is 0.0001, while for other optimizers is 0.1.
Mom
Nesterov
Adam
Adam
SGD-GC
SGD-GCC
Clean
86.07
86.80
81.00
80.72
86.70
86.67
PGD-10
56.60
56.34
52.54
50.32
56.06
56.14
AA
52.19
51.93
46.52
45.79
51.75
51.65
B.4 RANK IN THE AUTOATTACK BENCHMARK
The models evaluated in this paper are all retrained based on the released codes (Zhang et al., 2019b; Rice et al., 2020). Now we compare our trained models with the AutoAttack public benchmark, where the results of previous work are based on the released pretrained models. In Table 16, we retrieve our results in Table 9 the TRADES model where simply change the weight decay from 2 x 10-4 to 10 14m We can see that this scemingly unimportant difference sends the TRADES model back the state-of-the-an position in the benchmark.
Table 16:
We retrieve the results of top-rank methods from https: httptt/tttttttt/////////tttt.t com/ fra31/ auto-attack. All the methods listed below do not require additional training data on CIFAR-10 Here the model of Ours (TRADES) corresponds to lines of weight decay 5 x 10-4 eval BN mode and ReLU activation in Table 9, which only differs from the original TRADES weight decay. We run our methods 5 times with different random seeds, and report the mean and standard deviation.
Threat model: too constraint, = 8/255
Method
Architecture
Clean
AA
Ours (TRADES)
WRN-34-20
86.43
54.39
Ours (TRADES)
WRN-34-10
85.49 + 0.24
53.94 + 0.10
Pang et al. (2020c)
WRN-34-20
85.14
53.74
Zhang et al. (2020)
WRN-34-10
84,52
53.51
Rice et al. (2020)
WRN-34-20
85.34
53.42
Qin et al. (2019)
WRN-40-8
86.28
52.84
Threat model: too constraint, e = 0.031
Method
Architecture
Clean
AA
Ours (TRADES)
WRN-34-10
85.45 + 0.09
54.28 + 0.24
Huang et al. (2020)
WRN-34-10
83.48
53.34
Zhang et al. (2019b)
WRN-34-10
84.92
53.08
20

---

Published as a conference paper at ICLR 2021
B.5 MORE EVALUATIONS ON LABEL SMOOTHING
In Table 17 we further investigate the effect of label smoothing on adversarial training.
Table 17: Test accuracy (%) under different label smoothing on CIFAR-10. The model is ResNet- 18 trained by PGD-AT. We evaluate under PGD-1000 with different number of restarts and step sizes. Here we use the cross-entropy (CE) objective and C&W objective (Carlini & Wagner, 2017a), respectively. We also evaluate under the SPSA attack (Uesato al., 2018) for 10, 000 iteration steps. with batch size 128, perturbation size 0.001 and learning rate of 1 1/255.
Evaluation method
Label smoothing
Attack
Restart
Step size
0
0.1
0.2
0.3
0.4
2/255
52.45
52.95
53.08
53.10
53.14
PGD-1000
2/255
52.41
52.89
53.01
53.04
53.03
(CE objective)
10
2/255
52.31
52.85
52.92
53.02
52.96
0.5/255
52.63
52.94
53.33
53.30
53.25
2/255
50.64
50.76
51.07
50.96
50.54
PGD-1000
5
2/255
50.58
50.66
50.93
50.86
50.44
(C&W objective)
2/255
50.55
50.59
50.90
50.85
50.44
10
0.5/255
50.63
50.73
51.03
51.04
50.52
SPSA-10000
1/255
61.69
61.92
61.93
61.79
61.53
21


Reviews:
[{"review_id": "rl5WQU6Qdc", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "################################ Summary ###################################\n\nThe authors investigate the impact of various training hyperparameters such as weight-decay, batch size, use of batch normalization in eval/ train mode during adversary generation, smooth activations, optimizer and learning rate schedule. The authors demonstrate that use of the right hyperparameters can bring TRADES back to the top of the AutoAttack leaderboard. (~1.6% boost)\n\n################################# Pros ######################################\n\n  -  The paper presents a good review of the tricks used in different adversarial defense papers, and presents the impact of varying each of them on adversarial robustness and clean accuracy \n  -  The authors highlight the importance of using the right training hyperparameters across different baselines for a fair comparison.\n  -  Based on experiments with the PGD-AT model, the authors recommend a set of hyperparameter settings which can potentially generalize to other defenses. \n  -  The authors show that although the proposed settings were found on the PGD-AT model, they generalize well to the TRADES defense as well.\n\n################################# Cons ######################################\n\n  -  The paper lacks novelty, since it merely presents results of existing defenses with different hyperparameters. An analysis/ explanation of why these settings matter more for adversarial training could add value to the submission. \n  -  It is a well known fact that these hyperparameter settings are important for the standard training of Deep Networks. The work by Rice et al. [1] showed that early stopping is important for better adversarial robustness. They also showed the impact of varying settings such as learning rate schedule, weight decay and other regularizers in the supplementary section. This already highlights the importance of selecting the right hyperparameters for adversarial training. Hence, the finding in this paper is not too surprising. \n  -  All the results reported in the paper are for single runs, however they may be a result of variance due to random initialization. Reporting statistics (such as mean and variance) across multiple reruns would be more helpful. \n  -  The final hyperparameter settings suggested in the paper are very similar to those used by Rice et al., with the exception of using Batch normalization in eval mode during adversary generation. Also, the impact of train / eval mode during Batch norm is not consistent across Tables - 7, 8 and 9. Train mode seems to be better for large models on PGD-AT, whereas the eval mode is better for TRADES. \n  -  While the authors show that the hyperparameter settings found on PGD defense generalize well to TRADES, it would have been very useful to show its impact on some of the other defenses listed in Table-1 as well. This would highlight whether the use of a common set of hyperparameters across baselines must be encouraged, or whether it is better to stick to the implementation of the respective authors.\n  -  The paper mentions the use of AutoAttack to evaluate the impact of label smoothing in order to rule out gradient obfuscation. However, AutoAttack does not completely rule out the possibility of gradient masking in a defense [2]. The impact of label smoothing on adversarial robustness is still debatable [3, 4]. The defense by Pang et al. [5] is also broken by Tramer et al. [6]. \n\n[1] Rice et al., Overfitting in adversarially robust deep learning, ICML 2020, https://arxiv.org/abs/2002.11569 \n\n[2] Croce et al., RobustBench: a standardized adversarial robustness benchmark, https://arxiv.org/pdf/2010.09670.pdf\n\n[3] https://openreview.net/forum?id=Bylj6oC5K7\n\n[4] https://openreview.net/forum?id=BJlr0j0ctX\n\n[5] Pang et al., Improving adversarial robustness via promoting ensemble diversity, ICML 2019.\n\n[6] Tramer et al., On Adaptive Attacks to Adversarial Example Defenses, NeurIPS 2020, https://arxiv.org/pdf/2002.08347.pdf \n \n############################### Reasons for score ##############################\n \nAlthough the paper is an interesting read and guide for training adversarial defenses, the key finding of the paper (i.e., hyperparameter tuning can be important for adversarial robustness) is not new to the community, and has been highlighted in recent work [1]. Hence I vote to reject the paper. \n\n##############  Additional Feedback (not part of decision assessment) #####################\n\n  -  Could the authors clarify whether a validation split has been used for early stopping of TRADES in Table-13, and in general for other defenses reported throughout the paper? If yes, what is the size of the validation split? Also, what is the criteria used for early stopping?\n  -  As discussed in the paper, all other defenses in Table-13 also need to be rerun using the optimal settings for a fair comparison. \n  -  It would be useful to include WideResNet architecture with different capacities in Fig.2 of the paper, since this is a common architecture choice across multiple defenses. \n\n##############  Update after rebuttal ############## \n\nI would like to update the score to 6 based on the author's response. I have not increased further due to the limited novelty of the paper. However, the observations in the paper certainly add value to the research community.\n\nI request the authors to consider reporting performance of other defenses in Table-16 using the recommended settings in their final version. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "82ZHkBJXDLm", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper provides a comprehensive evaluations on the different hyper parameter settings of training adversarially robust models. Consequently, this paper also gives a few takeaways based on the results that they find. \n\nThe task of finding good hyper parameters for adversarial training is a challenge. So I like the idea of performing a study on the effects of the different settings. \nSince this paper is evaluating the various settings, I have a few concerns regarding consistency of the experiments performed here. \n\n- This is my major concern of this paper. \nMost of the results shown in the paper are quantitative. Some qualitative understanding of at least one of the settings---like weight decay, would have been very helpful. For example: Visualization of the loss landscape when changing a hyper parameter as shown in Figure 3 in [1] or visualization of the cross section of the decision boundary as shown in Figure 1 in [2]. \nSuch qualitative evaluations would have helped to also translate these findings across datasets. \n\n- Consistency of experimental settings: Table 2 (change in early stopping/warmup on lr/perturb), Table 5 (change in optimizers) and Table 6  (change in activations) report performance on ResNet-18. On the other hand, Table 8 and Table 9 report findings on WRN-34-10/20. \nIt would have been more consistent to check Table 2 and Table 5 with WRN-34-10/20 so that the reader finds exactly what happens when the respective changes are made. \n\n- Reportings: In the abstract as well as the introduction, claims are made without giving the context of the settings. \n“For example, a slightly different value of weight decay can reduce the model robust accuracy by more than 7%, which is probable to override the potential promotion induced by the proposed methods” and \n“In our experiments (e.g., Table 8), we show that the two slightly different settings can differ the robust accuracy by ∼ 4%, which is significant according to the reported benchmarks.”\nIt would be beneficial to the reader to mention the dataset used here. \n\n- A few minor corrections but not limited to: \n“where the learning rate decays at 75 epoch and the training is stopped at 76 epoch.”\nto “where the learning rate decays after 75 epochs and the training is stopped after 76 epochs.” \n\n[1] https://arxiv.org/pdf/1807.10272.pdf\n[2] https://openaccess.thecvf.com/content_CVPR_2019/papers/Moosavi-Dezfooli_Robustness_via_Curvature_Regularization_and_Vice_Versa_CVPR_2019_paper.pdf\n  ", "extra_scores": {}, "extra_text": {}}, {"review_id": "2q69CZ2nh1C", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper systematically reviews different hyperparameter settings and training strategies used for PGD adversarial training on CIFAR-10. Based on that, it derives practical takeaways (“bag of tricks”) and puts forward a standard baseline setting for future work.\n\nI think this work is a valuable contribution to the research on adversarial training. Some of the influencing factors that it examines, like early stopping or learning rates, have been occasionally discussed in prior works, others - like weight decay - seem to have been mostly overlooked. The systematic investigation of the effect of all these different factors leads to practical insights and will hopefully allow researchers to measure advances in adversarial training without confounding effects of different hyperparameter settings. In addition, the rich bibliography gives a great survey of the current state-of-the-art.\n\nHaving said all that, the paper has several limitations which should be either addressed or clearly pointed out:\n- The paper effectively only considers PGD adversarial training on CIFAR-10. In principle this is fine as this is a scenario that has been widely studied and still poses many open research questions. However, I think the limitation to CIFAR-10 should be clearly pointed out in the abstract (currently it’s only mentioned in passing at the end of the introduction), otherwise the percentage values referenced in the abstract are taken out of context, and readers who are interested in adversarial training on larger-scale dataset could be misled (ImageNet is completely outside the scope of this paper). Similarly, the paper does not include some of the recent “free” or “fast” adversarial training protocols in its investigation; again this is fine in principle, and I understand that investigating all the hyperparameters along this additional dimension would dramatically increase the complexity of this study, nevertheless I think it’s a limitation that should be pointed out upfront.\n- More details on the attack default settings should be provided. In particular, for PGD: what is the step size, how many (if any) random (re-)starts were performed? I suppose an untargeted PGD attack was used? Did you use the true or predicted labels in the untargeted attacks? Same for AutoAttack: what were the hyperparameters, in particular for AutoPGD? On a related note, was it necessary to run the whole suite of AutoAttack (including FAB and Square)? PGD adversarial training usually isn’t prone to gradient masking, so I would expect that a plain white-box attack like (Auto)PGD should be sufficient for evaluations and lead to more interpretable results. If gradient masking was an issue for some of the hyperparameter settings, this would be an important insight which should be explicitly pointed out. With AutoAttack, unfortunately, such aspects are hidden in a black-box.\n- In the takeaways on page 7, I wasn’t sure where (iv) had been discussed previously - could you please point me to it?\n- Finally, I’m trying to reconcile the reported results with those from Madry et al. (2017, https://arxiv.org/pdf/1706.06083.pdf) who report 45.8% adversarial accuracy under a PGD attack - which is lower than all except one result reported in this paper (if I’m parsing it correctly). Could you please help me understand the discrepancy or where the datapoint from Madry’s paper should fit in?\n\nA few minor observations:\np.1: “much higher than if in” -> remove “if”\np.1: “These motivate us” -> missing word\np.2: regarding certification methods: calling them “exciting” is a bit casual; also I wouldn’t say that they “cannot match” empirical performance of adversarial training (or do you have a proof?) but rather they “currently do not match” it.\np.3: “with the learning rate decays” -> “decaying”\np.3: “at 75 epoch” and “at 76 epoch” -> “at the 75th epoch” or “at epoch 75” etc.\np.6: “the weight decay in previous work almost falls in three values” -> I think this could be rephrased / explained more precisely\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "0KlHdsIIfo", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper provides an evaluation of different hyperparameter settings for adversarial training. Specifically, it evaluates combinations of warmup, early stopping, weight decay, batch size and other parameters on adversarially trained models. The paper states that its overarching goal is to ``investigate how the implementation details affect the performance of the adversarial trained models''. \n\nStrength\n- The papers extensive empirical results are useful to identify good hyperparameters, and it obtains some interesting findings, such that small differences in weight decay can make a big difference in performance, in contrast to standard performance. \n- It is interesting to point out, as the paper does, that the robust overfitting phenomena and in particular how to obtain good performance with early stopping depends on a number of hyperparameter setups and is quite sensitive to the particular choice of hyperparameters.\n\nWeaknesses\n- The papers novelty is low; essentially it is a rigorous study on how to choose hyperparameters for a specific adversarial training setup (i.e., adversarial training of CIFAR-10).\n- One of the papers main contribution is that ``our empirical results suggest that improper  training settings can largely degenerate the model performance while this degeneration may be mistakenly ascribed to the methods themself''. It is well known a given model only performs well if properly trained and if the hyperparameters are chosen appropriately. \n\nThe paper's value is in identifying good hyperparameters for adversarial training for CIFAR-10. The paper does a very good job in identifying good hyperparameters and setups, but its main finding that ``we should more carefully fine-tune the training settings in adversarial training'' is common sense and in my opinion not a sufficient contribution for a ICLR publication.\n\n---- \nUPDATE: Thanks for the response, I have responded below and kept the score constant. Since I'm currently not actively working on the practice of adversarial robustness, the other reviewers are likely a better judge on the usefulness of the results of the paper for the community.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors have conducted a thorough empirical study on the hyperparameters of representative adversarial training methods. The technical novelty of this paper might be insufficient.  But the empirical findings in this paper explain the strange and inconsistent reported algorithm results in the literature to some extent and remind the necessity and importance of a careful study on hyperparameters. The authors have actively interacted with the reviewers and through the discussions, many unclear issues have been fixed. 

Decision: Accept (Poster)