Towards fast and effective single-step adversarial training
Authors: Pau de Jorge
Adel Bibi
Riccardo Volpi
Amartya Sanyal
Philip Torr
Grégory Rogez
Puneet K. Dokania
Venue: ICLR 2022 Submitted (2022)
Keywords: single-step adversarial training
catastrophic overfitting
FGSM
efficient adversarial training
fast adversarial training

Abstract: Recently, Wong et al. (2020) showed adversarial training with single-step FGSM leads  to  a  characteristic  failure  mode  named catastrophic  overfitting (CO),  in which  a  model  becomes  suddenly  vulnerable  to  multi-step  attacks.   Moreover, they showed adding a random perturbation prior to FGSM (RS-FGSM) seemed to be sufficient to prevent CO. However, Andriushchenko & Flammarion (2020) observed that RS-FGSM still leads to CO for larger perturbations and argue that the  only  contribution  of  the  random  step  is  to  reduce  the  magnitude  of  the  attacks.  They suggest a regularizer (GradAlign) that avoids CO but is significantly more expensive than RS-FGSM. In this work,  we methodically revisit the role of noise and clipping in single-step adversarial training.  Contrary to previous intuitions, we find that not clipping the perturbation around the clean sample and using a stronger noise is highly effective in avoiding CO for large perturbation radii,  despite leading to an increase in the magnitude of the attacks.   Based on these observations, we propose a method called Noise-FGSM (N-FGSM), which attacks noise-augmented samples directly using a single-step. Empirical analyses on a large suite of experiments show that N-FGSM is able to match or surpass the performance of GradAlign while achieving a 3x speed-up.

Under review as a conference paper at ICLR 2022
TOWARDS FAST AND EFFECTIVE SINGLE-STED ADVER- SARIAL TRAINING
Anonymous authors Paper under double-blind review
ABSTRACT
Recently, Wong et al. (2020) showed adversarial training with single-step FGSM leads to characteristic failure mode named catastrophic overfitting (CO),
which a model becomes suddenly vulnerable to multi-step attacks. Moreover. they showed adding a random perturbation prior to FGSM (RS-FGSM) seemed to be sufficient to prevent CO. However, AAndrisschhnnoo & Flammarion (2020) observed that RS-FGSM still leads to CO for larger perturbations and argue that the only contribution of the random step is to reduce the magnitude of the at- tacks. They suggest regularizer (GradAlign) that avoids CO but is significantly more expensive than RS-FGSM. In this work, methodically revisit the role of noise and clipping in single-step adversarial training. Contrary to previous in- tuitions, find that not clipping the perturbation around the clean sample and using a stronger noise is highly effective in avoiding CO for large perturbation radii, despite leading to an increase the magnitude of the attacks. Based on these observations, we propose a method called Noise-FGSM (N-FGSM), which attacks noise- auseeeed samples directly using a single-step. Empirical analyses on large suite of experiments show that N-FGSM is able to match or surpass the performance GradAlign while achieving a 3x speed-up.
INTRODUCTION
Deep neural networks have achieved remarkable performance on variety of tasks (He al. 2015 Silver et al., 2016 Devlin et al.) 2019). However, it is well known that they are vulnerable to small worst-case perturbations around the input data commonly referred to as adversarial exam- ples (Szegedy et al. 2014). The existence of such adversarial examples poses security threat to deploying models in sensitive environments (Biggio & Roli 2018). This has motivated a large body of work towards improving the adversarial robustness of neural networks (Goodfellow et al. 2015
Papernot et al. 2016 Tramer et al.. 2018).
The most popular family of solutions to obtain robust neural networks is based the concept of adversarial training (Goodfellow et al., 2015;; Madry et al.) 2018). In a nutshell, adversarial training be posed min-max problem where instead of minimizing some loss
dataset of clean samples, we augment the inputs with worst-case perturbations that are generated online during training. However, obtaining such perturbations is NP-hard (Weng et al. 2018 and hence, different approaches have been suggested to approximate them. They are commonly referred to as adversarial attacks. In their seminal work, Goodrellow al. (2015) proposed the Fast Gradient Sign Method (FGSM), that generates adversarial attacks by running one step of gradient ascent on the loss function. However, while FGSM- based adversarial training provides robustness against single-step FGSM adversaries, Madry et al. (2018). 8); Tramer et al. (2018) showed that these models were still vulnerable multi-step attacks, namely those allowed to perform multiple gradient ascent steps instead of single one. Notably, |Mdryy et al. (2018) introduced the multi-step Projected Gradient Descent (PGD) attack.
PGD-based attacks have now become the de facto standard for adversarial training; yet, their cost increases linearly with the number of steps. As a result, several works have focused on reducing the cost of adversarial training by approximating the worst-case perturbations with single-step at- tacks (Wong et al.) 2020 Shafahi et al.] 2019) Vivek & Babu 2020). In particular, [Wong et al. 2020) studied FGSM adversarial training and discovered that it suffers from characteristic failure

---

Under review as a conference paper at ICLR 2022
CIFAR10 Dataset
80
relatuv FGSM
FGSM
N-FGSM
Adversarial Accuracy
60
40
E
N-FGSM (ours)
Free:AT AT
20
Gra dAliin
Kim et, al,
k
MultiGrad
RS FGSM
N-FGSM
GradAlign
MultiGrad
ZeroGrad
Free- AT
Kim et. al
RS FGSM
FGSM
RS-FGSM
ZeroGrad
FGSM
12 E for training and evaluation
8
Figure 1: Left: Visualization of FGSM (Goodfellow et al., 2015), RS-FGSM (Wong et al. 2020) and N-FGSM (ours) attacks. While RS-FGSM is limited to noise the c c ball, N-FGSM draws noise from an arbitrary k lo ball. Moreover, N-FGSM does not clip the perturbation around the clean sample. Middle: Comparison single-step methods on CIFAR-10 with PreactResNet18 over different perturbation radii (c is divided by 255). Our method, N-FGSM, can match surpass state- of-the-art results while reducing the cost by a 3x factor. Adversarial accuracy is based on PGD- 50- 10 and experiments are averaged over 3 seeds. Right: Comparison of training costs relative to FGSM baseline based on the number of Porward-Backward passes, see Appendix for details.
mode, in which a model suddenly becomes vulnerable to multi-step attacks despite remaining robust single-step attacks. This phenomenon is referred to catastophicc overfitting. Moreover, they argued that adding a random perturbation prior to FGSM (RS-FGSM) seemed sufficient to prevent catastrophic overfitting and yield robust models. Recently, Andriushchenko & Flammarion (2020) observed that RS-FGSM still Ieads to catastrophic overfitting as we increase the perturbation radii. They suggested a regularizer (GradAlign) that, on the one hand avoids catastrophic overfitting in all the settings they considered, but on the other hand requires the computation of double derivative which significantly increases the computational cost compared to RS-MGSM. This has motivated other works that aim at achieving the same level of robustness with a lower computational overhead (Golgooni et al. 2021 Kim et al. 202T).
In this paper, we revisit two key components that are common among previous works combining noise and FGSM (Tramèr et al. 2018 Wong et al. 2020): the role of noise, i.c. the random step, and the role of the clipping step. In Section robustness; our experiments suggest that adding noise with a large magnitude in the random step and removing the clipping step improves model robustness and prevents catastrophic overfitting, against large perturbation radii. We combine these observations and propose a new method called Noise-FGSM (N-FGSM), an illustration of which is presented in Figure (left). N-FGSM allows to match, or even surpass, the robust accuracy of the regularized FGSM introduced by Andriushchenko & Flammarion (2020), while providing 3x speed-up.
study how these two components affect model
To corroborate the effectiveness of our solution, we present an experimental survey of recently proposed single-step attacks and empirically demonstrate that N-FGSM trades-off robustness and computational cost better than other single-step approaches, evaluated over a large spectrum of perturbation radii (see Figure middle and right panels), over several datasets (CIFAR-10. CIFAR- 100, and SVHN) and architectures (PreActResNer1 8 and WideResNet28-100) We will release our code reproducing all experiments.
2 RELATED WORK
Since the discovery of adversarial examples, many defense mechanisms have been proposed. Pre- processing techniques try to modify the input image to neutralize adversarial attacks (Guo et al.]
2 Buckman et al., 2018 Song et al.] 2018). Adversarial detection methods focus on detecting and rejecting adversarial attacks (Carlini & Wagner 2017 Ma et al. 2018 Yang et al. 2020 Tran
et al. 2021). Certifiable defenses provide theoretical guarantees for the lower bound performance of networks subjected to worst-case adversarial attacks, however, they incur additional costs dur-
ing inference and, empirically, they yield sub-optimal performance (Cohen al. 2019 Wong &
2

---

Under review as a conference paper at ICLR 2022
Kolter 2018
Raghunathan al. 2018 Balunovic& & Vechev 2020). Adversarial training methods
are based on a special form of data augmentation designed to make the network robust to worst-case
perturbations (Zhang et al.) 2019; Athalye et al. 2018 Kurakin et al., 2017). However, comput- ing a worst-case perturbation is an NP-hard problem that needs to be solved at every iteration. To minimize the overhead of adversarial training. Goodfellow al. (2013) proposed FGSM which quires one additional gradient step per iteration. [Tramèr ct al. (2018) first proposed performing a random step before taking the adversarial step (R+FGSM), but they observed that neither method yields robust models against PGD attacks (Madry et al. 2018). Since then, augmenting the training with PGD attacks has been one of the most popular approaches for robustness, but its cost increases linearly with the number of steps, which presents severe practical limitation.
To reduce the cost of PGD, Shafahi et al. (2019) proposed Free Adversarial Training (Free-AT), that exploits single proppapppaattonn step to both update the network parameters and compute the attack. [Wong et al. (2020) explored a variation of R+FGSM, namely RS- RS-FGSM, and showed it can yield robust networks against multi-step attacks. Andriushchenko & Flammarion (2020) found that RS-FGSM only works for limited perturbation radii and introduced GradAlign a regularizer to linearize the loss surface. However, optimizing GradAlign triplicates the computational cost. This motivated a new series of works that aim at matching the performance of GradAlign without the additional computational overhead (Golgooni et al., 2021; Kim et al.] [022]. Other strategies that attempted to improve FGSM included introducing dropout in every layer (Vivek & Babu (0200) and
perturbing intermediate feature maps together with the input (Park & Lee| 2021). Li al. (2020)
suggested combining RS-FGSM and PGD attacks during training. however, the proposed strategy requires a frequent monitoring of the PGD robust accuracy and, in the worst-case, is computationally equivalent to PGD training.
Gilmer et al. (2019); Fawzi et al. (2018) suggested a strong link between robustness to adversarial attacks and to random noise. Motivated by this, we revisit the idea of combining noise and FGSM and propose N-FGSM. Our method is closely related to RS-FGSM, however, we find that using a larger amount of noise and removing the constraint that attacks must lie in the - loo ball is key to obtaining robust models. We note that Kang & Moosavi- Moosavi-Dezfooli (202I) concurrently studied RS-FGSM without clipping, however, opposed to our work, they did not investigate and provide insights on the impact of noise, and the learned models were not robust against large perturbations.
3 PRELIMINARIES ON SINGLE-STED ADVERSARIAL TRAINING
Given a classifier fo X' + Y parameterized by 0 and a perturbation set S, the classifier fe is said to be robust at x E X under S if the following holds for all 6 E S: fo(r + 6) = =f((2) (z). One of the most popular definitions for S is the e fxx ball, i.e. S = {8 |101 I <). This is known as the loc. threat model and is the setting we adopt throughout this work.
To train networks that are robust against fo threat models, adversarial training modifies the classical training procedure of minimizing loss function over a dataset D {(x,, j=)]i of images a; X and labels y/ E y In particular, adversarial training instead minimizes the worst-case loss over the perturbation set S, i.e. trains on adversarially perturbed samples {(2; +8, f=))-i 1IN. When using the loo threat model, we can formalize adversarial training solving the following problem:
min Σ max C(foaa + +8), vi), subject to ||0]| S C,
(I)
where is typically the cross-entropy loss for maasecllasiiiicttoon models. Due to the difficulty of finding the exact inner maximizer, the most common procedure for adversarial training is to approximate the worst-case perturbation through several PGD steps (Madry et al.] 2018). While this has been shown to yield robust models, it comes overhead with the number of PGD steps. As a result, several works have focused on reducing the cost of adversarial training by approximating the inner maximization with single-step attacks.
cost of linear increase in the computational
If we assume that the loss function is locally linear with respect to changes in the input, then the inner maximization of Equation (1) enjoys closed form solution. Goodfellow et al. (2015) leveraged this result to propose the FGSM method, which takes one step in the direction of the sign of the gradient. Tramer et al. (018)) proposed adding a random initialization prior to FGSM. However, both
3

---

Under review as a conference paper at ICLR 2022
methods were later shown to be vulnerable against multi-step attacks, such as PGD (adry al. 2018). Contrary to prior intuition, recent work from Wong et al. (2020) observed that combining a random step with FGSM can actually lead to promising robustness performance. In particular, note that most recent single-step methods approximate the worst-case perturbation that results from solving the inner maximization problem in Equation (1) with the following general form:
8= vs (n I (e,, + 7)) vi)) where 7 ~ Ω
and Q is the distribution from which we draw noise perturbations. For example, when sb is the projection operator onto the &o ball and Ω is the uniform distribution in [-c,c], this recovers RS- FGSM with the following update:
Projiem so (n+ a sign Vv, [(f(z; + m.))) where 7j ~ g~U1- (3)
ÓRS-FGSM
On the other hand, with a different noise setting where $2 = (c a) sign (N(O,, I)) and by choosing the step size to be in [0,e] we recover R+FGSM by Tramèr et al. (2018) that initially explored the idea of combining noise with FGSM but reported no improvement over adversarial training with FGSM. If we consider Ω to be deterministically 0 and & to be the identity map, we recover the FGSM. Finally, if we adjust the choice of the loss function L to include a gradient alignment regularizer, this recovers the GradAlign algorithm by Andriushchenko & Flammarion (2020).
4 NOISE AND FGSM
A common practice when performing adversarial training is to restrict the perturbations used during training to the same c c- lo ball that will be considered at test time. The rationale behind it that increasing the magnitude of perturbations could "unnecessarily" decrease the clean accuracy, since perturbations outside the ball will not be evaluated at test time. For instance, R+FGSM combines the noise step, with magnitude (e a),, and the FGSM step, with magnitude a, in a convex combination manner, thereof, restricting the perturbation to e. On the other hand, Wong et al. (2020) apply clipping operation after the FGSM step the challenge this common practice and explore the two key components in previous single-step methods that limit the magnitude of the perturbations. In particular, operation, i.e. y as a projection to fo ball; and (ii) the source and magnitude of noise for the random step, i.e. Ω. We thoroughly revisit the role of both components on the robustness attained in single- step methods. Throughout this work, unless stated otherwise, we consider noise perturbations sampled from a symmetric Uniform distribution, i.e. Ω u[-k; kJj, where d is the dimension of X' and we refer to k as the "noise magnitude".
experimentally foo ball. In the following,
explore (i) the role of the clipping
4.1 THE ROLE OF NOISE AND CLIPPING ON ROBUSTNESS IN SINGLE-STER METHODS
Clipping Reduces the Effectiveness of Single-Step Perturbations. To study the impact clip- ping has on model robustness, we consider the training of PreActResNer18 (He et al. 2016) on CIFAR-10 (Krizhevsky & Hinton, 2009) with an instance of Equation (2) single-step ad- versarial training. In particular, we consider the case where vé is a projection to the Ex0 ball of size ealip 2e, 3e, 2o). where oc denotes that &a is identity function, i.e. clipping is performed. Moreover, we consider noise sampled from a symmetric uniform distribution where (c, 2e, 4e}. We report in Figure 2 the robust accuracy using PGD-50-10 (i.e. PGD attack with 50 iterations and 10 restarts) with e 8/255. We observe in Figure 2
(left), that for all choices of noise magnitude k; as we expand the clipping fox ball, i.e we increase Ceipp. the adversarial ro- bustness improves. We believe that this is due to the fact that more aggressive clipping, i.e. smaller Eolip. reduces the strength of the computed single-step perturbations during training. To support this intuition, we report in Figure 2 prior to applying the clipping step and after applying the clipping step, with 6elip = c. Moreover, the negative impact of clipping during training is more prominent
(middle) the distribution of the loss measured at perturbed points
increase the noise magnitude k. This is to be expected since, for a fixed a, increasing the noise magnitude & will lead to a prevalence of the noise component over the sign gradient direction in Equation (2)
4

---

Under review as a conference paper at ICLR 2022
Adversaria Accuracy
50
0.125
After clip
Adversarial Accuracy
80
40
0.100
Before clip
60
Density
Noise: le
0.075
40
FGSM (Oc)
30
Noise: 2e
0.050
Noise 1s
20
Noise: 3g
Noise 2
0.025
20
Noise:
Noise 4
1r
38 Radius to clip the perturbation
2E
0.000
10 12
Loss at perturbed point
for trainining and evaluation
Figure 2: Left: N-FGSM + Clipping to different radii (xo means no clipping). As we constraint perturbations by reducing the clipping radius, adversarial accuracy drops. This effect is stronger as we increase the noise magnitude. Thus. clipping seems to have negative impact on robustness. Middle: Histogram of the loss value for perturbations before and after clipping the e Loo ball. There is clear shift in the distributions, which indicates that clipping reduces the adversarial effect of perturbations. Right: N-FGSM when varying the noise magnitude (c divided by 255). Increasing the amount of noise is key to avoiding catastrophic overfitting. For (left) and (right) plots, adversarial accuracy based on PGD-50-10 and experiments are averaged over seeds.
Thus, overall, we observe that clipping during training has negative impact on the robust accuracy. Despite unclipped perturbations may lie outside the e ball, we could observe any significant drop in clean accuracy. In fact, we obtain comparable clean accuracy to GradAlign as later reported in Section 5.5 Further analyses can be found Appendix C
The role of noise in single-step adversarial training. Neither Tramèr et al. (0018) nor Wong al.
2020) explored settings with an increased noise magnitude. Moreover, Andriushchenko & Flammarion (2020) argue that noise in RS-FGSM is not important per se, claiming that its main purpose only reducing the & norm the final perturbation ÓRS-FGSM so that the loss still in the linear regime. However, we empirically find that increasing the noise magnitude is key to avoiding
catastrophic overfitting - see Figure 2 (right).
Investigating further, similarly to Kim et al. (2021). plot the loss surface at the end of train- ing (see Figure 12 in Appendix D and find that, as observed by Kim et al. (202T), the loss surface of models trained via FGSM RS-FGSM appears distorted at the end of training, i.e. the loss increases sharply along the FGSM direction, but then it rapidly decreases back to the same loss val- ues associated with the clean sample. This is consistent with the gradient obfuscation observed by Tramer et al. (2018. However, when training without clipping with an increased noise magnitude, observe non-distorted loss surface, i.e. the loss gradually increases along the FGSM direction. Interestingly, we observe a similar effect when training with the GradAlign regularizer. Thus, com- bining noise with FGSM seems have regularizing effect that encourages the loss surface to be locally linear in a similar spirit to GradAlign. Note this is based on the empirical analyses and visual inspections that we performed. We do not provide theoretical justification behind this.
Despite the clear benefits of increasing the noise magnitude, we do observe slight but consistent decrease in robustness as we keep on increasing the noise, which we hypothesize due to over- regularization. We find k 2e to be the sweet spot in most of our experiments; we do not exclude that a more extensive hyperparameter tuning procedure may lead to improved results.
4.2
OUR APPROACH
In the previous section, we provided experimental analyses to show that increasing the noise magni- tude and not clipping prevents catastrophic overfitting and improves robustness significantly. Based on these observations, propose the following, simple and efficient, single-step method for adver- sarial training that we denote as Noise-FGSMM (N-FGSM):
ÓN-FGSM
sign ( [(fo(a; + 1),, v1),, ~~ 1~ kj
We detail our full adversarial training procedure in Algorithm
We now theoretically analyse N- N-FGSM to understand the role noise in single-step approaches. According to Andrisshchenkoo & Flammarion 2020), the main contribution
Theoretical insights
5

---

Under review as a conference paper at ICLR 2022
Algorithm N-FGSM adversarial training
1: Inputs: epochs T, batches M, radius e, step-size (default:e), noise magnitude k (default:2e).
2: for / 1,..., do
3:
for i = 1....,M. M do
4:
Perform N-FGSM adversarial attack
5:
Uniform[- k]d
6:
8 + sign( V [[fa((; v)) V = Fo (zę - 8), y:)
7:
8:
optimizer O Ve) standard weight update, (e.g. SGD)
9: end for
end for
of noise in single-step adversarial training is to reduce the effective & norm of the perturbation 6. Since N-FGSM does not involve clipping, we find the expected norm squared of N-FGSM perturba- tions to be larger than RS-FGSM perturbations. In fact, it is even larger than the FGSM perturbations which always lie the foo ball. These observations are formalized in Theorem T
Theorem 1. Let ÖX-FGSM be proposed single-ster step method defined by Equation (4) 8FGSM be the FGSM method (Goodfellow et al.) (015)) and ögs FGSM be the RS-FGSM method (Wong et al. 2020).
Then, with default hyperparameter values and for any > we have that
Eng [ISSM Eyy [[]6FasM]]
E, [[[ōRS.FGSM]
We present the Proof in Appendix where we also compute an empirical estimation of the expected Eg norm of different methods via Monte Carlo sampling and find them align with Theorem [ Differently from the hypothesis of Andriushchenko & Flammarion (002)), we find that despite that N-FGSM perturbations have larger l norms, they yield robust models even under larger e radii, for which RS-FGSM or FGSM fail to catastrophic overfitting. We believe that these contradic- tory observations can lead better understanding on the role of noise in adversarial training and catastrophic overfitting in future work.
5 EXPERIMENTS AND ANALYSES
We compare N-FGSM against several adversarial training methods, considering broad range of loo radii. Following Wong et al. (200)); Andriushchenko & Flammarion (020). we measure adversarial robustness CIFAR- 10/ 100 (Krizhevsky & Hinton 2009) and SVHN (Netzer et al.)
2011) datasets with PGD-50-10 attack - PGD (Madry et al. 2018 with 50 iterations and 10 restarts.
5.1 COMPARISON TO OTHER SINGLE-STEP METHIOSS
We start by comparing N-FGSM against other single-step methods. Note that not all single-step methods are equally expensive, since they may involve more or less computationally demanding operations. For instance, GradAlign (Andriushchenko Flammarion (2000) relies on a regularizer
that is considerably expensive, while, MultiGrad (Golgooni et al. (222)) requires evaluating the input gradients on multiple random points. For a comparison of traming cost over different single- step methods, we refer Figure (right). Following the standard practice (Wong al.) 2020 Andriushchenko & Flammarion 2020), we use a PreactResNetl 8 architecture (He et al. 2016).
We use RS-FGSM and Free-AT with the settings recommended by WWng al. (2020) We apply GradAlign following the hyperparameters reported in the official repository Golgooni et al. (2021) recommend applying MultiGrad with rameter setting for what concerns the ZeroGrad variant. Also Kim et al. (202T) do not recommend of hyperparameters, for a fair comparison, we ablate them and select the ones that provide the highest adversarial accuracy (for every combination of and dataset). We train on CIFAR- for 30 epochs and on SVHN for 15 epochs with cyelic learning rate. Only for Free-AT, we use 96 and 48 epochs for CIFAR-10/100 and SVHN, respectively, to obtain comparable results following
random samples, but do not provide default hyperpa-

---

Under review as a conference paper at ICLR 2022
CIFAR100 Dataset
SVHN Dataset
50
N-FGSM (ours)
Free-AT
Adversaria Accuracy
GradAlign
Kim et; al.
Adversarial Accuracy
40
Grad
RS-FGSM
ZeroGrad
FGSM
60
30
40
20
10
20
0
12 8 for training and evaluation
10
14
8
12
for training and evaluation
Figure 3: Comparison of single-step methods on CIFAR-100 (left) and SVHN (right) with Preac- (ResNet18 over different perturbation radius (e divided by 255). Our method, N-FGSM, can match or surpass state-of-the-art results while reducing the cost by 3x factor. Adversarial accuracy is based on PGD-50-10 and experiments are averaged over seeds. Legend shared among plots.
Wong et al. 2020) and (Andriushhhenko & Flammarion (2020). CIFAR-10 results are presented Figure (midlee), whereas CIFAR- 100 and SVHN results reported in Figure 3
As previously observed, FGSM and RS-FGSM both suffer from catastrophic overfitting on larger attacks. In contrast, N-FGSM prevents catastrophic overfitting and enjoys robustness properties comparable or superior to GradAlign for all c, while being times faster. With appropriate hyper- parameters, ZeroGrad is able to avoid catastrophic overfitting but obtains sub-obtimal robustness especially for large perturbations. Neither MultiGrad nor the method proposed by Kim et al. (2021) avoid catastrophic overfitting in all settings. We also observe that Free-AT AT cannot overcome catastrophic overfitting as also observed by Andrisscchenko & Flammarion 2020).
5.2
RANDOMIZED ALPHA
Kim et al. (2021) evaluate intermediate points along the RS-FGSM direction in order pick the "optimal" perturbation size. However, we find that increasing the number of intermediate evaluated points does not necessarily lead to increased adversarial accuracy. Moreover, for large perturbations we could not prevent catastrophic overfitting even with twice the number of evaluations tested by test very simple baseline where instead of evaluating in-
Kim al. (022I). This motivates termediate steps, the RS-FGSM perturbation size is randomly selected as: 5 = L ÔRS-FGIM where t [0 1]d Interestingly, as reported in Figure (left), we find that this very simple baseline, dubbed RandAlpha, is able to avoid catastrophic overfitting for all values of and outperforms Kim al. (202T) on CIFAR-10. This is aligned with our main finding that combining noise with adversar- ial attacks is indeed powerful tool that should be explored more thoroughly before developing more expensive solutions. We reach the same conclusions for CIFAR- 100 and SVHN in Appendix H
5.3 HYPERPARAMETER SELECTION
While FGSM relies on fixed step-size (equal to the maximum radius of perturbation to be used at test time, i.c. a = e), Wong et al. (2020) explored different values for during the development of RS-FGSM finding that an increase of the step-size improves the adversarial accuracy up to a magnitude before catastrophic overfitting occurs. We also ablate the value of a for N-FGSM in Figure 4 (middle). We find that by increasing the noise magnitude, N-FGSM can use larger values than RS-FGSM, without suffering from catastrophic overfitting. As observed by Woon et al. 2020), this in turn leads to increase in the adversarial accuracy the expense of a decrease in the clean accuracy. In light of this trade-off and following FGSM, we also use a = for N-FGSM.
Regarding N-FGSM noise hyperparameter, we find A 2e works in all but one SVHN experiment (c = 12, in which we set & = 3c), reducing the need for expensive hyperparameter tuning. In comparison. GradAlign regularizer hyperparameter for every radius with a noticeable shift between CIFAR-10 and SVHN hyperparameters, suggesting they may require additional tuning when applied to novel datasets.
ZeroGrad quantile value need to be defined
7

---

Under review as a conference paper at ICLR 2022
80
90
Comparison of Training Schedules
Adversaria Accuracy
N-FGSM
Adversarial Accuracy
RandAlpha Kim et. al.
80
Clean Ace
Robust Acc
60
70
Long schedule: Final model
N-FGSM robust
40
60
N-FGSM clean
83.18 0.11
36.56 + 0.26
Long schedule: Best model
20
50
80.8 + 0.36
48.48 + 0.27
40
Fast schedule: Final model
0
8 10 14 E for trainining and evaluation
16
N-FGSM step size a (E 8)
6
10
12 14
80.58 + 0.22
48.12 12 + 0.07
(202)) with RandAlpha, our baseline where we multiply the RS-FGSM perturbation by a value drawn uniformly in [0, 1], on CIFAR-10 and PreActResNetl (c is divided by 255). RandAlpha does not incur the extra cost of evaluating intermediate steps and does not require hyperparameter tuning. Middle:
Figure 4: Left: Comparison of Kim et al.
Ablation of step size a in N-FGSM for we increase the magnitude of the FGSM perturbation we observe an increase in robustness coupled with a drop on the clean accuracy. Right: Comparison of the "fast" training schedule from Wong et al. (2020) and "long" training schedule described in Rice et al. (2020). N-FGSM shows robust oberfitting but not catastrophic overfitting with the long schedule. Adversarial accuracy is based on PGD-50-10 10 and experiments are averaged over seeds.
As
5.4 LONG VS FAST TRAINING SCHEDULE
Throughout our experiments, we used the RS-FGSM training setting introduced in Wong et al. (020). However, Rice et al. (2020) suggest that longer training schedule coupled with early stopping may lead to a boost performance. Kim et al. (02T)) and Li et al. (2020) report that longer training schedules increase the chances of catastrophic overfitting for RS-FGSM and that this limits its performance. We test the longer training schedule with N-FGSM and find that presents robust overfitting, i.e. the adversarial accuracy the training set keeps increasing but it decreases when evaluated on the test set as described in Rice et al.] (2020). However, it does not suffer from catastrophic overfitting. In Figure (right), show the results for 8. Although we do observe a slight increase in performance when using the long training schedule, we find the performance remarkably competitive when considering the fast one which seems to avoid robust overfitting (when relying on early stopping, the best models are usually found at the end). Thus, it might be preferable to consider the fast training schedule if computational cost an important factor. In Appendix schedule compared to 48.48 + 0.27 for N-FGSM. Note that, in order prevent GradAlign from suffering from catastrophic overfitting for 8, we had to increase the regularizer hyperparameter (compared to the fast schedule), while N-FGSM is able to prevent catastrophic overfitring with the same settings.
we report robust accuracy of 47.86 + 0.1 for GradAlign with the longer
5.5 COMPARISON TO MULTI-STEP ATTACKS
In Section 5.1
we compared the performance of single-step methods and observed that N-FGSM able match or surpass the state-of-the-arr method, i.c. GradAlign. while reducing the com- putational cost by factor of 3x. In this section, we compare the performance of N-FGSM with multi-step attacks. We use PGD-2 with 2020) and keep the same training settings as described in Section 5.1 (note that PGD-x denotes PGD attack with iterations and no restarts).
s/2 and PGD-10 with a 2 following Wong et al.
In Figure 5 we observe that PGD-2 also presents catastrophic overfitting when we increase the perturbation radius e, which is consistent with results reported by /Andriushhhenko Flammarion (2020). On the other hand, despite all methods achieving comparable clean accuracy, there is gap on adversarial accuracy between PGD-10 and single-step methods which grows with the perturba- tion size. This can be partially expected since the search space grows exponentially with and PGD can explore it more thoroughly as we increase the number of iterations. Nevertheless, computing a PGD-10 attack is 10x more expensive than computing an N-FGSM one. An important direction for future work should be addressing this gap and analyse. both theoretically and empirically, whether single-step methods can actually match the performance of their multi-step counterparts.
8

---

Under review as a conference paper at ICLR 2022
CIFAR10 Dataset
SVHN Dataset
100
80
Clean and Adv. Acc.
80
Clean and Adv
60
60
40
40
N-FGSM
N-FGSM
20
GradAlign
20
GradAli gn
PGD-10
PGD-10
PGD-2
PGD-2
2
10 12
16
for trainining and evaluation 12
& for trainining and evaluation
Figure 5: Comparison of N-FGSM and GradAlign with multi-step methods on CIFAR- 10 (Left) and SVHN (Right) with PreactResNet18 over different perturbation radii (e is divided by 255). Despite all methods achieving comparable clean accuracy (dashed lines), there is gap in robust accuracy between PGD-10 10 and single-step methods. However, note that PGD-10 is 10x more expensive than N-FGSM. Adversarial accuracy is based on PGD-50-16 10 and experiments are averaged over 3 seeds.
5.6 EXPERIMENTS WITH WIDERESNET28-1 10 ARCHITECTURE
We also compare the performance of methods on WideResNer28-10 10 (Zagoruyko & Komodakis 2016) architecture in Figure 7 and Figure g (Appendix B). As in the experiments with the PreAc- 1ResNet18 architecture, N-FGSM obtains state-of-the-art PGD-50-10 accuracy among single-step methods. Nevertheless, difficult to prevent when using WideResNet. For instance, FGSM is able to consistently yield robust models up
general trend, we observe that catastrophic overfitting seems to be more
6 for PreActResNet118 can lead to catastrophic overfitting for WideResNet models. We hypothesize this is because it is has 36.5 M parameters, whereas PreActResNet18 has 11.2M. Regarding GradAlign, we had to increase the regularizer hyperparameter (compared to the settings for PreActResNet18) 88 order prevent catastrophic overfitting on CIFAR- Note that, to our surprise, we could not find competitive hyperparametter setting for GradAlign on the SVHN dataser for e 2 We tried both increasing the regularizer hyperparameter and decreasing the step size a, but some or all runs led to models close to constant classifier for each setting. We do not claim that GradAlign will not work, but finding a good configuration might require further tuning.
CIFAR-10, however, for some runs the same radius
more over-parametrized -WideResNer28:
On the other hand, as observed earlier, the default configuration for N-FGSM (a works well all settings except for = 16 on CIFAR-10 and = 10, 12 on SVHN. For CIFAR-IO. we increase the noise magnitude to k 4e. For SVHN, we find that decreasing as we tried for GradAlign works better than increasing the noise. However, in both cases N-FGSM can obtain more than trivial adversarial accuracy. Results presented in Appendix B
k 2c)
DISCUSSION
In this work, we explore the role of noise and clipping in single-step adversarial training. Contrary to previous intuitions, we show that increasing the noise magnitude and removing the e e Coo constraint allows improving adversarial robustness, while maintaining a competitive clean accuracy. These findings led us to propose N-FGSM, simple and effective approach that can match or surpass the performance of GradAlign (Andriushchenko & Flammarion 2020), while achieving a 3x speed-up.
We perform an extensive comparison with other relevant single-step methods, observing that all of them achieve sub-optimal performance and most of them are not able to avoid catastrophic overfit- ting for larger attacks. Moreover, we also analyze recent single-step methods and - inspired by Kim et al. (201)) - observe that uniformly choosing a step-size in [0, ai) avoids catastrophic over- fitting for RS- which reinforces our intuition that random noise is a powerful tool towards learning robust models. However, despite impressive improvements of single-step adversarial train- ing methods, there is still a gap between single-step and multi-step methods such PGD-10 increase the e radius. Therefore, future work should put an emphasis on formally understanding the limitations of single-step adversarial training and explore how, if possible, this gap can be reduced.
9

---

Under review as a conference paper at ICLR 2022
Ethics Statement. The existence of adversarial examples poses a potential threat to the deploy- ment of deep learning systems into the real world. Therefore, finding fast and effective methods to train models robust against this threat is of utmost importance. On the other hand, adversarial training methods are based on augmenting samples with adversarial perturbations, thus, they are partially based on building methods to attack neural networks. Research adversarial attacks is naturally sensitive path, since it can potentially be exploited for unethical purposes. However, the designing stronger attacks; rather, the methods we propose are designed ad
scope of our work hoc to improve the adversarial robustness of learning systems, not to break other models' defenses. Therefore, although we are aware that we are carrying out research within a sensitive topic, we are not particularly concerned that the research presented in this paper can lead to harmful applications on the contrary, we believe that it can help deploying safer machine learning applications.
Reproducibility Statement. In this paper, we compare against several adversarial training meth- ods. The general training and evaluation settings used described at the beginning of Section S Moreover, different hyperparameters have been used for each method depending on the dataset and network. These are detailed in the manuscript, where corresponding results are discussed. For all the reported experiments, we employ different random seeds to ensure replicability of our results, and report performances indicating the standard deviation values. To facilitate further research on this topic, as stated in Section [ we will release the code to reproduce all experiments. Regarding our theoretical results, we deferred the proofs to Appendix
REFERENCES
Maksym Andriushchenko and Nicolas Flammarion. Understanding and improving fast adversarial training. In Neural Information Processing Systems (NeurIPS), 2020.
Anish Athalye, Nicholas Carlini, and David A. Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In International Conference Machine Learning (ICML), 2018
Mislav Balunovic and Martin Vechev. Adversarial training and provable defenses: Bridging the gap. In International Conference on Learning Representations (ICLR), 2020.
Battista Biggio and Fabio Roli. Wild patterns: Ten years after the rise of adversarial machine learning. Puttern Recognition, 2018.
Jacob Buckman, Aurko Roy, Colin Raffel, and Ian Goodfellow. Thermometer encoding: One hot way to resist adversarial examples. In International Conference on Learning Representations (ICLR), 2018
Nicholas Carlini and David Wagner. Adversarial examples are easily detected: Bypassing ten detection methods. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, 2017.
Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In International Conference on Machine Learning (ICML), 2019.
Jacob Devlin, Ming- Wei Changm, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Annual Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL HLT), 2019.
Alhussein Fawzi, Seyed-Mohser Moosavi-Dezzae Dezaooi,, Pascal Frossard, and Stefano Soatto. Empirical study of the topology and geometry of deep networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Justin Gilmer, Nicolas Ford, Nicholas Carlini, and Ekin Cubuk. Adversarial examples natural consequence of test error in noise. In International Conference on Machine Learning (ICML), 2019.
10

---

Under review as a conference paper at ICLR 2022
Zeinab Golgooni, Mehrdad Saberi, Masih Eskandar, and Mohammad Hossein Rohban. Zerograd: Mitigating and explaining catastrophic overfitting in fgsm adversarial training. arXiv:2103.
[cs.LG], 2021.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and hamessing adversarial examples. International Conference on Learning Representations (ICLR), 2015.
Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens van der Maaten. Countering adversarial
images using input transformations. In International Conference on Learning Representations (ICLR), 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpass- ing human-level performance imagenet classification. In IEEE International Conference on Computer Vision (ICCV), 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European Conference on Computer Vision (ECCV), 2016.
Peilin Kang and Seyed-Mohsen Moosavi-Dezfooli. Understanding catastrophic overfitting in adver- sarial training. WwXiv:2/05.02944 [Cs, 2021.
Dezfooli.
Hoki Kim, Woojin Lee, and Jaewook Lee.
Understanding catastrophic overfitting in single-step adversarial training. In AAAI Conference on Artificial Intelligence (AAAI), 2021.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Mas- ter's s thesis, Department of Computer Science, University of Toronto, 2009.
Alexey Kurakin, lan Goodfellow, and Samy Bengio. Adversarial machine learning at scale. 2017.
Bai Li, Shiqi Wang, Suman Jana, and Lawrence Carin. Towards understanding fast adversarial training, arXiv:2006.03089 [cs. 2020.
Xingjun Ma, Bo Li, Yisen Wang, Sarah M Erfani, Sudanthi Wijewickrema, Grant Schoenebeck, Dawn Song, Michael E Houle, and James Bailey. Characterizing adversarial subspaces using local intrinsic dimensionality. In International Conference on Learning Representations (ICLR), 2018.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference Learning Representations (ICLR), 2018.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In Neural Information Processing Systems (NeurIPS), Workshops, 2011.
Nicolas Papernot, Patrick McDanicl, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as a defense adversarial perturbations against deep neural networks. In IEEE symposium on security and privacy (SP), 2016.
Geon Yeong Park and Sang Wan Lee. Reliably fast adversarial training via latent adversarial pertur- bation. In International Conference on Learning Representations (ICLR), Workshops, 2021.
Aditi Raghunathan, Jacob Steinhardt, and Percy Liang. Certified defenses against adversarial exam- ples. In International Conference on Learning Representations (ICLR), 2018.
Leslie Rice, Eric Wong, and Zico Kolter. Overfitting in adversarially robust deep learning. In International Conference Machine Learning (ICML), 2020.
Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! Neu- ral Information Processing Systems (NeurIPS), 2019.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 2016.
11

---

Under review as a conference paper at ICLR 2022
Yang Song, Taesup Kim, Sebastian Nowozin, Stefano Ermon, and Nate Kushman. Pixeldefend: Leveraging generative models to understand and defend against adversarial examples. In Inter- national Conference Learning Representations (ICLR), 2018.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations (ICLR), 2014.
Jinyu Tian, Jiantao Zhou, Yuanman Li, and Jia Duan. Detecting adversarial examples from sensi- tivity inconsistency of spatial-transform domain. In AAAI Conference on Artificial Intelligence (AAAI), 2021.
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick Mc- Daniel. Ensemble adversarial training: Attacks and defenses. In International Conference Learning Representations (ICLR), 2018.
BS Vivek and R Venkatesh Babu. Single-step adversarial training with dropout scheduling, In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Lily Weng, Huan Zhang, Hongge Chen, Zhao Song, Cho-Jui Hsieh, Luca Daniel, Duane Boning, and Inderjit Dhillon. Towards fast computation of certified robustness for relu networks. In International Conference on Machine Learning (ICML), 2018.
Eric Wong and Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International Conference Machine Learning (ICMI.), 2018.
Eric Wong, Leslie Rice, and J. Zico Kolter. Fast is better than free: Revisiting adversarial training.
In International Conference on Learning Representations (ICLR), 2020.
Puyudi Yang, Jianbo Chen, Cho-Jui Hsieh, Jane-Ling Wang, and Michael Jordan. MI-loo: Detect- ing adversarial examples with feature attribution. AAAI Conference on Artificial Intelligence (AAAI), 2020.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC British Machine Vision Conference (BMVC), 2016.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. International Conference on Machine Learning (ICML),
12

---

Under review as a conference paper at ICLR 2022
A ADDITIONAL PLOTS FOR PREACTRESNETIS EXPERIMENTS
In the main paper we compare N-FGSM with other single-step methods and multi-step methods separately and remove clean accuracies for better visualization. In this section present the curves for all methods with both the clean and robust accuracy. The tendency in the three datasets for N-FGSM PGD-50-10 accuracy to be slightly ahove that of GradAlign, while the opposite happens to the clean accuracy. We also observe that clean accuracy becomes significantly more noisy when catastrophic overfitting happens. Exact numbers for all the curves are in Appendix L
Standard and Adversaria Accuracy
CIFAR10 Dataset
Standard and Adversarial Accuracy
CIFAR100 Dataset
80
N-FGSM (ours)
60
GradAlign
60
PGD-10
PGD-2
40
40
MultiGrad
ZeroGrad
Free-AT AT
20
20
Kim
RS-FGSM
FGSM
8
10
12
14
16
2
4
10
12
16
for trainining and evaluation
E for trainining and evaluation
Standard and Adversarial Accuracy
SVHN Dataset
100
80
40
20
10
12
for trainining and evaluation
Figure 6: Comparison of all methods CIFAR-10, CIFAR-100 and SVHN with PreactResNet18 over different perturbation radius (e is divided by 255). We plot both the robust (solid line) and the clean (dashed line) accuracy for each method. Our method, N-FGSM, is able match surpass the state-of-the-aan single-step method GradAlign while reducing the cost by a 3x factor. Adversarial accuracy is based PGD-50-10 and experiments averaged over 3 seeds. Legend is shared among all plots.
B EXPERIMENTS WITH WIDERESNET28- 10 ARCHITECTURE
In this section we present the plots of our experiments with WideResNer12-10. We report the re- sults in two figures. In Figure we compare all single-step methods and do not plot the clean accuracy for better visualization. In Figure we plot all methods, including multi-step methods, and report the clean accuracy as well with dashed lines. Since we observed that our baseline, RandAl- pha, outperformed Kim et al. (202T) in all settings for PreActResNett13 we only report RandAlpha for WideResNet. As mentioned in the main paper, we observe that catastrophic overfitting seems to be more difficult to prevent for WideResNet. In particular, for GradAlign we observed the regu- larizer hyperparameter settings proposed by Andriushchenko & Flammarion (2020) for CIFAR- 10 (searched for PreActResNet18 worked well. However, those parameters led to Catastrophic Over- fitting for < < 12 in CIFAR-100 100. Since e = 14, 16 did not show Catastrophic Overfitting, we increased the GradAlign regularizer hyperparameter À for CIFAR-100 that each 6 c < 12 would have the default value corresponding to e + 2, for instance, for would be the default A in Andriushchenko & Flammarion (2020) for e 8.
13

---

Under review as a conference paper at ICLR 2022
CIFAR10 Dataset
CIFAR100 Dataset
80
50
Adversaria Accuracy
Adversarial Accuracy
60
40
30
40
20
N-FGSM (ours)
20
GradAlign
RaadAlpha
10
MultiGrad
ASSFGGM
ZeroGrad
FGSM
0
8
10
14
16
10
12
14
for training and evaluation
è for training and evaluation
SVHN Dataset
80
Adversarial aa Accuracy
60
40
20
4 training evaluation
10
12
Figure 7: Comparison of single-step methods on CIFAR-10, CIFAR-IO0 100 and SVHN with WideResNet28- 10 over different perturbation radius (e is divided by 255). Our method, N-FGSM, able to match or surpass the state-of-the-art single-step method GradAlign while reducing the cost by a 3x factor. Moreover, we could not find any competitive hyperparameter setting for GradAlign for e 2 6 in SVHN dataset. Adversarial accuracy is based on PGD-50-10 10 and experiments are averaged over 3 seeds. Legend is shared among all plots.
For SVHN we observed that the default values for À led to models close to constant classifier for e 2 6. We tried to increase the lambda for those e values to 1.25A but observed the same result. Since the model did not show typical catastrophic overfitting but rather it seemed as it was underfitting, we tried to reduce the step-size to = 0.75e and also both decreasing a and increasing X. When reducing the step size obtain accuracies above those of constant classifier for some radii, however, some or all seeds converge to a constant classifier for each setting, hence the large standard deviations. For N-FGSM, the default configuration of N-FGSM (a c, k 2e) works well in all settings except for 6 16 on CIFAR-10 and 10. 12 on SVHN. For CIFAR-10. we increase the noise magnitude to 4c. For SVHN we find that decreasing a as we tried for GradAlign works better than increasing the noise. We use 8 for both radii. Exact numbers for all the curves are in Appendix
C FURTHER ABLATION OF NOISE AND STEP SIZE IN N-FGSM
In Section we observed that both removing clipping and increasing noise were necessary to avoid catastrophic overfitting. However, when doing that, we increase the magnitude of the pertur- bations. In this section, we study more closely the interplay between the step-size a and the noise level & to ensure that it is indeed the increase in noise level, rather than merely increasing the mag- nitude of perturbations, what helps stabilize adversarial training and avoid catastrophic overfitting.
We fix étsst 8/255 for evaluation, and report the clean and robust accuracy of N-FGSM under different combinations of noise level and step size a. Note that when the noise level k = N-FGSM recovers plain FGSM, thus, as increase it is equivalent to using FGSM with an increased érain- On the other hand, when a = O. this is equivalent to training with only random noise augmentation. Based the results reported in Table we make the following observations:
14

---

Under review as a conference paper at ICLR 2022
CIFAR10 Dataset
Standard Adversarial Accuracy
CIFAR100 Dataset
Standard and Adversaria Acc
80
N-FGSM (aurs)
60
GradAlign
60
PGD-10
PGD
40
MultfGrad
ZeroGrad
20
20
RandAlpha ha RS-FGSM
FGSM
8
10
14
16
10
12
14
for trainining and evaluation
8 for trainining and evaluation
Standard and Adversarial Accuracy
SVHN Dataset
100
40
20
6 for trainining and evaluation
Figure 8: Comparison of all methods on CIFAR-10, CIFAR- 100 and SVHN with WideRes 10 over different perturbation radius (e is divided by 255). We plot both the robust (solid line) and the clean (dashed line) accuracy for each method. Legend is shared among all plots.
1) Increasing the perturbation size is not enough to avoid catastrophic overfitting, For instance, as observed in the first column of Table [ training with an increasing a without noise, i.e. (k- = 0 which is equivalent FGSM), leads to catastrophic overfitting despite the clear increase pertur- bation size due to the increase of a.
2) Catastrophic overfitting leads to a vulnerable model against smaller perturbations too. For instance, looking at the experiments with a 10 or a 12 in the first column, we observe that the resultant models are vulnerable to adversarial attacks for Ejest = 8/253 even though the perturbation radius used in training is larger than 8/255. This indicates that once model catastrophically overfits to perturbations of a given radius, can be vulnerable to smaller perturbations too.
3) Increasing the level of noise is necessary to stabilize training for larger perturbations. As we increase the step size a of the attack, we observe that we need to increase the ratio between noise k and step-size in order avoid catastrophic overfitting. This again further suggests that it is indeed increasing the perturbation size by increasing the noise level (and not simply increasing the perturbation budget) what mitigates catastrophic overfitting.
4) Training with noise perturbations has a much milder effect on the clean accuracy than adversarial training. Despite N-FGSM perturbations have larger radius, it does result in a significant drop of the clean accuracy, see Figure S Although it might seem counter-intuitive, we note that while N-FGSM has a larger perturbation size, this increase is not only merely in the adversary direction but also in the random noise direction. That is to say, while the perturbation size larger for N-FGSM, this not necessarily equivalent augmenting with adversaries with an a similarly larger perturbation size of adversaries due to the bias in the noise. In Table [ we observe how augmenting training samples with noise alone (when a = 0) has a much milder effect on the clean accuracy. In general, moving right on the table (increasing noise) is more forgiving on the clean accuracy than moving downwards (increasing FGSM step size). This is not so surprising considering that moving in random directions along the input space has much lower impact the loss than moving along the FGSM direction (see Figure [2] and that training with noise alone does not provide any significant robustness against larger attacks (for more detailed ablation see Figure 11..
15

---

Under review as a conference paper at ICLR 2022
k (no noise)
1/255 (0.5m))
#/266 (1emmi)
k 16/255 (2tmnin)
93.8 + 0.14
93.53 +: 0.12
93.06 + 0.02
91.76 + 0.07
0
0.0 = 0.0
0.0 + 0.0
0.0 = 0.0
0.01 = 0.0
88.77 = 0.04
88.61 = 0.07
88.42 + 0.03
87.79 + 0.05
4
35.88 + 0.55
36.12 = 0.09
36.7 + 0.23
37.27 + 0.23
85.58 + 0.11
85.52 + 0.23
85.03 + 0.09
84.49 + 0.1
43.85 = 0.12
44.14 + 0.24
44.44 + 0.13
44.44 + 0.15
86.41 0.7
81.54 +: 0.19
81.57 + 0.07
80.58 +
a
0.0 + 0.0
47.93 +: 0.11
48.16 1 0.21
48.12 +: 0.07
82.08 + 1.62
82.81 +: 1.11
77.32 1 0.14
76.49 + 0.14
10
0.0 0.0
0.0 1 0.0
49.68 0.25
49.77 1: 0.37
80.6 1: 2.59
81.75 + 1.1
82.0 1: 1.65
72.52 1. 0.16
12
0.0 = 0.0
0.0 + 0.0
0.0 + 0.0
50.17 + 0.22
Table 1: Ablation of the clean (top) and PGD-50-10 (bottom) accuracy when changing N-FGSM hyperparameters - noise level & and FGSM step size a. Results are averaged over seeds. All models evaluated with PGD-50-10 attack and East Etrain 8/255.
D ABLATION OF BASELINES INCREASING THE TRAINING EPSILON
We have seen in Theorem and Appendix that the magnitude of N-FGSM perturbations will (on expectation) be larger than that of other baselines. Moreover, since N-FGSM does not use clipping to ensure perturbations are within the fo ball, they could have -orm of up to k + as which for the default hyperparameter values (a = e, k = 2e) add up to 3e. In this section, we study the performance of all single-step step baselines
increase the training epsilon.
In Table 2 we present the result an experiment where we fix Gest = */255, then for each base- line increase the Euain from 8/253 to 16/255 (while always evaluating the final model at 8/25s). We use the same training hyperparameters as reported in Section 5.1 Results lead to three main observations:
1) Increasing Grrin ends up hurting test robust accuracy. Even though in some methods we ob- serve increase in adversarial accuracy we start to increase Erriin, it ends up decreasing. More- over, this increase in adversarial accuracy (see GradAlign, MultiGrad or N-FGSM) is at the expense of decrase in clean accuracy.
2) Catastrophic overfitting leads to a vulnerable model to smaller perturbations as well. As observed in Appendix C those models which suffer catastrophic overfitting become vulnerable to attacks with smaller perturbation radius than used during training. Here, we observe the same effect for various single-step baselines, which indicates this is likely general trend.
3) Changing N-FGSM step-size a leads to similar result to changing Crain in other baselines. Interestingly, we observe that although the presence of noise will lead to perturbations much larger than those of other methods, it is the step-size in the direction of the FGSM attack that will have larger impact on robust and clean accuracy. Thus, for larger perturbations the role of noise seems to be mainly to somehow mitigate catastrophic overfitting rather than strongly contributing to model robustness.
In light of the previous observations, we conclude that using larger perturbation for N-FGSM does not lead to an unfair comparison to other baselines, on the contrary, the best values for these baselines (considering the trade-off between clean and robust accuracy)
when using the same radius of perturbations for training and test (Ges = Cmin). On the other hand, this result highlights the particularity of increasing the perturbation size with noise rather than following the adversarial
16

---

Under review as a conference paper at ICLR 2022
Cirain
8/250 (16pa)
Eamin 13/250 (1.5tiea)
Ctrain
16/205 (2
86.41 + 0.7
80.6 + 2.59
77.14 + 2.46
FGSM
0.0 + 0.0
0.0 = 0.0
0.0 = 0.0
84.05 + 0.13
65.22 = 23.23
76.66 = 0.38
RS-FGSM
46.08 + 0.18
0.0 = 0.0
0.0 = 0.0
89.02 + 0.1
88.35 = 0.31
90.45 = 0.08
Kim et. al.
33.01 +
27.36=0.31
9.28 0.12
78.41 +
73.91 4.19
71.64 = 3.89
Free
46.03 + 0.36
32.4 1: 22.91
0.0 + 0.0
82.62 + 0.05
78.11 0.2
75.42 = 0.13
ZeroGrad
47.08 1 0.1
46.43 = 0.37
45.63 +: 0.39
82.33 1: 0.14
75.28 1. 0.2
71.42 1: 5.63
MultiGrad
47.29 + 0.07
50.0 = 0.79
16.01
81.9 0,22
73.29 = 0.23
61.3 +: 0.15
GradAlign
48.14 + 0.15
50.6 = 0.45
46.67 = 0.29
80.58 +:
71.46 = 0.14
63.18 + 0.49
N-FGSM
48.12 +
50.23 = 0.31
46.46 0.1
Table Ablation of the PGD-50-10 accuracy for single-step methods when increasing the Eurinn All models evaluated with PGD-50-11 10 attack and Gest #/255. Note that considering the trade-off between clean and robust accuracy, all methods perform best when training with the same epsilon to be applied at test time.
direction. Gaining deeper understanding of the role of noise in avoiding catastrophic overfitting is a promising direction for future work.
E LONGER TRAINING SCHEDULE
In our experiments, we have followed the "fast" training schedule introduced by Wong et (2020). However, |Rice et al. (2020) suggest that longer training schedule coupled with early stopping may lead to boost in performance. We also use the long training schedule for N-FGSM and observe that it does not lead to catastrophic overfitting. In Table we compare the performance of N-FGSM and GradAlign for the long training schedule. We observe that GradAlign does not seem to benefit from the long training schedule. On the other hand, although N-FGSM seems to obtain slight increase in performance, the "fast" schedule provides comparable performance. for GradAlign, the default regularizer hyperparameter for 8/255 and CIFAR- 10 (2 0.2) does not prevent catastrophic overfitting. We do a hyperparameter search and keep the value with the largest final robust accuracy (A 0.632).
worth mentioning that
17

---

Under review as a conference paper at ICLR 2022
N-FGSM
Grad Align
Clean Ace
Robust Ace
Clean Acc
Robust Acc
Long schedule: Final model
83.18 + 0.11
36.56 + 0.26
84.13 + 0.24
36.17 + 0.19
Long schedule: Best model
80.8 1 0.36
48.48 1 0.27
81.57 + 0.44
47.86 + 0.1
fast schedule: Final model
80.58 + 0.22
48. 12 + 0.07 81.9 = 0.22
48.14 + 0.15
Table Comparison of "long" (Rice al. 2020) and "fast" (Wong et al. 2020) training schedules for N-FGSM and GradAlign. GradAlign does not seem to benefit from the long training schedule. Although N-FGSM seems to obtain a slight increase performance, the "fast" schedule provides comparable performance.
18

---

Under review as a conference paper at ICLR 2022
F MAGNITUDE OF N-FGSM PERTURBATIONS
Lemma (Expected perturbation). Consider the N-FGSM perturbation as defined Equation 44)
5y-FGSM 1 + a siig (Vzl(f(z + m),y)), where 1) ~ S2
Let the distribution Q be the uniform distribution u (I-ke, and > O. Then,
En Ol = d (
122 +
and
kk22
Bn [llÓN-PasM|ll2 S Vd
a²
Proof. By Jensen's inequality, we have
E,, FoSNNe S E., [1]5w [1
Then let us consider the term E, [ [3] and use the shorthand V(n). = (Vz((f(x + v)),
E,
FOSSM
[2
= Ey 11 a sign (Vzlif(z + n).
d
En
(n + a sign v(n)i))
En
(Th + sign( ((()))))
(Th + a sign (V():)) siign( (V)))) =|]I,, p., [sig)(((())))))) = 1]
En
d
En
(n: + a- sign (g)()) sign (V())) = 1 P. (sign (V(7))) = -1]
1
ke
(1): + a)² dmk P. [ign ((r).)) = 1]
2ke
8=
ke
1
kr
a)² dn: (m==
P. (sign((((n)) -1]
2ke
-kr
pak
dd Pn [sign[ ((n).) = 1]
2kE
Ja-k
a
1
d
atke
2 22 Py sign(siniiinn ((n).) -1]
2ke
-a-ke
ttee
2 P. [sign( ((i).) 1]
2ke
ke
2d P, [sign( ((n)+) -1]
2k
kr
1 tke
22 dz Σ (P, (sign( ((n))) = 1] + P., (vig((((n)) -1])
2ke
Ja-ke
i=1
d
dk
[a + ke)3 - (a ke)
+ da²
6ke
3
19

---

Under review as a conference paper at ICLR 2022
Therefore,
k2t² 3 + a2
By [[[xx.FoM]lla]]
S
Vd
We state again Theorem 1 and present the proof.
Theorem 1. Let ÖN-FGSM be our proposed single-step method defined by Equation 8FGSM be the FGSM method (Goodfellow et al. (015) and ÖRS-FGSM be the RS-FGSM method (Wong et al. 2020). Then, with default hyperparameter values and any > we have that
En [16xx
En [[]6PGsM|
> En [lS8RS.FOsM|
Proof. From Lemma
we have that
By [[ d ( + a²
h222
On the other hand, Andriushchenko Flammarion (2020) showed that
E, [IloRs FOSM [2] = d( (- (T6c
a
+
a
2
6c
2 3
Finally, we note that
E, OP0S [2] |16PGsM 112 de²
The default hyperparameters for N-FGSM are k: = 2, a = ana RS-FGSM uses 5e/4. With these hyperparameters and any e > we have
7 de E. [[ [1 de E. [ FGSM|l
101
Ey rosM
de
3
128
In Lemma Twe compute the expected value of the squared &2 norm of N-FGSM perturbations and by Jensen's inequality we obtain an upper bound for the expected by norm of N-FGSM perturbations. However, obtaining the exact expected magnitude results, we approximate the & norm of FGSM, RS-FGSM and N-FGSM via Monte Carlo sampling. We observe that the empirical estimations are very close to the analytical upper bounds and that indeed, N-FGSM has magnitude significantly above that of FGSM or RS-FGSM.
more complex. To compliment our analytic
Results are presented Figure g
G N-FGSM WITH GAUSSIAN NOISE
In the main paper we have only explored noise sources coming from a Uniform distribution. Since we are measuring robustness against 4xo- attacks, the Uniform distribution is a natural choice be- cause the random perturbations will be bounded the loc ball defined by the span of the distribution. However, for the sake of completeness, we also explore the performance of augmenting the samples from Gaussian distribution where choose its standard deviation match that of the uniform distribution. In Table 4 we present a comparison of the clean (top) and PGD-50- 10 (bottom) accu- racy for different values a and noise magnitude with Uniform distribution - ks), therefore hyperparameter sets the noise magnitude.
$/255. Recall that by default we use
Increasing the FGSM step size without increasing the amount of noise leads to catastrophic over- fitting. Note results for t/ 0.5e. More importantly, results are very similar when the two noise distributions share the same standard deviation. Thus, using Gaussian instead of Uniform noise does not seem alter the results. Although this might be expected, remark that the Gaussian is an unbounded noise distribution and the common practice adversarial training is to always restrict the norm of the perturbations.
20

---

Under review as a conference paper at ICLR 2022
N-FGSM
20
Estimated f2 norm
FGSM / GradAlign
RS-FGSM
15
Upper bound N-FGSM
Upper bound RS-FGSM
10
5
2
4
8 10 12
14
16
Perturbation radius E
Figure 9: Monte Carlo estimations of the expected 12 norm of perturbations from different meth- ods and corresponding analytical upper bounds. As mentioned in Andriushchenko Flammarion (2020), observe that RS-FGSM perturbations have lower l2 norm than FGSM. However, N- FGSM perturbations have a significantly higher 12-norm than both RS-FGSM and FGSM. This seems to indicate that the role of random step not simply lower the norm previously suggested Andriushchenko & Flammarion 2020).
Uniform Noise
Gaussian Noise
9/250 (0.75c)
#/255 (1o)
10/250 (1.25c)
9/25s (0.75c)
8/250 (1r)
10/255 (1.25c)
0.19
82.1
81.71
k = 0.5c
46.14 1: 0.24
47.90 1:0.1
0.0 - 0.0
44.23 + 0.17
47.98 0.14
0.0 :: 0.0
71.32
81.35
77.22
A: = le
44.44 + 0.13
48.16 + 0.1
49.08 0.25
44.41 d: 0.04
48.21 0.11
49.83 d: 0.1
84.49
80.58
76.49
76.33
1: 40.44 + 0.15
48.12 0007
49.77
0.37
44.39 d: 0.22
48.34 =
49.77 0.25
Table 4: Comparison of the clean (top) and PGD-50-10 (bottom) accuracy across different values of step-size a and noise magnitude for the Uniform and Gaussian distributions with e 8/255. For every value of k, match the standard deviation, both distribution perform similarly.
Gaussian with matching standard deviation. We observe that when
H FURTHER RESULTS WITH RANDALPHA
In Section 5.2 we analyze the method presented by Kim et al. (2021 and suggest a baseline where, instead of evaluating intermediate points determine the "optimal" step size, we simply choose it randomly. That is, we multiply the RS-FGSM perturbation by random scalar sampled from uniform distribution [0, 1]. Interestingly we find that outperforms Kim al. (2021) without the additional cost of intermediate evaluations and without the need to perform hyperparameter selection to find the optimal number of intermediate evaluations.
Adversarial Accuracy
80
N-FGSM
Adversanial Accuraci
50
N-FGSM
Adversarial Accuracy
88
N-FGSM RandAlpha
80
RandAlpha
40
RandAlpha
Kim et. al.
Kim et.
60
Kim et. al.
30
40
20
20
10
20
12
12
for trainining and evaluation
for trainining and evaluation
for trainining and evaluation
Figure 10: Comparison of Kim et al. (2021) with RandomAlpha, our baseline where we multiply the RS-FGSM perturbation
scalar uniformly sampled in [0, 1]. We present results CIFAR-10. CIFAR- 100 and SVHN with PreActResNet18,
21

---

Under review as a conference paper at ICLR 2022
40
E 2 +
Adversaria Accuracy
6
30
8
20
10
0
Magnitude of uniform noise
Figure 11: Training with uniform noise augmented samples improves adversarial accuracy for small perturbations but not effective to protect against larger (o0 radius E. This motivates us to further augment the noisy samples with FGSM. All experiments are averaged over runs.
1 TRAINING WITH NOISE AUGMENTED SAMPLES
Gilmer et al. (2019) and Fawzi et al. (2018) report a close link between robustness to adversarial attacks and robustness to random noise. Actually, Gilmer et al. (2119) report that training with noise- augmented samples can improve adversarial accuracy and vice-versa. We note that N-FGSM can actually be seen as a combination of ausseetugmenttiion and adversarial attacks. Here we perform an ablation where train models with samples augmented with uniform noise U[-k, k] and then test the PGD-50-10 accuracy. We observe, that indeed random noise can increase the robustness to wort-case perturbations for small loo balls. However, longer very effective. However, with N-FGSM, we apply a weak attack to these noise-augmented samples and this seems be enough to make them effective for adversarial training.
increase noise augmentation is
VISUALIZATION OF THE LOSS SURFACE
In this section we present visualization of the loss surface. We adapted the code from Kim et al. (2021) to analyse the shape of the loss surface at the end of training for different methods. Kim ctt al. (202I) reported that after adversarial training catastrophic overfitting, the loss surface would become non-linear. In particular, they found that the FGSM perturbation seems to be misguided by local maxima very close to the clean image that result in ineffective attacks. We note this was already reported by Tramer al. (2018) which proposed to perform random step to escape those maxima. We argue that adding noise to the random step, when properly implemented, actually prevents those maxima to appear the first place.
K COMPARISON OF ADVERSARIAL TRAINING COST
In this section we describe how we compute relative training cost for single-step methods shown in Figure [
(right). We approximate the cost based on the number of forward/hackward passes each method uses, disregarding the cost other additional operations such as adding a random step for RS-FGSM N-FGSM. We understand these operations have negligible cost compared to full forward or backward pass.
FGSM: FGSM is the cheapest of all methods since it only uses forward/backward to compute the attack and an additional forward/backward to compute the weight update. Hence, Cost FGSM = 2 F/B.
RS-FGSM: As previously mentioned, we do not take into account the cost random steps or clipping, hence consider RS-FGSM to have the same cost standard FGSM. Cost RS-FGSM 2 F/B.
N-FGSM: Idem before, of N-FGSM F/B.
ZeroGrad: For ZeroGrad they need to do an additional sorting operation to find the smallest gradi- components. This could be potentially expensive, however, since the size of the input image is
22

---

Under review as a conference paper at ICLR 2022
N-FGSM model
GradAlign model
1.0
1.0
0.9
0.8
0.8
0.7
correct
wronq
0.8
0.8
0.4
0.4
0,2
FGSM
0.2
FGSM
0.0
0.0
RS-FGSM AT model
FGSM model
correct
conrect
wrong
wrong
0.8
0.0
1.0
1.0
Random
0.8
0.8
0.6
02
FGSH
0.2
0.4
FGSM
0.2
0.2
0.0
0.0
0.0
0.0
Figure 12: Visualization of the loss surface for models trained using different methods. Given clean sample from the test set in coordinate (0.0) 0). we compute the FGSM perturbation and evaluate the loss the subspace generated by the FGSM perturbation direction and random direction. That is, we evaluate Telean + OPGGS 12 RRandom where é1, 12 E [0, 1]. Note that FGSM and RS-FGSM both have catastrophic overfitting and the final models present highly non-linear loss surface, on the other hand, both N-FGSM and GradAlign produce final models with very linear loss surface which is key obtain meaningful perturbations.
several orders of magnitude smaller than that of the network, we also ignore this cost. Cost ZeroGrad 2 F/B.
MultiGrad:
MultiGrad computes 3 random steps and evaluates the gradient in all of them. There- fore, needs do 3 F/B compute the attack and additional one update the weights. Cost MultiGrad = 4 F/B.
Kim al. 2021): : Kim
(2021) compute the RS-FGSM perturbation and evaluate the model on c points along this direction. Therefore, they will spend IF/B on the RS-FGSM attack, c- on the evaluations since the clean image has already been evaluated; and F/B for the weight update. In our plot, we used = 3 since it was the most chosen setting. Kim et al. (2021) assume the cost of a forward is similar to that of backward pass, following this assumption, cost of Kim et al. (2021) 1 F/B 2 F + F/B = F/B
Shafahi et al. (2019) re-use the gradient from the previous backward pass to compute the FGSM perturbation of the current iteration. Hence, the cost of their training is only F/B per iteration. However, WWng et al. (2020) observed they needed a longer training schedule to produce comparable results. Therefore, the total training cost per iteration (1 F/B) is scaled by 96 in the of Free-AT, while it is only scaled by 30 for other methods. Relative cost Free = (96 F/B) / (30- 2 F/B).
Free-AT:
GradAlign: Finally, GradAlign uses FGSM with regularizer. However, this regularizer needs to compute second-order derivatives via double backpropagation, which does not have the same cost
23

---

Under review as a conference paper at ICLR 2022
regular backpropagation. Andriushchenko & Flammarion (2020) report that the cost of using GradAlign regularizer increased the cost of FGSM by 3.
L DETAILED RESULTS FOR SECTION 5.1 AND SECTION 5.6
In this section we present the tables with the exact numbers used in plots comparing adversarial training methods. For each method and Lo radius, the top number is the clean accuracy while the bottom number the PGD-50-10 accuracy. We separate single-step from multi-step methods with double line.
PreActResNed18 CIFAR-10 Dataset
< 2/258
c- 4/258
<- 5/255
- 5/255
<- 10/256
12/256
14/256
18/258
91 + 0.17
88.44 +: 0.00
84.72 +: 0.04
80.58 0.22
75.98 =: 0.1
71.46 = 0.14
67.11 1:11::10 0.37
63.18 0.49
N-FGSM
79.43 0.11
67.09
56.62 0.26
48.12
0.16
0.16
+ 0,93
91.73 + 004
88.76 + 0.0
85.67 +: 0.02
81.9 = 0.22
77.54 = 0.06
73.20 =: 0.23
68.01 = 0.32
61.3 01.3.tt: 0.15
Grad Align
79.16 003
67.13
56.27 0.31
48.14 0.15
40.75 0.28
34.51 d: 0.63
30.36
26.64 d: 0,27
91.6 1: 0.1
88.77 0.04
85.58 1: 0.11
86.41 1:
$2.08 =: 1.62
80.6 2.59
96.04 2.37
77.14 ::: 2.46
FGSM
79.35 + 0.06
67.11 0.00
56.33 =: 0.41
000 = 0.0
0.0 0: 0.0
0.0 0.0
0.0 4: 0.0
0.0 0.0
= 0.51
0.38
RS-RISM
78.64 1: 0.08
66.12 0.22
54.87 1: 0.22
46.08 0.18
0.0 0.0
0.0 :1: 0.0
0.0 4: 0.0
0.0 0.0
Kim et. al.
74.74 + 0.35
60.51 + 0.4
48.95 +: 0.45
33.01 0.09
24.49 = 0.84
13:11 - 0.63
5.86 + 0.57
1.88 0.05
87.99 0.16
84.98 0.13
0.11
73.91 4.19
14,94
71.64 3.89
AT Free
74.27 + 0:33
62.47 + 0.25
53.18 +: 0.15
46.03 0.36
39.87 = 0:07
22.99 16.26
+: 0.0
0.0 0.D
91.71 0.0
+: 0.11
85.71 0.1
82.62
78.11 0.2
66
st 0.13
ZeooGaad
79.36 0.05
67.32
56.14 1: 0.21
47.08 +:
37.58 0.2
27.41 :: 0.27
21.29 0.97
13.06 :: 0.22
91.57 1 0.16
88.74 0.12
85.75 :: 0.05
82.33 0.14
78.73 =: 0.16
75.28 0.2
80.94 5.94
71.42 5.63
MoliiGra
47.29
0.17
9.55
16:35
91.4 0.07
88.46 0.13
85.14 1: 0.13
81.41 0:05
77.18 = 0.15
72.9 =: 0.26
70.39 =:: 2.71
64.81 11.58
PGD-2
79.55 1: 0.15
67.62 1: 0.00
57.39 1: 0.13
19.58 0.08
43.3
38.13 1: 0.15
22.89 1: 15.26
9.6 13.37
91.25 1: 0.04
88.34 1:
81.79 1: 0.11
80.71
76.13 :: 0.35
71.24 0.3
66.7 0.39
62.11 :: 0.62
PGD-10
79.47 +: 0.13
68.29 + 0.24
58.85 +: 0.18
51.33 0.31
45.02 0.49
39.93 0.5
36,02 0.67
32.32 :: 0.64
24

---

Under review as a conference paper at ICLR 2022
PreActResNet18 C-FAR-100 Dataset
-- 8/255
1/255
6/255
5/205
10/255
12/255
16/255
16/255
69.12 1: 0.27
64.0 1: 0.06
59.53
0.02
54.9 0.2
50.6 d: 0.16
46.06 0.14
41.67 4: 0.25
57.91 d: 0.11
51,02 0.34
0.12
32,06
26.46
22.23
18.6
68.96 0.15
64.71 + 0.16
60.42 +: 0.23
56.33 0.51
54.06 0.44
48.87 =: 0.32
43.84 0.14
38.93 0.21
Grad Align
51.31 0.12
39.37 0.25
31.91 0.28
25.8 1: 0.14
18.7 1: 1,92
17.86
15.51 1: 0.16
13:62 :: 0.19
69.01 1: 0.13
64.47 1: 0.15
63.83 1: 2.18
53:42 0.65
45,06 2.29
46.14 2.58
41.66 1: 0.88
44.68 :: 1,74
FGSM
513 + 0.19
39.7 + 0.36
10.93 + 14.64
0.0 + 0.0
00.0 = 0.0
0.0 = 0.0
0.0 0.0
0.0 0.0
+
+
+ 12.12
36.07
5.56
RS-RGIM
50.13 1: 0.32
38.36 1: 0.19
30.82 1: 0.08
0.01 +: 0.00
0.0 0.0
0.0 4: 0.0
0.0 0.0
0.0 0.0
70.16
+:
74.09
+:
d:
Kim et. al.
44.19 1: 0.25
30.63 1: 0.28
22.0 4: 0.00
12,75 0.21
6.98 1: 0.23
0.0 0.0
0.0 :1: 0.0
0.0 0.0
63.01
0.37
0.37
4.86
20 11.25
AT Free
45.7 0.33
35.95 0.09
29.37 0.21
24.32 + 0.4
20.64 0.22
5.71 +: 8.05
0.0 0.0
0.0 0.0
69.35 0.36
64.59 + 0.32
60.09
36,94
54.55
52.97 0.34
+ 0.26
20.73 0.3
ZeroGrnd
39.38
31.73
25.87
19.49
14.32 0.08
10.92 4: 0.59
0.16
69.01 + 0.16
64.44 0.10
60.65 =:: 0.36
56.84 =::
53.62 0.25
$3.05 1.85
48.78 4: 0.66
45.78 11.14
MoliiGrd
21.37
69.18 0.1
64.32 0.14
60.21 d: 0.13
55.8 0.16
31.68 d: 0.1
48.2 0.1
46.14 d: 1.24
37.97 10.52
PGDD22
51.36 1: 0.03
40.06 0.14
32,99 0.24
27.38 0.16
23.39 0.19
19.83 0.29
10:55 4: 7.51
4.79 6.75
1: 0.07
63.87 1: 0.00
59.37
54.79
50.53 0.15
46.05 0.21
41.76 1: 0.07
:: 0.14
PGD-10
51.51 + 0.27
40.59 + 0.36
33.65 0.82
28.55 0.27
24.17 0.12
21.2 :: 0.12
18.72 :: 0.06
16.59 + 0.16
PreActResNet18 SVHN Dataset
2/2/255
4/260
= 6/250
8/250
10/250
12/2n5
96.01 0.04
94.54 + 0.15
92.25 d: 0.35
89.36 +: 0.49
56.74 0.86
$1.48 d: 1.64
33.96
96.02 0.05
94.56 4: 0.21
90.53 :: 0.24
90.1 0.34
87.03 0.75
$4.01 1: 0.46
Grdd
86.43
0.24
43.85 +: 0.14
32.87 0.33
23.02 0.41
96.04 0.07
95.67 =: 0.07
03.73 =: 0.68
91.74 0.86
90.76 :2: 0.63
87.17 0.43
1USM
86.5 0,05
13.61 5.83
0.56 0.72
0.26 0.36
0.1
0.0 + 0.0
96.18 0.11
95.09 0.09
95.11 4: 0.44
94.46 +: 0.16
93.88 0.24
92.74 0.5
RS-RGSM
86.16 0.14
71.28 0.4
0.11 0.08
0.0 0.0
0.0 :: 0.0
0.0 0.0
96.35
95.25
-
0.29
96.11
96.61 0.01
Kim
83.26 0.24
66.32 d: 0.63
48.27 d: 0.52
31.8 +: 1.1
0.18 4: 0.21
0.0 +:: 0.0
AT Free
84.55 0.27
71.61 :: 0.75
59.31 1.0
0.01 =: 0.0
0.0 =: 0.0
0.0 - 0.0
96.06
=: 0.26
1.29
90.34
88.09 0.4
ZenoGra
86.43 1:
71.59 0.22
51.72 1: 0.53
35.93 -1: 2.73
21.34 0.31
14.14 0.32
96.01 0.08
94.71 0.17
95.77 +: 0.58
=: 0.97
94.7 0.12
94.48 0.19
MultKGrad
86.4 1: 0.08
71.98 0.26
28.1 ::: 18.85
11.49
16.19
:1: 0.0
0.0 0.0
96.03 0.14
94.66 d: 0.1
93.77 -:: 0.61
94.63 +:: 1.29
$4.09 +:: 14.99
94.16 d: 0.54
PCD-2
86.72
0.73
20.68
0.02
95.92 0.08
94.37 =: 0.15
02.44 0.25
89.67 1: 0.34
85.75 0.65
80.08 =: 0.93
PGD--10
86.94 0.14
74.76 0.19
63.9 0.48
53.95 :1: 0.55
44.91 0.45
37.68 :: 0.53
25

---

Under review as a conference paper at ICLR 2022
WideResNe128-10 CIFAR-10 Dataset
c- 4/265
c- 4/265
c- 9/265
< 8/256
< 10/256
12/266
14/266
< 10/266
92.51 + 0:11
89.63 + 0.00
85.8 0.25
81.59 0.32
76.92 0.04
72.13 0.15
67.82 0.41
56.73 0.42
N-PGSM
81.43 0.3
69.11 1: 0.24
58.29 d: 0.14
19.53 0.25
42.37 0.36
36.85 0.2
31.66 0.6
25.01 0.23
92.59 0.03
89.95 0.3
86,98 1: 0.06
83.19 = 0.26
79.35 0.26
73.79 4: 0.72
66.38 4: 0.53
57.75 0.75
Grad Align
81.33 0.4
69.81 + 0.47
59,00 0.13
50,00 0.05
41.48 = 0,51
35.06 4: 0.74
30.83 + 0.39
26.26 0.13
+ 0.18
= 82.67 1.78
+
POSM
81.38 1: 0.22
69.59 1: 0.25
38.69 26.54
0.0 0.0
0.0 1: 0.0
0.0 01 0.0
0.0 0.0
0.0 :: 0.0
RS FGSM
90.73
88.24 + 0.19
=
+
68.64
80.9 1: 0.13
68.23 1: 0.17
57.21 1: 0.17
0.0 0.0
0.0 0:0::00 0.0
0.0 0.0
0.0 0.0
0.0
0.22
83.23
RamdAlpha
77.67 +: 0.06
63.73 +: 0.31
50.4 0.14
39.37 = 0.42
30.13 =: 0.9
23.13 4: 0.33
16.0 0.22
8.47 d: 0.66
90.66 0:25
0.15
86.11 0.29
83.5
80.52 0.52
d: 1.35
+ 15.8
42.59 d: 27.96
AT Free
0.27
64.25 4: 0.33
53.76 4: 0.48
= 0.30
31.87 :: 5.53
0.0 0.0
0.0
0.0 0.0
92.62 + 0.11
90,17 0.05
86.98 0.28
84.25 0.28
81.72 0.29
79.34 0.82
78.14 0:46
75.34
ZenGrnd
+
+
14.6
92.64 + 0.1
90.18 0.13
87.11 = 0.36
83.87 0.46
50.89 0.14
62.88 4: 2.85
86.6 1.52
85.46 1:46:::0 3.73
92.69 1: 0.14
90.18 0.19
86.87 0.18
83.31 0.16
79:61 0:47
75.81 1: 0.24
71.41 1.38
67.2 1: 14.94
PGD-2
81.54 + 0.18
69.87 + 0.26
19.4 0.19
50.88 0.16
43.94 0.24
57.37 4: 0.57
21.06 13.39
0.0 0.0
92.24 + 0.31
1: 0.33
86.91 0.51
82.82 0.7
78.63 0.66
74.0
1:
64.17 0.72
PGD-10
81.18 + 0.57
70.34 +: 0.26
60.59 +: 0.21
$2.58 0.2
45.92 0.38
40.44 = 0.17
35.98 + 0.56
32.5 + 0.61
WideResNee28-100 CIFAR-100 Dataset
c= 2/265
<= 5/265
c= G/266
< 8/256
< 10/250
< 12/266
<= 14/265
<= 10/266
71.56 + 0.13
66.49 0.46
61.38 +: 0.68
56.23 0.59
51.54 +: 0.6
46.43 0.6
42.11 + 0.52
38.34 0.47
N-FGSM
52.23 1: 0.33
39.93 0.37
30,97 1: 0.21
26.77 0.65
2303 1: 0.54
19.3 0.59
16.67 0.4
14.27 0.33
71,68 1: 0.33
67.09
62.86 0.1
58.55 0.41
1 0.73
46.94 0.86
42.63 0.5
36.17 0.45
Grad Align
51.5 0.45
39.9 +: 0.42
33.0 0.22
26.9 +: 0.62
22.63 + 0.62
19.9 + 0.65
16.93 + 0.12
14,03 +: 0.24
56.87
+
PGSM
$2.83 + 0.37
39,83 0.31
0.0 0.0
0.03 :: 0.05
0.0 0.0
0.0 + 0.0
010 + 0.0
0.0 0.0
1:25
PGSM
31.63 +: 0.53
39.57 =:: 0.09
26.63 2.8
0.0 000:2:00 0.0
00 100::00 0.0
0.0 0.0
0.0 0.0
0.0 0.0
1.9:::0.15
71.17 11.17:200113
0.22
66.42
64.05 0.5
0.57
0.78
RandAlpha
19.13 4: 0.91
34.3 +: 0.54
25.5 0.33
20.27 0.98
16.3 0.14
12.4 +: 0.29
6.93 4: 0.19
3.63 0.12
67.62 4: 0.24
63.27 0.72
59.53 d: 0.31
55.77 0.28
47.00 4: 3.83
33.52 9.24
1.78
20.92 21.48
AT Free
+ 0.31
37.93
34.43
0.0
0.0
71.68 + 0.07
67.2 4: 0.14
63.69 4: 0.14
60.77 =:: 0.26
61.05 4: 0.38
58.39 0.16
56.19 0.11
56.38 =: 0..8
71.8 + 0.15
67.73 01.88
63.24 4: 0.33
60.05 0.79
56.39 4: 0.49
56.79 8.27
59.8 90.8.b10 3.77
$2.96 +: 5.58
0.37
:1
71.62 + 0.15
67.25 0.43
-: 0.36
59.02
1 0.45
50.01 0.35
41.03
40.13
PGD-2
51.73 4 0.48
40.27 4: 0.7
32.23 4: 0.19
27.13 0.37
23:43 4: 0.31
20.23 0.39
0.03 +: 0.05
0.0 0.0
66.9 +: 0.57
0.88
39.2
PGD-10
$2.5 0.59
40.73 0.56
32.8 0.29
27.97 0.59
24.7 0.36
21.8 :: 0.57
18.87 :: 0.6
16.8 d: 0.57
26

---

Under review as a conference paper at ICLR 2022
WideResNee28-100 SVHN Dataset
=/2/255
4/255
= 6/255
8/255
= 10/255
12/255
95.64 + 0.09
93.66 0.41
91.77 0.42
53.39 +: 0.58
88.07 + 0.59
87.52 + 0.49
84.1
05:41 :: 0.06
93.9 =: 0.48
68. 68.36 34.49
42.62 32.75
19.3 0.21
19.53 03.33.210 0.08
Grad
84.57 0.56
67.27
39.53 4: 14.89
24.7 9.34
17:63 0.62
18.13
95.83 0.1
95.0 4: 0.24
94.23 0.79
91.11 :: 1.36
$8.83 #: 1.71
86.74 :::
IGSM
85.03 0.37
31,53 6.57
1.7 :: 1.36
0.13 = 0.19
0.0 0.0
0.0 - 0.0
95.81 0.25
94.53 d:
95.23 0.26
94.68 :: 0.62
93.9 0.52
91.64 2.98
RSJKISM
83.8 = 0.43
66.67 0.65
0.53 4: 0.26
0.0 100::00 0.0
0.0 0.0
0.0 - 0.0
95.47
94.69
+
RandAlpha
82.5 = 0.45
63.33 0.53
47.7 11 0.99
35.73 1: 0.34
23.17 1: 1.97
11.1 :: 3.05
AT Free
83.13 0.17
68.67 0.53
54.93 2.58
0.03 0.05
0.0 0.0
+: 0.0
95.78 0.21
94.06 0.52
92.13
91.04 0.4
0.92
89.8 +: 1.36
ZeroGrad
84.47 0.83
66.1 :: 0.37
47.3 1: 0.62
29.33 :: 0.56
20.77 =: 0.63
9.33 :: 0.76
95.63 0.16
94.27 0.38
93.64 1.21
94.83 =: 1.55
95.26 4: 0.34
95.22 0.15
MuloGral
67.27
05. 0.35
94.66 =: D.1
93.77 0.61
92.99 =: 1.11
88.1 =: 0.93
83.17 4.78
PGTD-2
86.25 0.7
73.29 0.25
60.53 0.72
40.77 ::: 4.39
34.33 2.76
26.8 1: 3.31
95.92 de 0.08
94.36 0.15
92 0.25
89.67 =: 0.34
83.98 =: 0.59
80.08 0.93
PGD
86.94 0.13
74.46 0.54
63,87 0.49
53.95 :: 0.55
44.59 :1 0.14
37.64 0.49
27


Reviews:
[{"review_id": "vwycuNwfylb", "reviewer": "Reviewer_ZY47", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "##########################################################################\n\nPros: \n \n1. The paper attempts to improve the efficiency for adversarial training. For me, the problem itself is real and important. \n \n2. This paper proposes a method called Noise-FGSM (N-FGSM), which attacks noise-augmented samples directly using a single-step. N-FGSM is simple but effective. \n \n3. Extensive experiments shows N-FGSM achieves the SOTA results while achieving a 3x speed-up.\n \n##########################################################################\n\nCons: \n \n1. Although the authors give theoretical analysis to understand the role of noise in\nsingle-step approaches, the theoretical analysis on that increasing the noise magnitude\nand not clipping prevents catastrophic overfitting is not provided. It will be better to explore theoretical justification behind this.\n \n2. Another concern lies in the accuracy on CIFAR 100 and SVHN datasets. As shown in Fig.3, the performance improvement is very slight.\n", "summary_of_the_paper": "Different from previous intuitions, this paper find that not clipping the perturbation around the clean sample and using a stronger noise is highly effective in avoiding CO for large perturbation radii. Based on these observations, the authors propose a method called Noise-FGSM (N-FGSM), which achieves the comparable results to GradAlign while achieving a 3x speed-up.", "summary_of_the_review": "Overall, I vote for accepting. This paper find that not clipping the perturbation around the clean sample and using a stronger noise is highly effective in avoiding CO for large perturbation radii, which is interesting. Based on this, it proposes a simple but effective method called N-FGSM, which achieves the SOTA results while achieving a 3x speed-up. One major concern is about the theoretical analysis on the observations. Hopefully the author can address my concern in the rebuttal period.\n\n---------------------------------------------------------------------------------------------------------------------------------------------------------------------------\nUPDATE\n\nI have carafully read your response. My main concern has not been well addressed. The main drawback of this paper is the lack of theoretical analysis. The reason why it works is not clear enough. I believe that the paper may have potential but as its current form has some weakness. I turn to the rating of marginally above the acceptance threshold.\n\nBy the way, the performance improvement refers to accuracy rather than efficiency.\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "fTfYHPkfAYc", "reviewer": "Reviewer_Gq3c", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. The authors conducted many experiments to verify the effectiveness of their proposed N-FGSM. They presented their results through tables and figures.\n2. The experimental results indicate that  the proposed N-FGSM can achieve better performance than RS-FGSM and GradAlign in terms of adversarial accuracy(Robustness).\n\nWeakness:\n1. The contribution of this paper is limited. This paper provides little insight as a research paper as the two improvements of N-FGSM are empirical and heuristic to some extent. \n2. The motivation of this paper is unclear. The authors raised the CO problem but turn to the two \"key components\". The two so-called \"key components\" of adversarial training, clipping and random steps, are unconvincing. It is more like achieving the improvements first and then finding the motivations to fit them in. \n3. The evaluation metric is unfair. By eq(4), the adversarial perturbation will be definitely larger than the baselines. Theorem 1 is obvious and hard to be the contribution of this paper. The magnitude of the $\\epsilon$ for the baselines should be triple. \n4. To my understanding, increasing the magnitude of the adversarial perturbation may lead to a worse clean accuracy. Figure 5 verifies the clean accuracy drop. However, the table in Figure 4 seems to illustrate that the clean accuracy of N-FGSM is maintained compared with the baselines. And the table is very hard to understand. \n5. What is the $\\epsilon$ axis in Figure 3 and 4? Does it represent the magnitude of $\\epsilon$ for training? If yes, why the robustness decreases as the $\\epsilon$ increases? If the $\\epsilon$ is the magnitude of the evaluation, then PGD-50-10 usually represents a PGD attack with 50 steps and 10/255 l-infi norm.  It isn't very clear here. ", "summary_of_the_paper": "This paper proposes an adversarial training method, noise-FGSM(N-FGSM), which improves the RS-FGSM by removing the clipping perturbation and increasing the magnitude of adding noise. The training speed of N-FGSM is the same as the RS-FGSM, while the experimental results indicate that N-FGSM outperforms RS-FGSM. ", "summary_of_the_review": "To summary, this paper is more like an experiments report instead of a research paper. The contribution is limited, and the evaluation metric seems to be unfair. I suggest a clear rejection for this paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "OiixkMdIxxY", "reviewer": "Reviewer_iWaX", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ Overall, this paper is well written. \n+ Empirical results look fairly good. \n\nWeaknesses:\n- The absence of clipping operation will violate the $\\ell_p$-ball constraints on perturbation and therefore lead to an unfair comparison with the baselines. The perturbation of adversarial examples is by convention, constrained in an \\ell_p norm ball, to indicate the strengths of adversarial attacks and the clipping operation is the key to maintaining the constraint. A larger $\\epsilon$ represents larger perturbation budgets and always leads to stronger attacks. Likewise, perturbation without clipping in adversarial training naturally equips models with better robustness. Therefore, although N-FGSM indeed brings better robustness, it may not be fair to directly compare N-FGSM with its baselines. The choice of very large $\\eta$ (e.g. 2$\\epsilon$) in Algorithm 1, further equipped the adversarial perturbation with a larger attack budget. The real perturbation budget of N-FGSM is likely to be larger than $\\epislon$ itself. It will be better for the authors to present the real $\\ell_p$ norm of perturbations generated by N-FGSM.\n\n- Based on the above, the role of noise in promoting robustness is also suspicious, because it is hard to tell whether the presence of noise or a larger budget contributes more to better robustness. \n\n\n", "summary_of_the_paper": "This paper methodically studied the catastrophic overfitting in fast adversarial training (Fast-AT), and revisited the role of noise and clipping operation in Fast-AT. Based on the empirical findings, this paper discovered that the absence of clipping as well as using stronger noise could help avoid catastrophic overfitting. The author further proposed Noise-FGSM, utilizing single-step FGSM and noise-augmented samples to generate adversarial examples for training. Empirical studies showed the superiority of N-FGSM both in terms of performance and speed.\n", "summary_of_the_review": "In all, although the method proposed in this paper beats the baselines, it may not be a fair comparison and there are some fundamental errors in the experimental settings.\n\nPost-rebuttal adjustment:\n\nThanks for the additional clarification. I decide to increase my score by a point due to the authors' effortful response.\n\nI did not increase the score to 6 since I am not fully convinced on why noise augmentation + with adequate strength and without clipping is a principled solution to improve the effectiveness of fast adversarial training. I did not penalize its simplicity (this is fine to me), however, its effectiveness should be further justified either empirically or theoretically, e.g., whether or not the proposed design is applicable to fast adversarial training using TRADES-type loss, or other similar and in-depth studies.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "K_6gZuxO0KI", "reviewer": "Reviewer_6yud", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is overall well-written and easy to follow. The proposed methodology is quite simple and easy to implement. Detailed and comprehensive empirical evidence is provided to support the proposed method. However, I have several major comments regarding the presentation and intuition.\n\n\n* From my understanding, the proposed method can be rephrased as using a training perturbation radius significantly larger than the test perturbation radius (typically twice as mentioned by the authors), since the perturbation is randomly initialized in a larger radius, and no clipping is used. Could the authors confirm if my understanding is correct?\n\n\n* The paper found that a larger training perturbation radius can mitigate the catastrophic overfitting. This is somewhat counter-intuitive. I was expecting a figure clearly showing that as the training perturbation becomes larger, the catastrophic overfitting is mitigated (probably redesign Figure 2 right). I think more (empirical) analyses will be helpful to understand why this is true. There are some analyses in Section 4.1 on the loss landscape, but the intuition is still lacking.\n\n\n* I don't quite understand why a larger training perturbation radius will not lead to a clean accuracy drop. I think this observation is quite important to the paper's contribution but there is almost no justification. Providing at least some empirical analyses would be helpful.\n\n\nMinor comments:\n* The observation that larger perturbation does not necessarily hurt the performance is important in this paper. Section 4.2 follows [1] and show such observation is contradictory to their claim. It is better to merge this part of the analysis with Section 4.1 and provide a more integrated and intuitive understanding.\n\n\n* The notation epsilon is abused without clearly stating whether it is training perturbation or test perturbation. In Figure 2 I believe the epsilon here refers to only the test perturbation radius while in Figure 3 it could be both training and test?\n\n\n* Consider include some numerical results in the main paper. For example, move part of the performance tables in Appendix J to the experiment section, along with the training cost or speed up factor. This would justify the proposed method better if the main claim is efficiency.\n\n\n\n\n\n\n\n\n", "summary_of_the_paper": "This paper aims to address a failure mode in the traditional single-step adversarial training known as catastrophic overfitting. They show that compared to the common practice of generating the adversarial perturbation, adopting larger random initialization and avoiding clipping the perturbation can effectively mitigate catastrophic overfitting. The effects of these two techniques are analyzed empirically in detail, followed by a comprehensive comparison to other methods.", "summary_of_the_review": "I believe this paper made some interesting observations towards understanding and mitigating catastrophic overfitting in adversarial training. However, I am not fully convinced as the support analyses cannot provide sufficient evidence of why the proposed method could work. I am willing to increase my score if the authors can better clarify the proposed method and provide more intuition.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper aims to address the catastrophic overfitting issue in single-step adversarial training. Specifically, this paper finds that 1) using larger random noise initialization and 2) avoiding clipping adversarial perturbations are the two keys for stabilizing single-step adversarial training. 

Overall, the reviewers find this paper is well-written and the empirical results look promising. The reviewers originally misunderstood certain technical details of this paper, but got clarified in the discussion period.  However, the biggest concern shared by the reviewers is that the motivation of using larger random noise initialization and avoiding clipping adversarial perturbation is pretty unclear---they all fail to (either empirically or theoretically) understand how and why these two techniques are helpful to preventing catastrophic overfitting. Given the main contribution of this paper is a revisiting of existing techniques, it is a legitimate concern from the reviewer side for demanding the in-depth empirical analysis or the theoretical proof to help them better understand the proposed method; otherwise, the novelty contribution of this paper may get trivialized. 

I encourage the authors to delve deeper into the proposed method and make a stronger submission next time.
Decision: Reject