Make Some Noise: Reliable and Efficient Single-Step Adversarial Training
Authors: Pau de Jorge
Adel Bibi
Riccardo Volpi
Amartya Sanyal
Philip Torr
Grégory Rogez
Puneet K. Dokania
Venue: NeurIPS 2022 Accept (2022)
Keywords: single-step adversarial training
catastrophic overfitting
FGSM
efficient adversarial training
fast adversarial training

Abstract: Recently, Wong et al. (2020) showed that adversarial training with single-step FGSM leads to a characteristic failure mode named catastrophic overfitting (CO), in which a model becomes suddenly vulnerable to multi-step attacks. Experimentally they showed that simply adding a random perturbation prior to FGSM (RS-FGSM) could prevent CO. However,  Andriushchenko & Flammarion (2020) observed that RS-FGSM still leads to CO for larger perturbations, and proposed a computationally expensive regularizer (GradAlign) to avoid it. In this work, we methodically revisit the role of noise and clipping in single-step adversarial training. Contrary to previous intuitions, we find that using a stronger noise around the clean sample combined with \textit{not clipping} is highly effective in avoiding CO for large perturbation radii. We then propose Noise-FGSM (N-FGSM) that, while providing the benefits of single-step adversarial training, does not suffer from CO. Empirical analyses on a large suite of experiments show that N-FGSM is able to match or surpass the performance of previous state of-the-art GradAlign while achieving 3$\times$ speed-up.

Make Some Noise: Reliable and Efficient Single-Step Adversarial Training
Pau de Jorge" University of Oxford NAVER LABS Europe
Adel Bibi University of Oxford
Riccardo Volpi NAVER LABS Europe
Amartya Sanyal ETH Zürich ETH Al Center
Philip H. S. Torr University of Oxford
Grégory Rogez NAVER LABS Europe
Puneet K. Dokania University of Oxford Five AI Ltd.
Abstract
Recently, Wong et [35] showed that adversarial training with single-step FGSM leads to characteristic failure mode named Catastrophic Overfitting (CO), in which a model becomes suddenly vulnerable to multi-step attacks. Experimentally they showed that simply adding random perturbation prior to FGSM (RS-FGSM) could prevent CO. However, Andriushchenko and Flammarion [1] observed that RS-FGSM still leads to CO for larger perturbations, and proposed computationally expensive regularizer (GradAlign) to avoid it. In this work, we methodically revisit the role of noise and clipping in single-step step adversarial training. Contrary previous intuitions, we find that using a stronger noise around the clean sample combined with clipping is highly effective in avoiding CO for large perturbation radii. We then propose Noise-FGSS (N-FGSM) that, while providing the benefits single-step adversarial training, does not suffer from CO. Empirical analy yses on a large suite of experiments show that N-FGSM is able to match or surpass the performance of previous offte-of-the-art GradAlign, while achieving 3x speed-up. Code can be found in https: //gi thub. com/ honn/ddjjogg////e
Introduction
Deep neural networks have achieved remarkable performance variety of tasks 12
27
However, it is well known that they are vulnerable to small worst-case perturbations around the input data-commonly referred as adversarial examples [31]. The existence of such adversarial examples poses a security threat to deploying models in sensitive environments (2 4). This has motivated a large
body of work towards improving the adversarial robustness of neural networks [1] 22
32 25 5
The most popular family of methods for learning robust neural networks based on the concept min-max problem where instcad of minimizing some loss over a dataset of clean samples, we augment the inputs with worst-case perturbations that generated online during training. However, obtaining such perturbations is NP-hard 34] and hence, different adversarial attacks have been suggested that approximate them. In their seminal work, Goodfellow et al. [I1] proposed the Fast Gradient Sign Method (FGSM), that generates adversarial attacks by performing a gradient ascent step on the loss function. Yet, while FGSM-based adversarial training provides robustness against single-step FGSM adversaries. Tramèr et al. [32] showed that these models are still vulnerable to multi-step attacks, namely those allowed perform multiple gradient ascent steps. Given their better (robust)
of adversarial training [11] [20]. In nutshell, adversarial training can be posed
"Correspondence pauêrobots. ox. ac.uk
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

CIFAR10 Dataset
80
3.0
FGSM
N-FGSM
2.5
Adversaral Ncc.
60
2.0
1.5
40
1.0
N-FGSM (ours)
Pree-AT
0..5
GradAlign
k
MultiGred
RS-FESM
0.0
GreeAiion N-FeSM
MullGraad
ZereGred
Kin et a
RS-resM
ZeroGrad
RS-FGSM
6 for training and evaluation De 255)
16
16
Figure 1: Left: Visualization of FGSM [T), RS-FGSM [35 and N-FGSM (ours) attacks. While
RS-FGSM is limited to noise in the Loe ball, N-FGSM draws noise from arbitrary k - Loo ball. Moreover, N-FGSM does not clip the perturbation around the clean sample. Middle: Comparison of single-step methods on CIFAR-10 with PreactResNet18 8 over different perturbation radii (c is divided by 255). Our method, N-FGSM, can match or surpass state-of-the-ar results while reducing the cost by 3x factor. Adversarial accuracy is based on PGD-50-10 50-10 and experiments are averaged over 3 seeds. Right: Comparison of training costs relative to FGSM baseline based on the number of Forward-Backwart passes, see Appendix Qfor details.
performance, multi-step attacks such as Projected Gradient Descent (PGD) 20 have now become the de facto standard for adversarial training.
The main downside of multi-step adversarial training is that the cost of these attacks increases linearly with the number of steps, making their applicability often computationally prohibitive. For this reason, several works have focused reducing the cost of adversarial training by approximating the worst-case perturbations with single-step attacks [35] 26 33 In particular, Wong et al. [35] studied FGSM adversarial training and discovered that it suffers from characteristic failure mode, in which a model suddenly becomes vulnerable to multi-step attacks despite remaining robust to single-step attacks. This phenomenon is referred to as Catastrophic Overfitting (CO). As solution, they argued that adding a random perturbation prior to FGSM (RS-FGSM) seemed sufficient to prevent CO and produce robust models. Yet, Andriushchenko and Flammarion [ recently observed that RS-FGSM still leads to CO as one increases the perturbation radii of the attacks. They suggested a regularizer (GradAlign) that avoid CO in the settings they considered, but at the expense of computing double derivative significantly increasing the computational cost with respect to RS-FGSM.
In this paper, we revisit the idea of including noise in single-step attacks. Differently from previous methods that consider the noise as part of the attack, we propose an adversarial training procedure where the noise is used form of data augmentation. As detail in Section 4 this motivates introduce two main changes with respect to previous methods: 1) We center adversarial perturbations with respect noise-augment augmentdd samples and therefore, unlike previous RS-FGSM, we do not clip around the clean samples. 2) We use noise perturbations larger than the ball, since they are restricted by the strength of the attack anymore. Our experiments show that performing data augmentation with sufficiently strong noise and removing the clipping step improves model robustness and prevents CO, even against large perturbation radii. Our new method, termed N-FGSM, matches, or even surpasses, the robust accuracy of the regularized FGSM introduced by Andriushchenko and Flammarion [ (GradAlign). while providing a 3x speed-up.
To corroborate the effectiveness of our solution, we present an experimental survey of recently proposed single-step attacks and empirically demonstrate that N-FGSM trades-off robustness and computational cost better than any other single-step approach, evaluated over a large spectrum of perturbation radii (see Figure [ 100, and SVHN) and architectures (PreActResNer118 and WideResNer28-100 We will release our code reproduce the experiments.
middle and right panels), over several datasets (CIFAR-10, CIFAR-
Related Work
Since the discovery of adversarial examples, many defense mechanisms have been proposed, adver- sarial training being of the most popular and empirically validated. We can categorise adversarial

---

training methods based on how they approximate the perturbations applied to training samples. Multi- step approaches approximate an inner maximization problem to find the worst-case perturbation with several gradient ascent steps [37
18 20 While this provides better approximation, it is also more expensive. At the other end of the spectrum, single-step methods only use one gradient step to approx- imate the worst case perturbation. Goodfellow et al. [11 first proposed FGSM; Tramer et al. [32 pro- posed a new variant with an additional random step (R+GGSM), but observed that both methods were vulnerable to multi-step attacks. Shafahi et al. (26) proposed Free Adversarial Training (Free-AT), which successfully reduced the computational cost of training by using a single backward pass to com- pute both weight update and attack. Motivated by this, Wong et al. [35 explored variant of R+ R+FGSM, namely RS- RS-FGMM that uses a less restrictive form of noise and showed this can improve robustness for moderate perturbation radii at the same cost FGSM. Recently, Andriushchenko and Flammarion [1 proposed the GradAlign regularizer, Combining FGSM with GradAlign results in robust models even larger perturbation radii. However, GradAlign suffers from a threefold increase in the training cost to as compared to FGSM. The need for more efficient solutions has motivated a growing body of work whose goal is the design of computationally lighter single-step methods [TO] 15 33 23 19]
In this work, we revisit the idea of combining noise with the FGSM attack. Our method builds upon FGSM and intuitions from R+FGSM and RS-FGSM to combine it with random perturbations, however, we consider the noise step as data augmentation rather than part of the attack. This motivates us to stronger noise without clipping. As opposed to [14), our thorough study leads to a practically effective approach that yields robustness also against large perturbation radii.
3 Preliminaries on Single-Step Adversarial Training
Given a classifier fo X' + y parameterized by 0 and a perturbation set S, fa is defined as robust at X on the S if for all 8 S we have sa( + 6) = fo(r)). One of the most popular definitions for S is the e - foo ball, i.e., S = (ō ||0| I e). This is known as the 4x0. threat model which we adopt throughout this work.
To train networks that are robust against la threat models, adversarial training modifies the classical training procedure of minimizing loss function over dataset D {(x;, b:)]==:Nv of images I; E X' and labels yi E y. In particular, adversarial training instead minimizes the worst-case loss over the perturbation set S, i.e., training is the adversarially perturbed samples {(I; + 3,)] Under the Ix threat model, we can formalize adversarial training as solving the following problem:
min Σ max [(fo(1; + 8), s.t. ||ō||cc S, 8
(1)
where L is typically the cross- cross-entropy loss. Due to the difficulty of finding the exact inner maximizer, the most common procedure for adversarial training is to approximate the worst-case perturbation through several PGD steps 20). While PGD has been shown to yield robust models, its cost increases linearly with the number of steps. As a result, several works have focused on reducing the cost of adversarial training by approximating the inner maximization with single-step.
If the loss function is linear with respect to the input, the inner maximization of Equation (1) will enjoy a closed form solution. Goodfellow et al. [I]] leveraged this to propose FGSM, where the vevesariaal perturbation follows the direction of the sign of the gradient. Tramèr et al. [32] proposed adding a random initialization prior to FGSM However, both methods were later shown to be vulnerable against multi-step attacks, such as PGD. Contrary to prior intuition, recent work from Wong et al. (35] observed that combining a random step with FGSM can actually lead to promising robustness performance. In particular, most recent single-step methods approximate the worst-case perturbation solving the inner maximization problem in Equation (1) with the following general form:
8= v (1 a <v, + n).u:))
where 17 is drawn from a distribution Qt For example, when &e is the projection operator onto the to ball and is the uniform distribution [-c, cjdd where d is the dimension of X, this recovers RS-FGSM. Under a different noise setting where Ωt = (c a) sign (N(O.. Ia)) and by choosing the step size ar to be in [0,e]. we recover R+FGSM by Tramèr et al. (32. This was among the first works to explore the application of noise to FGSM, but did not report improvements over it. If we consider to be deterninistically 0 and &i to be the identity map, we recover FGSM. Finally, if we take FGSM and add gradient alignment regularizer, this recovers GradAlign.
3

---

50
Effective FGSH stop a (1255)
80
FGSM (Del
40
Noise le
Adversarial Acc.
Adversorial Arc
6.6
Noise
30
Noise
40
20
20
10
No Clipping
No Clipping
Clipping
Clipping
36
4
2E
36
4E
10 12 for train and eval (x 255)
16
Noise magnitude&
Noise magnitude
Figure 2: Left: Ablation of clipping vs not clipping around the clean sample for 8/255. Clipping
leads to a significant drop in robustness which increases with the strength the noise augmentations. Middle: Analysis of the effective FGSM step size after clipping. We observe that clipping leads to a decrease in the effective FGSM step size, thus, adversarial perturbations will be more similar to random noise. Right: N-FGSM (ours) when varying the noise magnitude le (e is divided by 255). Increasing the amount noise is key to avoiding CO. For (left) and (right) plots, adversarial accuracy is based on PGD-50-10 10 and experiments averaged over seeds.
Noise and FGSM
Previous methods that combined noise with FGSM, e.g,, R+FGSM [32 and RS-FGSM B5 have considered the noise random step integrated within the attack. Since it is a common practice to restrict adversarial perturbations to the ball,, we argue that this introduces a trade-off between the magnitude of the random step and that of the attack. For illustration, consider the purple lines corresponding to RS-FGSM in Figure (left). If the initial random step significantly larger than the ball, then the final clipping step will have noticeable impact the perturbation, possibly removing a considerable portion of the FGSM step (middle arrow). To prevent this from happening, R+FGSM and RS-FGSM restrict the random step to lie within the e e-ball, thereof implicitly entangling the noise magnitude and the attack strength.
Contrary to previous methods, we note that adding noise to the clean sample can be considered as form of data augmentation to be applied independently from the attack. We make two considerations from this perspective 1) When one performs data augmentation during adversarial training, the input after the corresponding transformation is the starting point to compute the adversarial perturbations, therefore, we argue that adversarial attacks should be centered around the noise- cuise-an samples. This motivates us to avoid clipping around the clean sample. 2) We not need to restrict the noise augmentation to lie inside an é ball, since its strength is disentangled from that of the attack. Thus, we can use stronger noise-augmenntations than previous methods.
These modifications lead to a novel adversarial training method that combines noise-based data augmentations with FGSM. We denote it as NoSM (N-FGSM), Following the notation in- troduced in Section we define the noise augmented sample as Taug I + where is sampled <). Then the adversarially perturbed from a uniform distribution on [-k, k)d (where we can have k >> samples have the following form:
FN Taug + a sign (V Ving C(fo (Faug), v))
(3)
This construction corresponds to augmenting the clean sample x with the perturbation defined in Equation
(2) where vp is the identity map and is the uniform ditribution spanning [-k, d We detail our full adversarial training procedure in Algorithm [ In what follows, we analyse the effect of treating the noise as data augmentation as opposed to treating it as a random step within the attack. In particular, we show that clipping around the clean sample (as done in RS-FGSM) can strongly degrade the robustness of the network. Moreover, we show that as we increase the e radīi of adversarial attacks, we need stronger noise perturbations than previously used prevent CO.
Clipping around clean sample hinders the effectiveness of perturbations. We analyse two variants, one where perturbations are clipped around the clean sample (as done in previous methods) and another where no clipping applied. In Figure (left), we report the robust accuracy using PGD- 50-10 (i.e., PGD attack with 50 iterations and 10 restarts) with 8/255 and observe that clipping significantly degrades the effectiveness of FGSM training. To understand this drop, consider the fol- lowing perturbations; (1) a baseline perturbation where only noise Grandom ((n)) and (2) per-

---

Algorithm 1 N-FGSM adversarial training
1: Inputs: epochs T, batches M, radius e, step-size a (default: e), noise magnitude k (default: 2e).
2: for 1,..., do
3: for = 1,... MM do
4:
7 ~ Uniform(-ke ki kijd
5:
Buy x¹ + Augment sample with additive noise.
6:
2 FGSM = Fang + a sign (= (
3f)) /I N-FGSM augmented sample. Compute gradients of model's weights
7:
Vg= (F
8: 0 = optimizer(0.. Vo) // Standard weight update, (e.g., SGD)
turbation that combines noise with FGSM ónu = ( sign V (zz[(f(I +)). Moreover, we consider two cases in which either define 3 as clipping operator or as the identity. We define the effective FGSM step size as the magnitude corresponding to the ratio Qlffeettiee 11 1amammllll/llllnn which measures the contribution of the FGSM step in the final perturbation compared to simply following the noise direction r7. In Figure 2 the effective magnitude of FGSM, thus, perturbations become more similar only using random noise. On the other hand, without clipping we always take the full step in the FGSM direction. This highlights the trade-off between noise magnitude and attack strength discussed above.
(middle), we observe that the clipping operator reduces
Larger noise is also necessary to prevent CO. As discussed above, previous work did not investigate the effects of using noise perturbations potentially larger than the attack strength. However, we empir- ically find that increasing the noise magnitude is key to avoiding CO. In particular, as seen in Figure 2 (right), when no clipping is performed, it is crucial that we augment with larger noise magnitude in order to prevent CO in all settings. We find the noise magnitude of k = 2e to work well in most of experiments, however, more extensive hyperparameter tuning might improve
results further.
Note that these results are contrary previous intuitions: Andriushchenko and Flammarion M suggested that the random step in RS-FGSM is not important per arguing that its main role reducing the & norm of the perturbations, so that the loss remains be approximately locally linear. In contrast, N-FGSM perturbations are larger on expectation than those of RS-FGSM, while they do not suffer from CO (refer Appendix Nj. We believe that our findings will lead to better understanding of the role of noise in avoiding CO in future work. Moreover, in Section 6 we conduct extensive analyses to show that, despite N-FGSM obtains larger perturbations, clean accuracy does not degrade and other methods do not benefit from simply increasing the strength of their attacks.
Why does noise augmentation avoid CO? Andriushchenko and Flammarion [ found that after CO, the gradients of the loss with respect to the input around clean samples became strongly misaligned, which is a sign of non- non-linearity, Moreover, Kim al. [15] showed that the loss surface of models suffering from CO appears distorted, i.e., there is a sharp peak in the loss surface along the FGSM direction, which seems to render FGSM ineffective (observe from Figure 5 how after after CO, visually, FGSM perturbations change drastically). In order to prevent CO, GradAlign explicitly regularizes the loss surface so it remains linear. To investigate further, we plot the loss surface at the end of training for different methods (see Figure [4 in Appendix) and find that, while FGSM or RS RS-FGSM lead to a distorted loss, N-FGSM obtains a non-distorted loss surface similar to that obtained by GradAlign regularizer. Thus, it seems that adding strong noise-augmentations implicitly regularizes the loss landscape, leading to more effective single-step attacks. This aligns with previous work that theoretically link noise augmentations with regularizer that encourages Lipschitzness 3
5 Robustness Evaluations and Comparisons
We compare N-FGSM against several adversarial training methods, on a broad range of e Ioc.
radii. Following Wong et al. [35], we evaluate adversarial robustness on CIFAR-10/100 [16] and
SVHN [21) with PGD-50-11 10 attacks, using both PreactResNet18 [13 and WideRessNe28-110 [36].
Evaluations with AutoAttack 66] are also in Appendix G
"The denominator ||æ||2 is simply normalize the l norm and be comparable the FGSM step size a.
5

---

CIFAR100 Dataset
SVHN Dataset
50
N-FGSM (ours)
Free:AT
80
GracAlign
Kim ct al.
40
MultiGrad
ZeroGrad
R5-FGSM
FGSM
Advcsarial ACC.
60
30
Adversaria
40
20
10
20
2
10 12 14 16
6
8 10 12
E for training and evaluation (x7 255)
E for training and evaluation (x/ 255)
Figure 3: Comparison of single-step methods on CIFAR- 100 (left) and SVHN (right) with PreactRes- Net18 over different perturbation radius (e is divided by 255). Our method, N-FGSM, can match or surpass prior art results while reducing the cost by a 3 factor. Adversarial accuracy is based on PGD-50-10 and experiments are averaged over seeds. Legend is shared among plots.
5.1 Comparison against Single-Step Methods
We start by comparing N-FGSM against other single-step methods. Note that not all single-step methods equally expensive, since they may involve more less computationally demanding operations. For instance, GradAlign uses a regularizer that is considerably expensive, while MultiGrad requires evaluating input gradients on multiple random points. For a comparison of training costs of different single-step methods, we refer the reader to Figure (right). We use RS-FGSM and Free-AT with the settings recommended by Wong et al. [35] . We apply GradAlign with hyperparameters reported in the official repository ZeroGrad and Kim et al. [15] do not have a recommended set of hyperparameters; for a fair comparison we ablate them and select the ones with highest adversarial accuracy (for every and dataset). We train on CIFAR- 100--0/11001 00 for 30 epochs and on SVHN for 15 epochs with a cyclic learning rate. Only for AT, we use 96 and 48 epochs for CIFAR- 10/100000000 and S SVHN, respectively, to obtain comparable results following Wong et al. [35] CIFAR- 10 results are in Figure (middle)) whereas CIFAR-100 and SVHN are in Figure 3
As observed in Figure [ and Figure 3 FGSM and RS-FGSM suffer from CO for larger e attacks on all reported datasets. For instance, RS-FGSM fails against attacks with 8/255 on CIFAR-10 and CIFAR-100 and against e = 6/255 on SVHN. With appropriate hyperparameters. ZeroGrad is able to consistently avoid CO. However, it obtains sub-par robustness compared to N-FGSM and GradAlign, especially against large attacks. Neither MultiGrad nor Kim et al. | avoid CO in all settings despite being more expensive. Free-AT also suffers from CO on all three datasets as also observed by Andriushchenko and Flammarion [. In contrast. N-FGSM avoids CO on all datasets, achieving comparable or superior robustness to GradAlign while being 3 times faster.
5.2 Comparison against Multi-Step Attacks
In Section we compared the performance of single-step methods and observed that N-FGSM able to match or surpass the ofate-of-the-art method, iee, GradAlign, while reducing the computational cost by factor 3. In this section, we compare the performance of N-FGSM against multi-step attacks. In particular, we compare against PGD-2 with the same training settings as described in Section 5.1 PGD-x denotes x iterations and no restarts.
f/2 and PGD-10 with a 2/235, keeping
In Figure 4 we observe that PGD-2, despite being a multi-step method, still suffers from CO for larger e as opposed to our proposed N-FGSM. On the other hand, despite achieving comparable clean accuracies, there is a gap in adversarial accuracies between P0D-10, and other single-step methods that grows with perturbation size. This can be partially expected since the search space for adversaries grows exponentially with e; and PGD, with more iterations, can explore it more thoroughly. Nevertheless, computing a PGD- 10 attack is 10x more expensive to N-FGSM. An important direction for future work would be addressing this gap and analysing, both theoretically and empirically, whether single-step methods can match the performance of their multi step counterparts.
https//ggthhbbcoommmmmm/roouuuuu fast-aaddddddd---d------------a- training/

---

CIFAR10 Dataset
SVHN Dataset
Clean and Adv AcC.
80
80
60
and Adv.
60
90
40
N-FGSM
N-FGSM
20
GradAlign
GradAlign
PGD-10
PGD-10
PGD-2
PGD-2
10
12 14
16
6
8
10
12
for trainining and evaluation (K/ 255)
for trainining and evaluation (x 255)
Figure 4: Comparison of N-FGSM and GradAlign with multi-step methods on CIFAR-10 (Left) and SVHN (Right) with PreactResNet18 8 over different perturbation radii (e is divided by 255). Despite all methods achieving comparable clean accuracy (dashed lines), there is gap in robust accuracy between PGD-10 10 and single-step methods. However, note that PGD-10 is 10x more expensive than N-FGSM. Adversarial accuracy is based on PGD-50-10 and experiments are averaged over 3 seeds.
5.3 Analysis of Gradients and Adversarial Perturbations
To gain further insights into CO, we visually explore the perturbations generated with FGSM, RS- FGSM, N-FGSM, and PGD-10 attacks. We show that N-FGSM generates perturbations that exhibit behavior similar to PGD-10. In particular, for a given test sample, we average the adversarial perturbations (6) and gradients across several epochs at the beginning of training (Epoch 2 to 8) and at the end (Epoch 24 to 30) and visualise them in Figure [3] (see also Figure 12 in Appendix for more examples). We observe that, during early stages in training all, methods generate consistent and interpretable 5. However, after CO, FGSM and RS-FGSM generate ó that are harder to interpret, similarly to their gradients. On the other hand, we observe that N-FGSM provides consistent and interpretable 5 throughout training, similar to those generated by PGD-10. This provides further evidence that N-FGSM enjoys similar properties to the more expensive PGD-10 training.
Figure analyzes the gradients and å throughout the test set. Aside from loosing interpretability, post-CO the gradient norm increases by several orders of magnitude for FGSM and RS-FGSM while it remains low for N-FGSM and PGD-10. We also compute the effective rank of 5 for each example before and after CO to measure the consistency of á before and after CO. We consider three training intervals, (Epoch 2 to 8): before CO for all methods; (Epoch 16 to 22): after FGSM suffers CO but not RS-FGSM; (Epoch 24 to 30): after both FGSM and RS-FGSM suffer CO. Prior CO. PGD-10 has a larger effective rank (i.e. the perturbations span a larger subspace) than FGSM and RS-FGSM. N-FGSM has the highest effective rank, arguably due to the higher noise magnitude. Note that RS-FGSM, which has a smaller noise magnitude and clipping, also has a larger effective rank than FGSM, however, the difference is much lower. When either FGSM or RS-FGSM suffer from CO, the effective rank of their ò increases significantly above that of PGD-10 and N-FGSM. This would suggest that loose consistency after CO and is aligned with our visualizations in Figure 5 All of these show properties of 8 and gradients that are consistent across methods (N- (N-FGSM and PGD) that avoid CO and different from methods like RS-FGSM and FGSM, which suffer from CO.
6 Increasing Adversarial Perturbations
In Section 4
we observed that removing clipping and increasing the noise magnitude were both necessary for the improved performance of N-FGSM. However, as discussed in Theorem N.2 this will result in an increase the squared norm of the training perturbation 6N-FGSM as compared to FGSM. In this section, we perform further ablations to corroborate that it is indeed the increase in noise mag- nitude- - and not the mere increase of the perturbation's 's magnitude - that helps to stabilize N-FGSM.
Increasing a alone is not sufficient. N-FGSM combines a noise perturbation with an FGSM step. Thus, we can increase the perturbation magnitude by increasing any of the two. This begs the question: Is it sufficient to increase the N-FGSM step-size a to avoid CO without adding any noise? We observe in Figure g (A) that training without noise (essentially, FGSM) leads to CO, with robust accuracy
WWe compute effective rank the number of singular vectors required to explain 90% of the variance.
7

---

6.0
Clean N-FSS PGD-10
PGD-10
RS FGSM
100 (1/9 [M
N-FGSM
Effective rank:
5.5
SeRe S35 5333 2430
1.0
TGSM
RS-FOSM
4.0
N-FCSM
PGD 30
100
20 30
3.5
Ep
Ep 16:22 Ep 24-30
Epochs
Training intervals
Adversarial perturbations (20)
Xiter
Gradients
Figure 5: Left: Mean & norm of per-sanmle-rradients across all test set samples. After CO, both FGSM and RS-FGSM gradients increase by several orders of magnitude. Middle: Effective rank of perturbations (6) across three training intervals Ep 2-8 before CO for all methods; Ep 16-22: after FGSM presents CO but not RS-FGSM; Ep 24-30: after both FGSM and RS-FGSM had CO. Right: Visualization of ò and gradients averaged across several epochs at the beginning (top) and end (bottom) of training. After CO, FGSM and RS-FGSM obtain d and gradients that are hard to interpret.
equal to zero, even for large values of a. This indicates that it is not an increase in the perturbation norm, but the combination with noise which plays essential role in circumventing CO for N-FGSM.
Increasing a requires adjusting the noise magnitude. As observed in Figure 6 (A), increasing for N-FGSM leads to CO if the noise magnitude is not large enough. For example, while noise magnitude k 1c and an adversarial step size a 1.25c yield robust accuracy of 49. 08,68, increasing to 1.5c while keeping the same noise magnitude results in CO - with robust accuracy equal to zero. This further suggests that an increase in the adversarial step-size a requires commensurate increase in the noise magnitude. We find that setting the noise magnitude k = 2 works well for most settings.
Larger noise perturbations preserve clean accuracy. Increasing the norm of training perturbations by increasing results drop in the clean accuracy (discussed later in Section 7 This has also been observed in prior works B3 However, we show in Figure 4 that the clean accuracy for N-FGSM is similar to that of Grign despite the magnitude of the perturbations being larger. We ablate the effects of adversarial and noise perturbations on the clean accuracy in Figure 6 (B): we observe that augmenting training samples with noise alone (i.e., = 0) has a much milder effect on the clean accuracy than augmenting in an adversarial direction. In general, increasing noise is more forgiving on the clean accuracy than increasing the adversarial step size. This not surprising, considering that moving in random directions along the input space has significantly lower impact on the loss than moving along the FGSM direction (see Figure [4]in in the Appendix) and that training with noise alone does not provide any significant robustness against larger attacks (for more detailed ablation, see Appendix Figure [6].
Other methods do not benefit from larger training c. As previously mentioned. N-FGSM perturbations have Lx -nom larger than e. We have seen that the benefits of N-FGSM can not be reproduced by simply increasing a without increasing the noise. However, for the sake of completeness, we also ablate other single-step step baselines by using a larger e during training, while testing with a fixed 6 8/255 decrease in robustness for most methods; for instance, PGD-50-10 accuracy for RS-FGSM drops from 46.08 + 0.18 when training with $/255 to 0.0 with 12/255. In two cases (GradAlign and MultiGrad), we observe a small increase, with the highest increase being for GradAlign, which improves from 48.14 + 0.15 to 50.6 + 0.45; yet, the clean accuracy drops from 81.9 + 0.22 to 73.29 + 0.23. This is similar to increasing a for N-FGSM (see Figure (C)). However, this is tied to significant degradation of clean accuracy. All in all, taking into account both clean and robust accuracy, we conclude that all single-step baselines suffer from either CO or a severe degradation in their clean accuracy when increasing the training Full results presented in Table in Appendix.
10. We observe that increasing Grain seems to lead to
7 Additional Ablations
Hyperparaaeetrr selection. While FGSM relies on a fixed step-size (i.e., a = <), Wong et [35 explored different values of a for RS-FGSM, finding that an increase of the step-size improves the ad- versarial accuracy - up to a point where CO occurs. We also ablate the value of a for N-FGSM in Fig- ure 6 (C). We find that by increasing the noise magnitude, N-FGSM can use larger values than RS-
8

---

Comparison of Training Schedules
50
Clean Acc
Robust Ace
40
30
20
Noise 1
Nolse Ox
5r
Acc.
90
Claan 3
80
70
robust FGSM clean
83.18
Long schedule: Final model
Long schedole: Best model
+
36.56 0.26
10
Nolse 2z
88
Uniffom
50
80.8
0.36
48.48 1 0.27
Adversarial
Fast schodule: Final model
0.5g 0,75c 1,25 1.5g Step size for N-FoSM attack
Radius of perturbation (x/ 2551
N-FGSM stop 5220 16 B)
12 80.58 + 0.22
48.12 + 0.07
Figure 6: Different ablations on N-FGSM parameters and training schedule. From left to right: A: Adversarial accuracy when varying step-size a and noise magnitude & (e = 8). Increasing does not suffice to prevent CO, we must also increase the noise magnitude. B: Clean accuracy after training with random adversarial perturbations. With comparable radius, random perturbations have a much milder effect than adversarial. C: Ablation of step size a in N-FGSM c = 8, k = 2c. As we increase the magnitude of the FGSM perturbation we observe increase in robustness coupled with a drop on the clean accuracy. D: Comparison of the "fast" training schedule from (35) and "long"
training schedule described in (24). N-FGSM shows robust oberfitting but not CO with the long schedule, Adversarial accuracy is based on PGD-50-10 and experiments are averaged over 3 seeds.
FGSM, without suffering from CO. This leads to an increase in the adversarial accuracy at the expense of decrease in the clean accuracy. In light of this trade-off, we also use a = for N-FGSM. Regard- ing the noise hyperparameter k, we find that A 2e works in all but one SVHN experiment (e 12, in which we set k = 3e). In comparison, GradAlign regularizer hyperparameter or ZeroGrad quantile value need to be modified for every radius with noticeable shift between CIFAR- 10 and SVHN hyperparameters, suggesting they may require additional tuning when applied to novel datasets.
Long vs fast training schedules. Throughout our experiments, we used the RS-FGSM training setting introduced in [35]. However, Rice et al. [24 suggest that a longer training schedule coupled with early stopping may lead to boost in performance. Kim et al. [15] and Li et al. [1 report that longer training schedules increase the chances of CO for RS-FGSM and that this limits its performance. We test the longer training schedule with N-FGSM and find that it does not suffer from CO. However, it does suffer from robust overfitting, i.e., adversarial accuracy on the training set is larger than on the test set as described in (24] for PGD-10. Notice the difference between the robust accuracy of the final and best models in Figure 6 (D). Interestingly, although we observe a slight increase in performance when using the long training schedule, find the fast training schedule to be remarkably competitive. See more results in Appendix D including a comparison to GradAlign.
Experiments with WideResNNe28-100 10. We also compare the performance of all methods on WideResNN:28-100 10 (36 architecture in Figure and Figure D in Appendix. As in the experiments with PreActResNett18, N-FGSM obtains the best performance/cost trade-off. We had to increase the regularizer hyperparameter for GradAlign (compared to the settings for PreActResNer18) to prevent CO on CIFAR- and, to surprise, we could not find a competitive hyperparameter setting for GradAlign on the SVHN dataset for E 2 6. We tried both increasing the regularizer hyperparameter and decreasing the step size a, but some or all runs led to models close constant classifier for each setting. We do not claim that GradAlign will not work, but finding a good configuration might require further tuning. The default configuration for N-FGSM (a = c, 2c) works well in all settings except for e = 16 on CIFAR-10 and c = 10, 12 on SVHN. For CIFAR-10. we increase the noise magnitude to & 4e. For SVHN, find that decreasing works better than increasing the noise. In both cases, N-FGSM yields nontrivial adversarial accuracy.
order
Experiments on Imagenet. We present results on the Imagenet dataset 17 in Table Due to the high computational demands of Imagenet training and testing we focus on the main basclines of comparable cost to FGSM. Namely FGSM, RS-FGSM and N-FGSM. We observe that FGSM presents CO for e = 6/255 while neither RS-FGSM nor N-FGSM present CO. However, N-FGSM has better robustness. For instance, at e 6/255 N-FGSM obtains PGD50-10 10 accuracy of 17.12% while RS-FGSM yields 16.5% and FGSM 0.08% (due to CO). Thus, N-FGSM also avoids CO in ImageNet, improving robustness over same-cost baselines. For experimental details refer to Appendix [
have been proposed to improve performance of single step methods at moderate perturbation radii 6/255. However, we observe that with the default settings (which use a version of RS-FGSM with Bernoulli noise) they lead to CO for larger Then compare them with N-FGSM + Regularizer where apply
Combining N-FGSM with additional regularizers. Recent works 29 30

---

Table 1: Clean accuracy (top) and PGD50-10 10 accuracy (bottom) of N-FGSM and other same-cost baselines on Imagenet dataset. We observe that FGSM presents CO for 9/255 while both RS-FGSM and N-FGSM avoid CO. N-FGSM has consistently better robustness than baselines.
9/255
9/255
9/255
54.72
48.50
48.55
FGSM
38.21
25.86
0.08
56.29
50.81
47.67
RS-FGSM
36.86
25.12
16.49
54.39
47.56
47.70
N-FGSM
38.07
26.28
17.12
their proposed regularizers to N-FGSM. If we apply GAT' or NuAT regularizers to N-FGSM then we do observe CO and usually a boost in performance. For instance, at c 10/25,, GAT has a robust acc (with PGD50-10) of 43.34 + 0.23 while N-FGSM+GAT regularizer obtains 44.97 + 0.07, in comparison plain N-FGSM has 41.56 + 0.16. This is extremely compelling as suggests N-FGSM can be combined with other regularizers designed to improve FGSM performance and mutually benefit each other. Full results of the comparison are presented in Table in Appendix
8 Conclusion
In this work, we explore the role of noise and clipping in single-step adversarial training. Contrary to previous intuitions, we show that increasing the noise magnitude and removing the - loc constraint leads an improvement adversarial robustness while maintaining competitive clean accuracy. These findings led us to propose N-FGSM, a simple and effective approach that can match or surpass the performance of GradAlign [1, while achieving 3x speed-up.
We perform an extensive comparison with other relevant single-step methods, observing that all of them achieve sub-optimal performance and most of them are not able to avoid CO for larger e attacks. Moreover, we also analyze gradients and adversarial perturbations during training and observe that they have similar behaviour for N-FGSM and PGD-10 opposed to other methods that present CO such as FGSM and RS-FGSM. However, despite impressive improvements of single-step adversarial training methods, there is still a gap between single-step and multi-step methods such as PGD- 10 as we increase the radius. Therefore, future work should put an emphasis on formally understanding the limitations of single-step adversarial training and explore how, if possible, this gap can be reduced.
Acknowledgments and Disclosure of Funding
We thank Guillermo Ortiz-Jiménez for the fruitful discussions and feedback. This work is supported by the UKRI grant: Turing AI Fellowship EP/W002981/1 and EPSRC/MURI grant: EP/N011474/11. We would also like thank the Royal Academy Engineering and FiveAI. A. Sanyal acknowledges support from the ETH AI Center postdoctoral fellowship. Pau de Jorge is fully funded by NAVER LABS Europe.
References
[1] Maksym Andriushchenko and Nicolas Flammarion. Understanding and improving fast adver- sarial training. In Neural Information Processing Systems (NeurIPS), 2020.
[2] Battista Biggio and Fabio Roli. Wild patterns: Ten years after the rise of adversarial machine learning. Pattern Recognition, 2018.
[3] Chris Bishop. Training with noise equivalent tikhonov regularization, Neural computa- tion, 7(1):108-116, 1995.
10

---

[4] Adith Boloor, Xin He, Christopher D. Gill, Yevgeniy Vorobeychik, and Xuan Zhang. Simple physical adversarial examples against end-to-end autonomous driving models. arxiv: 57, 2019.
[5] Moustapha Cisse, Piotr Bojanowski, Edouard Grave, Yann Dauphin, and Nicolas Usumier. Parseval networks: Improving robustness to adversarial examples. In International Conference on Machine Learning (ICML), 2017.
[6] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International Conference on Machine Learning (ICMI.), 2020.
[7] Jacob Devlin, Ming-Wei Changm, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL HLT), 2019.
[8] Alhussein Fawzi, Seyed-
Moosavi-Dezfooli, Pascal Frossard, and Stefano Soatto. Fm- pirical study of the topology and geometry of deep networks. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2018.
[9] Justin Gilmer, Nicolas Ford, Nicholas Carlini, and Ekin Cubuk. Adversarial examples are a natural consequence of test error in noise. In International Conference on Machine Learning (ICML), 2019.
[10] Zeinab Golgooni, Mehrdad Saberi, Masih Eskandar, and Mohammad Hossein Rohban. Zerograd: Mitigating and explaining catastrophic overfitting in fgsm adversarial training. WrXiv:2/33.15466 2021.
[11] Ian Goodfellow. Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. International Conference on Learning Representations (ICLR), 2015.
[12] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Sur- passing human-level performance on imagenet classification. In IEEE International Conference on Computer Vision (ICCV), 2015.
[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. European Conference on Computer Vision (ECCV), 2016.
[14] Peilin Kang and Scyed-Mohsen
Understanding catastrophic overfitting in adversarial training. arXiv:2/05.02944 2021.
[15] Hoki Kim, Woojin Lee, and Jaewook Lee. Understanding catastrophic overfitting in single-step adversarial training. In AAAI Conference Artificial Intelligence (AAAI), 2021.
[16] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009.
[17] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Neural Information Processing Systems (NeurIPS), 2012.
[18] Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning scale. In International Conference on Learning Representations (ICLR), 2017.
[19] Bai Li. Shiqi Wang, Suman Jana, and Lawrence Carin. Towards understanding fast adversarial 2020.
training. arXiv:2006.03000
[20] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. In International Conference on Learning Representations (ICLR), 2018.
[21] Yuval Netzer, Tao Wang. Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In Neural Information Processing Systems (NeurIPS), Workshops, 2011.
[22] Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as a defense to adversarial perturbations against deep neural networks. In IEEE symposium on security and privacy (SP), 2016.
[23] Geon Yeong Park and Sang Wan Lee. Reliably fast adversarial training via latent adversarial perturbation. In International Conference on Learning Representations (ICLR), Workshops, 2021.
11

---

[24] Leslie Rice, Eric Wong, and Zico Kolter. Overfitting in adversarially robust deep learning, In International Conference on Machine Learning (ICML), 2020.
[25] Amartya Sanyal, Varun Kanade, and Philip H. S. Torr. Robustness via deep low-rank represen- tations. 1okiv:/804.07700 2018.
[26] Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! Neural Information Processing Systems (NeurIPS), 2019.
[27] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che. Julian Schrittwieser. Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 2016.
[28] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
[29] Gaurang Sriramanan, Sravanti Addepalli, Arya Baburaj, al. Guided adversarial attack for evaluating and enhancing adversarial defenses. Advances in Neural Information Processing Systems, 33:20297-20300 2020.
[30] Gaurang Sriramanan, Sravanti Addepalli, Arya Baburaj, al. Towards efficient and effective adversarial training. Advances in Neural Information Processing Systems, 34:1 1821-11833,
2021.
[31] Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Good- fellow, and Rob Fergus. Intriguing propertics of neural networks. In International Conference on Learning Representations (ICLR), 2014.
[32] Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Bonch, and Patrick McDaniel. Ensemble adversarial training: Attacks and defenses. In International Conference on Learning Representations (ICLR), 2018.
[33] BS Vivek and R Venkatesh Babu. Single-step adversarial training with dropout scheduling. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2020.
[34] Lily Weng, Huan Zhang, Hongge Chen, Zhao Song, Cho-Jui Hsieh, Luca Daniel, Duane Boning, and Inderjit Dhillon. Towards fast computation of certified robustness for relu networks. In International Conference Machine Learning (ICML), 2018
[35] Eric Wong, Leslic Rice, and J. Zico Kolter. Fast is better than free: Revisiting adversarial training. In International Conference on Learning Representations (ICLR), 2020.
[36] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC British Machine Vision Conference (BMVC). 2016.
[37] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International Conference on Machine Learning (ICMI.), 2019.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] We provide experimental validation for all our claims.
(b) Did you describe the limitations of your work? [Yes] Yes, in conclusions and when appropriate along the paper.
(c) Did you discuss any potential negative societal impacts of your work? [N/A] We do not consider research on training robust networks can have negative impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments...
12

---

(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] We include the
URL to our code in the abstract.
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] When relevant in the text.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] All plots or numbers have standard deviation. Although sometimes too small to be visible.
(d) Did you include the total amount of compute and the type of resources used (e.g. type of GPUs, internal cluster, or cloud provider)? [Yes] Yes, see appendix R
4. If you are using existing assets (e.g.. code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either the supplemental material or as URL? [Yes] We include the URL to our code in the abstract.
(d) Did you discuss whether and how consent obtained from people whose data you're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "iCB51SmKeu", "reviewer": "Reviewer_cQ7D", "summary": "The authors have proposed a single-step adversarial training procedure, termed Noise-FGSM (N-FGSM), by considering added noise step as data augmentation rather than part of the attack. The resultant method centers adversarial perturbations with respect to noise-augmented samples (rather than wrt the clean one) and does not clip around the clean samples either, which allows much larger magnitude of the noise during the attack steps. Doing so can effectively avoid catastrophic overfitting (CO) in which a model that is adversarially trained with single-step FGSM suddenly becomes vulnerable to multi-step attacks. Also, the clean accuracy does not hurt. Experimentally, the proposed N-FGSM method reaches on-par or surpasses the SOTA method GradAlign while enjoying 3x speed-up.\n", "questions": "I do not have questions at this point. \n", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nPreventing the catastrophic overfitting (CO) in which a model suddenly becomes vulnerable to multi-step attacks despite remaining robust to single-step attacks is quite important in adversarial defense. The authors have proposed an adversarial training procedure where the noise is used as a form of data augmentation. The paper has good organization and presentation. Although the proposed method is not significantly different from prior work in terms of mathematical formulation, I do believe that it has its own merit and the authors have provided pretty thorough discussion and analysis in Section 4 and 5, helping readers to understand better the difference between the proposed method and prior work on single-step adversarial training that also incorporates noise. Experimentally, the proposed method performs competitively against prior work. Also, some important ablative studies are also conducted. \n\nWeaknesses:\nI think overall this is a solid paper. At this reviewing stage, I do not spot apparent weaknesses. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "VWvsD2l9pAo", "reviewer": "Reviewer_khDd", "summary": "This paper focus on the catastrophic overfitting issue in FGSM adversarial training. The work proposes to avoid clipping adversarial perturbations to mitigate the CO and they regard adding noise as a data augmentation method. This paper analyze the proposed tricks in detail and provide a huge of experimental results to prove them.  ", "questions": "- Do you provide the comparisons with other single-step methods against AutoAttack?\n- Do you consider to compare with this single-step AT method[1]?\n\nCitations:\n[1] Guided Adversarial Attack for Evaluating and Enhancing Adversarial Defenses", "limitations": "See above", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- This work is easy to follow.  And authors use clear math formulas to show their methods. Their experimental results look good.\n- They regard the random initialization  as a data augmentation trick which can help explain many problems.\n\nWeakness:\n- Although the authors provide some theoretical proofs and the loss landscapes,  the proof and figures are more about the influence brought by the noise augmentation. They dose not show why the larger noise magnitude can prevent the CO better. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "QGcPa_Z32Ye", "reviewer": "Reviewer_QcoK", "summary": "This paper finds that using a stronger noise around the clean images combined with not clipping can prevent catastrophic overftting and propose Noise-FGSM (N-FGSM) to improve model robustness. Evaluations on some datasets demonstrate that the proposed method outperform existing methods.", "questions": "1. supplementary material is same as the submitted manuscript\n\n2. The main problem is that the authors have not compared their approach properly with some other methods that exist in the literature [1][2][3]\n\n3. Lack of comparative experiments on large datasets, such as ImageNet. Previous works all conduct experiments on ImageNet\n\n[1] Andriushchenko, M., Flammarion, N.: Understanding and improving fast adversarial training. In: Annual Conference on Neural Information Processing Systems (2020)\n\n[2] Sriramanan, G., Addepalli, S., Baburaj, A., et al.: Guided adversarial attack for evaluating and enhancing adversarial defenses. Advances in Neural Information Processing Systems 33, 20297–20308 (2020)\n\n[3] Sriramanan, G., Addepalli, S., Baburaj, A., et al.: Towards efficient and effective adversarial training. Advances in Neural Information Processing Systems 34 (2021)", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The novelty in my opinion is limited.  Using a larger noise combined with not clipping to conduct fast adversarial training is unfair to other  fast adversarial training methods. The authors should compare the proposed method with other fast adversarial training with  a larger noise combined with not clipping. Moreover, the training epoch is very small. Some research [1][2][3] have found that when the epoch of  fast adversarial training is set to larger, the fast adversarial training can also meet catastrophic overftting, such as 100, 200.\n\n[1] Andriushchenko, M., Flammarion, N.: Understanding and improving fast adversarial training. In: Annual Conference on Neural Information Processing Systems (2020)\n\n[2] Sriramanan, G., Addepalli, S., Baburaj, A., et al.: Guided adversarial attack for evaluating and enhancing adversarial defenses. Advances in Neural Information Processing Systems 33, 20297–20308 (2020)\n\n[3] Sriramanan, G., Addepalli, S., Baburaj, A., et al.: Towards efficient and effective adversarial training. Advances in Neural Information Processing Systems 34 (2021)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "MIAJr5o65SE", "reviewer": "Reviewer_1zew", "summary": "This paper revisits the generation process of adversarial perturbations in adversarial training with single-step FGSM attack. To solve the catastrophic overfitting problem, they propose to add random noises on benign images, and utilize FGSM to optimize adversarial perturbations based on these augment samples without restricting perturbations to lie inside ε-ball. This unrestricted method helps to balance the effect of noises and perturbations. Their experiments demonstrate that the proposed method achieves comparable robustness while being faster than previous methods.", "questions": "1. How will the performance of this method change after improving ε of PGD?\n2. Why noise augmentation can works as a regularizer?\n", "limitations": "Yes\n", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. The presentation of this paper is good.\n2. Fig 1 (left) provides a good illustration for Noise-FGSM. And the analysis between the noise magnitude and the attack strength is interesting.\n3. This paper provides a different view to understanding single-step FGSM in adversarial training.\n\nWeaknesses\n1. The trade-off between noise magnitude and attack strength should be discussed in detail. Maybe authors can dynamically adjust these two to further prove their points.\n2. The clipping ensures that generated adversarial examples are still lie inside the class space. The unrestricted method with a larger noise may undermine this assumption. In addition, the toy experiment (Fig 2 (right)) shows that the unrestricted method works well when the magnitude of noise is equal to 2ε or 4ε. It suggests that larger noise (4ε) may not further prevent CO. The statement of “Larger noise is also necessary to prevent CO” is not rigorous.\n3. As shown in line 173-183, N-FGSM obtains a non-distorted loss surface similar to that obtained by GradAlign regularizer. It would be better if the authors could explain why noise augmentation can works as a regularizer?\n4. RS-FGSM and GradAlign both performed experiments on ImageNet. Thus, the experiments on ImageNet should also be included in this paper.\n5. How will the performance of this method change after improving ε of PGD?\n6. Overall, this paper proposes an effective but simple method, and some points are not clear.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept