Year: 2021

Abstract:
Recently, Wong et al. (2020) showed adversarial training with single-step FGSM leads to a characteristic failure mode named catastrophic overfitting (CO), in which a model becomes suddenly vulnerable to multi-step attacks. Moreover, they showed adding a random perturbation prior to FGSM (RS-FGSM) seemed to be sufficient to prevent CO. However, Andriushchenko & Flammarion (2020) observed that RS-FGSM still leads to CO for larger perturbations and argue that the only contribution of the random step is to reduce the magnitude of the attacks. They suggest a regularizer (GradAlign) that avoids CO but is significantly more expensive than RS-FGSM. In this work, we methodically revisit the role of noise and clipping in single-step adversarial training. Contrary to previous intuitions, we find that not clipping the perturbation around the clean sample and using a stronger noise is highly effective in avoiding CO for large perturbation radii, despite leading to an increase in the magnitude of the attacks. Based on these observations, we propose a method called Noise-FGSM (N-FGSM), which attacks noise-augmented samples directly using a single-step. Empirical analyses on a large suite of experiments show that N-FGSM is able to match or surpass the performance of GradAlign while achieving a 3x speed-up.

heading: 1 INTRODUCTION
text: Deep neural networks have achieved remarkable performance on a variety of tasks (He et al., 2015; Silver et al., 2016; Devlin et al., 2019). However, it is well known that they are vulnerable to small worst-case perturbations around the input data – commonly referred to as adversarial examples (Szegedy et al., 2014). The existence of such adversarial examples poses a security threat to deploying models in sensitive environments (Biggio & Roli, 2018). This has motivated a large body of work towards improving the adversarial robustness of neural networks (Goodfellow et al., 2015; Papernot et al., 2016; Tramèr et al., 2018).
The most popular family of solutions to obtain robust neural networks is based on the concept of adversarial training (Goodfellow et al., 2015; Madry et al., 2018). In a nutshell, adversarial training can be posed as a min-max problem where instead of minimizing some loss over a dataset of clean samples, we augment the inputs with worst-case perturbations that are generated online during training. However, obtaining such perturbations is NP-hard (Weng et al., 2018) and hence, different approaches have been suggested to approximate them. They are commonly referred to as adversarial attacks. In their seminal work, Goodfellow et al. (2015) proposed the Fast Gradient Sign Method (FGSM), that generates adversarial attacks by running one step of gradient ascent on the loss function. However, while FGSM-based adversarial training provides robustness against single-step FGSM adversaries, Madry et al. (2018); Tramèr et al. (2018) showed that these models were still vulnerable to multi-step attacks, namely those allowed to perform multiple gradient ascent steps instead of a single one. Notably, Madry et al. (2018) introduced the multi-step Projected Gradient Descent (PGD) attack.
PGD-based attacks have now become the de facto standard for adversarial training; yet, their cost increases linearly with the number of steps. As a result, several works have focused on reducing the cost of adversarial training by approximating the worst-case perturbations with single-step attacks (Wong et al., 2020; Shafahi et al., 2019; Vivek & Babu, 2020). In particular, Wong et al. (2020) studied FGSM adversarial training and discovered that it suffers from a characteristic failure
mode, in which a model suddenly becomes vulnerable to multi-step attacks despite remaining robust to single-step attacks. This phenomenon is referred to as catastrophic overfitting. Moreover, they argued that adding a random perturbation prior to FGSM (RS-FGSM) seemed sufficient to prevent catastrophic overfitting and yield robust models. Recently, Andriushchenko & Flammarion (2020) observed that RS-FGSM still leads to catastrophic overfitting as we increase the perturbation radii. They suggested a regularizer (GradAlign) that, on the one hand avoids catastrophic overfitting in all the settings they considered, but on the other hand requires the computation of a double derivative – which significantly increases the computational cost compared to RS-FGSM. This has motivated other works that aim at achieving the same level of robustness with a lower computational overhead (Golgooni et al., 2021; Kim et al., 2021).
In this paper, we revisit two key components that are common among previous works combining noise and FGSM (Tramèr et al., 2018; Wong et al., 2020): the role of noise, i.e. the random step, and the role of the clipping step. In Section 4.1, we study how these two components affect model robustness; our experiments suggest that adding noise with a large magnitude in the random step and removing the clipping step improves model robustness and prevents catastrophic overfitting, even against large perturbation radii. We combine these observations and propose a new method called Noise-FGSM (N-FGSM), an illustration of which is presented in Figure 1 (left). N-FGSM allows to match, or even surpass, the robust accuracy of the regularized FGSM introduced by Andriushchenko & Flammarion (2020), while providing a 3× speed-up. To corroborate the effectiveness of our solution, we present an experimental survey of recently proposed single-step attacks and empirically demonstrate that N-FGSM trades-off robustness and computational cost better than other single-step approaches, evaluated over a large spectrum of perturbation radii (see Figure 1, middle and right panels), over several datasets (CIFAR-10, CIFAR100, and SVHN) and architectures (PreActResNet18 and WideResNet28-10). We will release our code reproducing all experiments.

heading: 2 RELATED WORK
text: Since the discovery of adversarial examples, many defense mechanisms have been proposed. Preprocessing techniques try to modify the input image to neutralize adversarial attacks (Guo et al., 2018; Buckman et al., 2018; Song et al., 2018). Adversarial detection methods focus on detecting and rejecting adversarial attacks (Carlini & Wagner, 2017; Ma et al., 2018; Yang et al., 2020; Tian et al., 2021). Certifiable defenses provide theoretical guarantees for the lower bound performance of networks subjected to worst-case adversarial attacks, however, they incur additional costs during inference and, empirically, they yield sub-optimal performance (Cohen et al., 2019; Wong &
Kolter, 2018; Raghunathan et al., 2018; Balunovic & Vechev, 2020). Adversarial training methods are based on a special form of data augmentation designed to make the network robust to worst-case perturbations (Zhang et al., 2019; Athalye et al., 2018; Kurakin et al., 2017). However, computing a worst-case perturbation is an NP-hard problem that needs to be solved at every iteration. To minimize the overhead of adversarial training, Goodfellow et al. (2015) proposed FGSM which requires one additional gradient step per iteration. Tramèr et al. (2018) first proposed performing a random step before taking the adversarial step (R+FGSM), but they observed that neither method yields robust models against PGD attacks (Madry et al., 2018). Since then, augmenting the training with PGD attacks has been one of the most popular approaches for robustness, but its cost increases linearly with the number of steps, which presents a severe practical limitation.
To reduce the cost of PGD, Shafahi et al. (2019) proposed Free Adversarial Training (Free-AT), that exploits a single back-propagation step to both update the network parameters and compute the attack. Wong et al. (2020) explored a variation of R+FGSM, namely RS-FGSM, and showed it can yield robust networks against multi-step attacks. Andriushchenko & Flammarion (2020) found that RS-FGSM only works for limited perturbation radii and introduced GradAlign – a regularizer to linearize the loss surface. However, optimizing GradAlign triplicates the computational cost. This motivated a new series of works that aim at matching the performance of GradAlign without the additional computational overhead (Golgooni et al., 2021; Kim et al., 2021). Other strategies that attempted to improve FGSM included introducing dropout in every layer (Vivek & Babu, 2020) and perturbing intermediate feature maps together with the input (Park & Lee, 2021). Li et al. (2020) suggested combining RS-FGSM and PGD attacks during training, however, the proposed strategy requires a frequent monitoring of the PGD robust accuracy and, in the worst-case, is computationally equivalent to PGD training.
Gilmer et al. (2019); Fawzi et al. (2018) suggested a strong link between robustness to adversarial attacks and to random noise. Motivated by this, we revisit the idea of combining noise and FGSM and propose N-FGSM. Our method is closely related to RS-FGSM, however, we find that using a larger amount of noise and removing the constraint that attacks must lie in the − l∞ ball is key to obtaining robust models. We note that Kang & Moosavi-Dezfooli (2021) concurrently studied RS-FGSM without clipping, however, as opposed to our work, they did not investigate and provide insights on the impact of noise, and the learned models were not robust against large perturbations.

heading: 3 PRELIMINARIES ON SINGLE-STEP ADVERSARIAL TRAINING
text: Given a classifier fθ : X → Y parameterized by θ and a perturbation set S, the classifier fθ is said to be robust at x ∈ X under S if the following holds for all δ ∈ S: fθ(x + δ) = fθ(x). One of the most popular definitions for S is the − `∞ ball, i.e. S = {δ : ‖δ‖∞ ≤ }. This is known as the l∞ threat model and is the setting we adopt throughout this work.
To train networks that are robust against `∞ threat models, adversarial training modifies the classical training procedure of minimizing a loss function over a dataset D = {(xi, yi)}i=1:N of images xi ∈ X and labels yi ∈ Y . In particular, adversarial training instead minimizes the worst-case loss over the perturbation set S, i.e. trains on adversarially perturbed samples {(xi+δi, yi)}i=1:N . When using the l∞ threat model, we can formalize adversarial training as solving the following problem:
min θ N∑ i=1 max δ L(fθ(xi + δ), yi), subject to ‖δ‖∞ ≤ , (1)
where L is typically the cross-entropy loss for image-classification models. Due to the difficulty of finding the exact inner maximizer, the most common procedure for adversarial training is to approximate the worst-case perturbation through several PGD steps (Madry et al., 2018). While this has been shown to yield robust models, it comes at a cost of a linear increase in the computational overhead with the number of PGD steps. As a result, several works have focused on reducing the cost of adversarial training by approximating the inner maximization with single-step attacks.
If we assume that the loss function is locally linear with respect to changes in the input, then the inner maximization of Equation (1) enjoys a closed form solution. Goodfellow et al. (2015) leveraged this result to propose the FGSM method, which takes one step in the direction of the sign of the gradient. Tramèr et al. (2018) proposed adding a random initialization prior to FGSM. However, both
methods were later shown to be vulnerable against multi-step attacks, such as PGD (Madry et al., 2018). Contrary to prior intuition, recent work from Wong et al. (2020) observed that combining a random step with FGSM can actually lead to a promising robustness performance. In particular, we note that most recent single-step methods approximate the worst-case perturbation that results from solving the inner maximization problem in Equation (1) with the following general form:
δ = ψ ( η + α · sign ( ∇xiL(fθ(xi + η), yi) )) , where η ∼ Ω (2)
and Ω is the distribution from which we draw noise perturbations. For example, when ψ is the projection operator onto the `∞ ball and Ω is the uniform distribution in [− , ], this recovers RSFGSM with the following update:
δRS-FGSM = Proj‖δ‖∞≤ ( η + α · sign ( ∇xiL(f(xi + η), yi) )) , where η ∼ U [− , ]d. (3)
On the other hand, with a different noise setting where Ω = ( −α) · sign (N (0, I)) and by choosing the step size α to be in [0, ] we recover R+FGSM by Tramèr et al. (2018) that initially explored the idea of combining noise with FGSM but reported no improvement over adversarial training with FGSM. If we consider Ω to be deterministically 0 and ψ to be the identity map, we recover the FGSM. Finally, if we adjust the choice of the loss function L to include a gradient alignment regularizer, this recovers the GradAlign algorithm by Andriushchenko & Flammarion (2020).

heading: 4 NOISE AND FGSM
text: A common practice when performing adversarial training is to restrict the perturbations used during training to the same − `∞ ball that will be considered at test time. The rationale behind it is that increasing the magnitude of perturbations could “unnecessarily” decrease the clean accuracy, since perturbations outside the ball will not be evaluated at test time. For instance, R+FGSM combines the noise step, with magnitude ( −α), and the FGSM step, with magnitude α, in a convex combination manner, thereof, restricting the perturbation to . On the other hand, Wong et al. (2020) apply a clipping operation after the FGSM step to the − `∞ ball. In the following, we experimentally challenge this common practice and explore the two key components in previous single-step methods that limit the magnitude of the perturbations. In particular, we explore (i) the role of the clipping operation, i.e. ψ as a projection to `∞ ball; and (ii) the source and magnitude of noise for the random step, i.e. Ω. We thoroughly revisit the role of both components on the robustness attained in singlestep methods. Throughout this work, unless stated otherwise, we consider noise perturbations η sampled from a symmetric Uniform distribution, i.e. Ω = U [−k, k]d, where d is the dimension of X and we refer to k as the “noise magnitude”.

heading: 4.1 THE ROLE OF NOISE AND CLIPPING ON ROBUSTNESS IN SINGLE-STEP METHODS
text: Clipping Reduces the Effectiveness of Single-Step Perturbations. To study the impact clipping has on model robustness, we consider the training of PreActResNet18 (He et al., 2016) on CIFAR-10 (Krizhevsky & Hinton, 2009) with an instance of Equation (2) as a single-step adversarial training. In particular, we consider the case where ψ is a projection to the `∞ ball of size clip ∈ { , 2 , 3 ,∞}, where ∞ denotes that ψ is an identity function, i.e. no clipping is performed. Moreover, we consider noise sampled from a symmetric uniform distribution where k ∈ { , 2 , 3 , 4 }. We report in Figure 2 the robust accuracy using PGD-50-10 (i.e. PGD attack with 50 iterations and 10 restarts) with = 8/255. We observe in Figure 2 (left), that for all choices of noise magnitude k, as we expand the clipping `∞ ball, i.e. we increase clip, the adversarial robustness improves. We believe that this is due to the fact that more aggressive clipping, i.e. smaller clip, reduces the strength of the computed single-step perturbations during training. To support this intuition, we report in Figure 2 (middle) the distribution of the loss measured at perturbed points prior to applying the clipping step and after applying the clipping step, with clip = . Moreover, the negative impact of clipping during training is more prominent as we increase the noise magnitude k. This is to be expected since, for a fixed α, increasing the noise magnitude k will lead to a prevalence of the noise component over the sign gradient direction in Equation (2).
Thus, overall, we observe that clipping during training has negative impact on the robust accuracy. Despite unclipped perturbations may lie outside the −`∞ ball, we could not observe any significant drop in clean accuracy. In fact, we obtain comparable clean accuracy to GradAlign as later reported in Section 5.5. Further analyses can be found in Appendix C.
The role of noise in single-step adversarial training. Neither Tramèr et al. (2018) nor Wong et al. (2020) explored settings with an increased noise magnitude. Moreover, Andriushchenko & Flammarion (2020) argue that noise in RS-FGSM is not important per se, claiming that its main purpose is only reducing the `2 norm of the final perturbation δRS-FGSM so that the loss is still in the linear regime. However, we empirically find that increasing the noise magnitude is key to avoiding catastrophic overfitting – see Figure 2 (right).
Investigating further, similarly to Kim et al. (2021), we plot the loss surface at the end of training (see Figure 12 in Appendix J) and find that, as observed by Kim et al. (2021), the loss surface of models trained via FGSM or RS-FGSM appears distorted at the end of training, i.e. the loss increases sharply along the FGSM direction, but then it rapidly decreases back to the same loss values associated with the clean sample. This is consistent with the gradient obfuscation observed by Tramèr et al. (2018). However, when training without clipping with an increased noise magnitude, we observe a non-distorted loss surface, i.e. the loss gradually increases along the FGSM direction. Interestingly, we observe a similar effect when training with the GradAlign regularizer. Thus, combining noise with FGSM seems to have a regularizing effect that encourages the loss surface to be locally linear in a similar spirit to GradAlign. Note this is based on the empirical analyses and visual inspections that we performed. We do not provide theoretical justification behind this.
Despite the clear benefits of increasing the noise magnitude, we do observe a slight but consistent decrease in robustness as we keep on increasing the noise, which we hypothesize is due to overregularization. We find k = 2 to be the sweet spot in most of our experiments; we do not exclude that a more extensive hyperparameter tuning procedure may lead to improved results.

heading: 4.2 OUR APPROACH
text: In the previous section, we provided experimental analyses to show that increasing the noise magnitude and not clipping prevents catastrophic overfitting and improves robustness significantly. Based on these observations, we propose the following, simple and efficient, single-step method for adversarial training that we denote as Noise-FGSM (N-FGSM):
δN-FGSM = η + α · sign ( ∇xiL(fθ(xi + η), yi) ) , η ∼ U [−k, k]d. (4)
We detail our full adversarial training procedure in Algorithm 1.
Theoretical insights We now theoretically analyse N-FGSM to understand the role of noise in single-step approaches. According to Andriushchenko & Flammarion (2020), the main contribution
Algorithm 1 N-FGSM adversarial training 1: Inputs: epochs T , batches M , radius , step-size α (default: ), noise magnitude k (default:2 ). 2: for t = 1, . . . , T do 3: for i = 1, . . . ,M do 4: // Perform N-FGSM adversarial attack 5: η ∼ Uniform[−k, k]d 6: δ = η + α · sign ( ∇xiL(fθ(xi + η), yi)
) 7: ∇θ = ∇θL(fθ(xi + δ), yi) 8: θ = optimizer(θ,∇θ) // standard weight update, (e.g. SGD) 9: end for
10: end for
of noise in single-step adversarial training is to reduce the effective `2 norm of the perturbation δ. Since N-FGSM does not involve clipping, we find the expected norm squared of N-FGSM perturbations to be larger than RS-FGSM perturbations. In fact, it is even larger than the FGSM perturbations which always lie on the − `∞ ball. These observations are formalized in Theorem 1. Theorem 1. Let δN-FGSM be our proposed single-step method defined by Equation (4), δFGSM be the FGSM method (Goodfellow et al., 2015) and δRS-FGSM be the RS-FGSM method (Wong et al., 2020). Then, with default hyperparameter values and for any > 0, we have that
Eη [ ‖δN-FGSM‖22 ] > Eη [ ‖δFGSM‖22 ] > Eη [ ‖δRS-FGSM‖22 ] .
We present the Proof in Appendix F where we also compute an empirical estimation of the expected `2 norm of different methods via Monte Carlo sampling and find them to align with Theorem 1. Differently from the hypothesis of Andriushchenko & Flammarion (2020), we find that despite that N-FGSM perturbations have larger `2 norms, they yield robust models even under larger radii, for which RS-FGSM or FGSM fail to catastrophic overfitting. We believe that these contradictory observations can lead to a better understanding on the role of noise in adversarial training and catastrophic overfitting in future work.

heading: 5 EXPERIMENTS AND ANALYSES
text: We compare N-FGSM against several adversarial training methods, considering a broad range of − l∞ radii. Following Wong et al. (2020); Andriushchenko & Flammarion (2020), we measure adversarial robustness on CIFAR-10/100 (Krizhevsky & Hinton, 2009) and SVHN (Netzer et al., 2011) datasets with PGD-50-10 attack – PGD (Madry et al., 2018) with 50 iterations and 10 restarts.

heading: 5.1 COMPARISON TO OTHER SINGLE-STEP METHODS
text: We start by comparing N-FGSM against other single-step methods. Note that not all single-step methods are equally expensive, since they may involve more or less computationally demanding operations. For instance, GradAlign (Andriushchenko & Flammarion, 2020) relies on a regularizer that is considerably expensive, while, MultiGrad (Golgooni et al., 2021) requires evaluating the input gradients on multiple random points. For a comparison of training cost over different singlestep methods, we refer to Figure 1 (right). Following the standard practice (Wong et al., 2020; Andriushchenko & Flammarion, 2020), we use a PreactResNet18 architecture (He et al., 2016).
We use RS-FGSM and Free-AT with the settings recommended by Wong et al. (2020). We apply GradAlign following the hyperparameters reported in the official repository 1. Golgooni et al. (2021) recommend applying MultiGrad with n = 3 random samples, but do not provide a default hyperparameter setting for what concerns the ZeroGrad variant. Also Kim et al. (2021) do not recommend a set of hyperparameters; for a fair comparison, we ablate them and select the ones that provide the highest adversarial accuracy (for every combination of and dataset). We train on CIFAR-10/100 for 30 epochs and on SVHN for 15 epochs with a cyclic learning rate. Only for Free-AT, we use 96 and 48 epochs for CIFAR-10/100 and SVHN, respectively, to obtain comparable results following
1https://github.com/tml-epfl/understanding-fast-adv-training/
Wong et al. (2020) and Andriushchenko & Flammarion (2020). CIFAR-10 results are presented in Figure 1 (middle), whereas CIFAR-100 and SVHN results are reported in Figure 3.
As previously observed, FGSM and RS-FGSM both suffer from catastrophic overfitting on larger attacks. In contrast, N-FGSM prevents catastrophic overfitting and enjoys robustness properties comparable or superior to GradAlign for all , while being 3 times faster. With appropriate hyperparameters, ZeroGrad is able to avoid catastrophic overfitting but obtains sub-obtimal robustness – especially for large perturbations. Neither MultiGrad nor the method proposed by Kim et al. (2021) can avoid catastrophic overfitting in all settings. We also observe that Free-AT cannot overcome catastrophic overfitting as also observed by Andriushchenko & Flammarion (2020).

heading: 5.2 RANDOMIZED ALPHA
text: Kim et al. (2021) evaluate intermediate points along the RS-FGSM direction in order to pick the “optimal” perturbation size. However, we find that increasing the number of intermediate evaluated points does not necessarily lead to increased adversarial accuracy. Moreover, for large perturbations we could not prevent catastrophic overfitting even with twice the number of evaluations tested by Kim et al. (2021). This motivates us to test a very simple baseline where instead of evaluating intermediate steps, the RS-FGSM perturbation size is randomly selected as: δ = t · δRS-FGSM where t ∼ U [0, 1]d. Interestingly, as reported in Figure 4 (left), we find that this very simple baseline, dubbed RandAlpha, is able to avoid catastrophic overfitting for all values of and outperforms Kim et al. (2021) on CIFAR-10. This is aligned with our main finding that combining noise with adversarial attacks is indeed a powerful tool that should be explored more thoroughly before developing more expensive solutions. We reach the same conclusions for CIFAR-100 and SVHN in Appendix H.

heading: 5.3 HYPERPARAMETER SELECTION
text: While FGSM relies on a fixed step-size (equal to the maximum radius of perturbation to be used at test time, i.e. α = ), Wong et al. (2020) explored different values for α during the development of RS-FGSM finding that an increase of the step-size improves the adversarial accuracy – up to a magnitude before catastrophic overfitting occurs. We also ablate the value of α for N-FGSM in Figure 4 (middle). We find that by increasing the noise magnitude, N-FGSM can use larger α values than RS-FGSM, without suffering from catastrophic overfitting. As observed by Wong et al. (2020), this in turn leads to an increase in the adversarial accuracy at the expense of a decrease in the clean accuracy. In light of this trade-off and following FGSM, we also use α = for N-FGSM.
Regarding N-FGSM noise hyperparameter, we find k = 2 works in all but one SVHN experiment ( = 12, in which we set k = 3 ), reducing the need for expensive hyperparameter tuning. In comparison, GradAlign regularizer hyperparameter or ZeroGrad quantile value need to be defined for every radius with a noticeable shift between CIFAR-10 and SVHN hyperparameters, suggesting they may require additional tuning when applied to novel datasets.

heading: 5.4 LONG VS FAST TRAINING SCHEDULE
text: Throughout our experiments, we used the RS-FGSM training setting introduced in Wong et al. (2020). However, Rice et al. (2020) suggest that a longer training schedule coupled with early stopping may lead to a boost in performance. Kim et al. (2021) and Li et al. (2020) report that longer training schedules increase the chances of catastrophic overfitting for RS-FGSM and that this limits its performance. We test the longer training schedule with N-FGSM and find that it presents robust overfitting, i.e. the adversarial accuracy on the training set keeps increasing but it decreases when evaluated on the test set as described in Rice et al. (2020). However, it does not suffer from catastrophic overfitting. In Figure 4 (right), we show the results for = 8. Although we do observe a slight increase in performance when using the long training schedule, we find the performance remarkably competitive when considering the fast one which seems to avoid robust overfitting (when relying on early stopping, the best models are usually found at the end). Thus, it might be preferable to consider the fast training schedule if computational cost is an important factor. In Appendix E, we report a robust accuracy of 47.86 ± 0.1 for GradAlign with the longer schedule compared to 48.48 ± 0.27 for N-FGSM. Note that, in order to prevent GradAlign from suffering from catastrophic overfitting for = 8, we had to increase the regularizer hyperparameter (compared to the fast schedule), while N-FGSM is able to prevent catastrophic overfitting with the same settings.

heading: 5.5 COMPARISON TO MULTI-STEP ATTACKS
text: In Section 5.1, we compared the performance of single-step methods and observed that N-FGSM is able to match or surpass the state-of-the-art method, i.e. GradAlign, while reducing the computational cost by a factor of 3×. In this section, we compare the performance of N-FGSM with multi-step attacks. We use PGD-2 with α = /2 and PGD-10 with α = 2 following Wong et al. (2020) and keep the same training settings as described in Section 5.1 (note that PGD-x denotes PGD attack with x iterations and no restarts).
In Figure 5, we observe that PGD-2 also presents catastrophic overfitting when we increase the perturbation radius , which is consistent with results reported by Andriushchenko & Flammarion (2020). On the other hand, despite all methods achieving comparable clean accuracy, there is a gap on adversarial accuracy between PGD-10 and single-step methods which grows with the perturbation size. This can be partially expected since the search space grows exponentially with and PGD can explore it more thoroughly as we increase the number of iterations. Nevertheless, computing a PGD-10 attack is 10× more expensive than computing an N-FGSM one. An important direction for future work should be addressing this gap and analyse, both theoretically and empirically, whether single-step methods can actually match the performance of their multi-step counterparts.

heading: 5.6 EXPERIMENTS WITH WIDERESNET28-10 ARCHITECTURE
text: We also compare the performance of all methods on WideResNet28-10 (Zagoruyko & Komodakis, 2016) architecture in Figure 7 and Figure 8 (Appendix B). As in the experiments with the PreActResNet18 architecture, N-FGSM obtains state-of-the-art PGD-50-10 accuracy among single-step methods. Nevertheless, as a general trend, we observe that catastrophic overfitting seems to be more difficult to prevent when using WideResNet. For instance, FGSM is able to consistently yield robust models up to = 6 for PreActResNet18 on CIFAR-10, however, for some runs the same radius can lead to catastrophic overfitting for WideResNet models. We hypothesize this is because it is more over-parametrized –WideResNet28-10 has 36.5 M parameters, whereas PreActResNet18 has 11.2M. Regarding GradAlign, we had to increase the regularizer hyperparameter (compared to the settings for PreActResNet18) in order to prevent catastrophic overfitting on CIFAR-100. Note that, to our surprise, we could not find a competitive hyperparameter setting for GradAlign on the SVHN dataset for ≥ 6. We tried both increasing the regularizer hyperparameter and decreasing the step size α, but some or all runs led to models close to a constant classifier for each setting. We do not claim that GradAlign will not work, but finding a good configuration might require further tuning.
On the other hand, as observed earlier, the default configuration for N-FGSM (α = , k = 2 ) works well in all settings except for = 16 on CIFAR-10 and = 10, 12 on SVHN. For CIFAR-10, we increase the noise magnitude to k = 4 . For SVHN, we find that decreasing α as we tried for GradAlign works better than increasing the noise. However, in both cases N-FGSM can obtain more than a trivial adversarial accuracy. Results are presented in Appendix B.

heading: 6 DISCUSSION
text: In this work, we explore the role of noise and clipping in single-step adversarial training. Contrary to previous intuitions, we show that increasing the noise magnitude and removing the −`∞ constraint allows improving adversarial robustness, while maintaining a competitive clean accuracy. These findings led us to propose N-FGSM, a simple and effective approach that can match or surpass the performance of GradAlign (Andriushchenko & Flammarion, 2020), while achieving a 3× speed-up. We perform an extensive comparison with other relevant single-step methods, observing that all of them achieve sub-optimal performance and most of them are not able to avoid catastrophic overfitting for larger attacks. Moreover, we also analyze recent single-step methods and – inspired by Kim et al. (2021) – observe that uniformly choosing a step-size in [0, α] avoids catastrophic overfitting for RS-FGSM, which reinforces our intuition that random noise is a powerful tool towards learning robust models. However, despite impressive improvements of single-step adversarial training methods, there is still a gap between single-step and multi-step methods such as PGD-10 as we increase the radius. Therefore, future work should put an emphasis on formally understanding the limitations of single-step adversarial training and explore how, if possible, this gap can be reduced.
Ethics Statement. The existence of adversarial examples poses a potential threat to the deployment of deep learning systems into the real world. Therefore, finding fast and effective methods to train models robust against this threat is of utmost importance. On the other hand, adversarial training methods are based on augmenting samples with adversarial perturbations, thus, they are partially based on building methods to attack neural networks. Research on adversarial attacks is naturally a sensitive path, since it can potentially be exploited for unethical purposes. However, the scope of our work is not designing stronger attacks; rather, the methods we propose are designed ad hoc to improve the adversarial robustness of learning systems, not to break other models’ defenses. Therefore, although we are aware that we are carrying out research within a sensitive topic, we are not particularly concerned that the research presented in this paper can lead to harmful applications – on the contrary, we believe that it can help deploying safer machine learning applications.
Reproducibility Statement. In this paper, we compare against several adversarial training methods. The general training and evaluation settings used are described at the beginning of Section 5. Moreover, different hyperparameters have been used for each method depending on the dataset and network. These are detailed in the manuscript, where corresponding results are discussed. For all the reported experiments, we employ different random seeds to ensure replicability of our results, and report performances indicating the standard deviation values. To facilitate further research on this topic, as stated in Section 1, we will release the code to reproduce all experiments. Regarding our theoretical results, we deferred the proofs to Appendix F.

heading: A ADDITIONAL PLOTS FOR PREACTRESNET18 EXPERIMENTS
text: In the main paper we compare N-FGSM with other single-step methods and multi-step methods separately and remove clean accuracies for better visualization. In this section we present the curves for all methods with both the clean and robust accuracy. The tendency in the three datasets is for N-FGSM PGD-50-10 accuracy to be slightly above that of GradAlign, while the opposite happens to the clean accuracy. We also observe that clean accuracy becomes significantly more noisy when catastrophic overfitting happens. Exact numbers for all the curves are in Appendix L.

heading: B EXPERIMENTS WITH WIDERESNET28-10 ARCHITECTURE
text: In this section we present the plots of our experiments with WideResNet28-10. We report the results in two figures. In Figure 7 we compare all single-step methods and we do not plot the clean accuracy for better visualization. In Figure 8 we plot all methods, including multi-step methods, and report the clean accuracy as well with dashed lines. Since we observed that our baseline, RandAlpha, outperformed Kim et al. (2021) in all settings for PreActResNet18, we only report RandAlpha for WideResNet. As mentioned in the main paper, we observe that catastrophic overfitting seems to be more difficult to prevent for WideResNet. In particular, for GradAlign we observed the regularizer hyperparameter settings proposed by Andriushchenko & Flammarion (2020) for CIFAR-10 (searched for a PreActResNet18) worked well. However, those parameters led to Catastrophic Overfitting for 6 ≤ ≤ 12 in CIFAR-100. Since = 14, 16 did not show Catastrophic Overfitting, we increased the GradAlign regularizer hyperparameter λ for CIFAR-100 so that each 6 ≤ ≤ 12 would have the default value corresponding to + 2, for instance, λ for = 6 would be the default λ in Andriushchenko & Flammarion (2020) for = 8.
For SVHN we observed that the default values for λ led to models close to a constant classifier for ≥ 6. We tried to increase the lambda for those values to 1.25λ but observed the same result. Since the model did not show typical catastrophic overfitting but rather it seemed as it was underfitting, we tried to reduce the step-size to α = 0.75 and also both decreasing α and increasing λ. When reducing the step size we obtain accuracies above those of a constant classifier for some radii, however, some or all seeds converge to a constant classifier for each setting, hence the large standard deviations. For N-FGSM, the default configuration of N-FGSM (α = , k = 2 ) works well in all settings except for = 16 on CIFAR-10 and = 10, 12 on SVHN. For CIFAR-10, we increase the noise magnitude to k = 4 . For SVHN we find that decreasing α as we tried for GradAlign works better than increasing the noise. We use α = 8 for both radii. Exact numbers for all the curves are in Appendix L

heading: C FURTHER ABLATION OF NOISE AND STEP SIZE IN N-FGSM
text: In Section 4.1, we observed that both removing clipping and increasing noise were necessary to avoid catastrophic overfitting. However, when doing that, we increase the magnitude of the perturbations. In this section, we study more closely the interplay between the step-size α and the noise level k to ensure that it is indeed the increase in noise level, rather than merely increasing the magnitude of perturbations, what helps stabilize adversarial training and avoid catastrophic overfitting.
We fix test = 8/255 for evaluation, and report the clean and robust accuracy of N-FGSM under different combinations of noise level k and step size α. Note that when the noise level k = 0 N-FGSM recovers plain FGSM, thus, as we increase α it is equivalent to using FGSM with an increased train. On the other hand, when α = 0, this is equivalent to training with only random noise augmentation. Based on the results reported in Table 1, we make the following observations:
1) Increasing the perturbation size is not enough to avoid catastrophic overfitting. For instance, as observed in the first column of Table 1, training with an increasing α without noise, i.e. (k = 0 which is equivalent to FGSM), leads to catastrophic overfitting despite the clear increase in perturbation size due to the increase of α.
2) Catastrophic overfitting leads to a vulnerable model against smaller perturbations too. For instance, looking at the experiments with α = 10 or α = 12 in the first column, we observe that the resultant models are vulnerable to adversarial attacks for test = 8/255 even though the perturbation radius used in training is larger than 8/255. This indicates that once a model catastrophically overfits to perturbations of a given radius, it can be vulnerable to smaller perturbations too.
3) Increasing the level of noise is necessary to stabilize training for larger perturbations. As we increase the step size α of the attack, we observe that we need to increase the ratio between noise k and step-size α in order to avoid catastrophic overfitting. This again further suggests that it is indeed increasing the perturbation size by increasing the noise level (and not simply increasing the perturbation budget) what mitigates catastrophic overfitting.
4) Training with noise perturbations has a much milder effect on the clean accuracy than adversarial training. Despite N-FGSM perturbations have a larger radius, it does not result in a significant drop of the clean accuracy, see Figure 5. Although it might seem counter-intuitive, we note that while N-FGSM has a larger perturbation size, this increase is not only merely in the adversary direction but also in the random noise direction. That is to say, while the perturbation size is larger for N-FGSM, this is not necessarily equivalent to augmenting with adversaries with an a similarly larger perturbation size of adversaries due to the bias in the noise. In Table 1, we observe how augmenting training samples with noise alone (when α = 0) has a much milder effect on the clean accuracy. In general, moving right on the table (increasing noise) is more forgiving on the clean accuracy than moving downwards (increasing FGSM step size). This is not so surprising considering that moving in random directions along the input space has a much lower impact on the loss than moving along the FGSM direction (see Figure 12) and that training with noise alone does not provide any significant robustness against larger attacks (for a more detailed ablation see Figure 11).

heading: D ABLATION OF BASELINES INCREASING THE TRAINING EPSILON
text: We have seen in Theorem 1 and Appendix F that the magnitude of N-FGSM perturbations will (on expectation) be larger than that of other baselines. Moreover, since N-FGSM does not use clipping to ensure perturbations are within the − `∞ ball, they could have `∞−norm of up to k+ α; which for the default hyperparameter values (α = , k = 2 ) add up to 3 . In this section, we study the performance of all single-step baselines as we increase the training epsilon.
In Table 2, we present the result of an experiment where we fix test = 8/255, then for each baseline we increase the train from 8/255 to 16/255 (while always evaluating the final model at 8/255). We use the same training hyperparameters as reported in Section 5.1. Results lead to three main observations:
1) Increasing train ends up hurting test robust accuracy. Even though in some methods we observe an increase in adversarial accuracy as we start to increase train, it ends up decreasing. Moreover, this increase in adversarial accuracy (see GradAlign, MultiGrad or N-FGSM) is at the expense of a decrase in clean accuracy.
2) Catastrophic overfitting leads to a vulnerable model to smaller perturbations as well. As observed in Appendix C, those models which suffer catastrophic overfitting become vulnerable to attacks with smaller perturbation radius than used during training. Here, we observe the same effect for various single-step baselines, which indicates this is likely a general trend.
3) Changing N-FGSM step-size α leads to similar result to changing train in other baselines. Interestingly, we observe that although the presence of noise will lead to perturbations much larger than those of other methods, it is the step-size in the direction of the FGSM attack that will have a larger impact on robust and clean accuracy. Thus, for larger perturbations the role of noise seems to be mainly to somehow mitigate catastrophic overfitting rather than strongly contributing to model robustness.
In light of the previous observations, we conclude that using a larger perturbation for N-FGSM does not lead to an unfair comparison to other baselines, on the contrary, the best values for these baselines (considering the trade-off between clean and robust accuracy) are when using the same radius of perturbations for training and test ( test = train). On the other hand, this result highlights the particularity of increasing the perturbation size with noise rather than following the adversarial
direction. Gaining a deeper understanding of the role of noise in avoiding catastrophic overfitting is a promising direction for future work.

heading: E LONGER TRAINING SCHEDULE
text: In our experiments, we have followed the “fast” training schedule introduced by Wong et al. (2020). However, Rice et al. (2020) suggest that a longer training schedule coupled with early stopping may lead to a boost in performance. We also use the long training schedule for N-FGSM and observe that it does not lead to catastrophic overfitting. In Table 3 we compare the performance of N-FGSM and GradAlign for the long training schedule. We observe that GradAlign does not seem to benefit from the long training schedule. On the other hand, although N-FGSM seems to obtain a slight increase in performance, the “fast” schedule provides comparable performance. It is worth mentioning that for GradAlign, the default regularizer hyperparameter for = 8/255 and CIFAR-10 (λ = 0.2) does not prevent catastrophic overfitting. We do a hyperparameter search and keep the value with the largest final robust accuracy (λ = 0.632).

heading: F MAGNITUDE OF N-FGSM PERTURBATIONS
text: Lemma 1 (Expected perturbation). Consider the N-FGSM perturbation as defined in Equation (4) δN-FGSM = η + α · sign (∇x`(f(x+ η), y)) , where η ∼ Ω.
Let the distribution Ω be the uniform distribution U ( [−k , k ]d ) and α > 0. Then,
Eη [ ‖δN-FGSM‖|22 ] = d
( k2 2
3 + α2 ) and
Eη [‖δN-FGSM‖|2] ≤ √ d ( k2 2
3 + α2 ) Proof. By Jensen’s inequality, we have
Eη [‖δN-FGSM‖2] ≤ √ Eη [‖δN-FGSM‖22]
Then let us consider the term Eη [ ‖δN-FGSM‖22 ] and use the shorthand∇(η)i = (∇x`(f(x+ η), y))i. Eη [ ‖δN-FGSM‖22 ] =Eη‖η + α · sign (∇x`(f(x+ η), y)) ‖22
=Eη [ d∑ i=1 (ηi + α · sign(∇(η)i))2 ]
= d∑ i=1 Eη [ (ηi + α · sign(∇(η)i))2 ] =
d∑ i=1 Eη [ (ηi + α · sign(∇(η)i))2 |sign(∇(η)i) = 1 ] Pη [sign(∇(η)i) = 1]
+ d∑ i=1 Eη [ (ηi + α · sign(∇(η)i))2 |sign(∇(η)i) = −1 ] Pη [sign(∇(η)i) = −1]
= d∑ i=1 1 2k ∫ k −k (ηi + α) 2 dηi · Pη [sign(∇(η)i) = 1]
+ 1
2k d∑ i=1 ∫ k −k (ηi − α)2 dηi · Pη [sign(∇(η)i) = −1]
= d∑ i=1 1 2k ∫ α+k α−k z2dz · Pη [sign(∇(η)i) = 1]
+ 1
2k d∑ i=1 ∫ −α+k −α−k z2dz · Pη [sign(∇(η)i) = −1]
= d∑ i=1 1 2k ∫ α+k α−k z2dz · Pη [sign(∇(η)i) = 1]
+ 1
2k d∑ i=1 ∫ α+k α−k z2dz · Pη [sign(∇(η)i) = −1]
= 1
2k ∫ α+k α−k z2dz d∑ i=1 (Pη [sign(∇(η)i) = 1] + Pη [sign(∇(η)i) = −1])
= d
6k
[ (α+ k )3 − (α− k )3 ] = dk2 2
3 + dα2
Therefore,
Eη [‖δN-FGSM‖|2] ≤ √ d ( k2 2
3 + α2
) .
We state again Theorem 1 and present the proof. Theorem 1. Let δN-FGSM be our proposed single-step method defined by Equation (4), δFGSM be the FGSM method (Goodfellow et al., 2015) and δRS-FGSM be the RS-FGSM method (Wong et al., 2020). Then, with default hyperparameter values and for any > 0, we have that
Eη [ ‖δN-FGSM‖22 ] > Eη [ ‖δFGSM‖22 ] > Eη [ ‖δRS-FGSM‖22 ] .
Proof. From Lemma 1 we have that
Eη [ ‖δN-FGSM‖|22 ] = d
( k2 2
3 + α2
) .
On the other hand, Andriushchenko & Flammarion (2020) showed that
Eη [ ‖δRS-FGSM‖22 ] = d ( − 1
6 α3 +
1 2 α2 + 1 3 2 ) .
Finally, we note that Eη [ ‖δFGSM‖22 ] = ‖δFGSM‖22 = d 2.
The default hyperparameters for N-FGSM are k = 2, α = and RS-FGSM uses α = 5 /4. With these hyperparameters and any > 0 we have
Eη [ ‖δN-FGSM‖|22 ] = 7
3 d 2 > Eη
[ ‖δFGSM‖|22 ] = d 2 > Eη [ ‖δRS-FGSM‖|22 ] = 101
128 d 2
In Lemma 1 we compute the expected value of the squared `2 norm of N-FGSM perturbations and by Jensen’s inequality we obtain an upper bound for the expected `2 norm of N-FGSM perturbations. However, obtaining the exact expected magnitude is more complex. To compliment our analytic results, we approximate the `2 norm of FGSM, RS-FGSM and N-FGSM via Monte Carlo sampling. Results are presented in Figure 9. We observe that the empirical estimations are very close to the analytical upper bounds and that indeed, N-FGSM has a magnitude significantly above that of FGSM or RS-FGSM.

heading: G N-FGSM WITH GAUSSIAN NOISE
text: In the main paper we have only explored noise sources coming from a Uniform distribution. Since we are measuring robustness against l∞− attacks, the Uniform distribution is a natural choice because the random perturbations will be bounded to the l∞ ball defined by the span of the distribution. However, for the sake of completeness, we also explore the performance of augmenting the samples from a Gaussian distribution where we choose its standard deviation to match that of the uniform distribution. In Table 4 we present a comparison of the clean (top) and PGD-50-10 (bottom) accuracy for different values of α and noise magnitude with = 8/255. Recall that by default we use Uniform distribution U [−k, k], therefore hyperparameter k sets the noise magnitude. Increasing the FGSM step size without increasing the amount of noise leads to catastrophic overfitting. Note results for k = 0.5 . More importantly, results are very similar when the two noise distributions share the same standard deviation. Thus, using Gaussian instead of Uniform noise does not seem to alter the results. Although this might be expected, we remark that the Gaussian is an unbounded noise distribution and the common practice in adversarial training is to always restrict the norm of the perturbations.

heading: H FURTHER RESULTS WITH RANDALPHA
text: In Section 5.2 we analyze the method presented by Kim et al. (2021) and suggest a baseline where, instead of evaluating intermediate points to determine the “optimal” step size, we simply choose it randomly. That is, we multiply the RS-FGSM perturbation by a random scalar sampled from a uniform distribution in [0, 1]. Interestingly we find that it outperforms Kim et al. (2021) without the additional cost of intermediate evaluations and without the need to perform hyperparameter selection to find the optimal number of intermediate evaluations.

heading: I TRAINING WITH NOISE AUGMENTED SAMPLES
text: Gilmer et al. (2019) and Fawzi et al. (2018) report a close link between robustness to adversarial attacks and robustness to random noise. Actually, Gilmer et al. (2019) report that training with noiseaugmented samples can improve adversarial accuracy and vice-versa. We note that N-FGSM can actually be seen as a combination of noise-augmentation and adversarial attacks. Here we perform an ablation where we train models with samples augmented with uniform noise U [−k, k] and then test the PGD-50-10 accuracy. We observe, that indeed random noise can increase the robustness to wort-case perturbations for small − l∞ balls. However, as we increase , noise augmentation is no longer very effective. However, with N-FGSM, we apply a weak attack to these noise-augmented samples and this seems to be enough to make them effective for adversarial training.
J VISUALIZATION OF THE LOSS SURFACE
In this section we present a visualization of the loss surface. We adapted the code from Kim et al. (2021) to analyse the shape of the loss surface at the end of training for different methods. Kim et al. (2021) reported that after adversarial training catastrophic overfitting, the loss surface would become non-linear. In particular, they found that the FGSM perturbation seems to be misguided by local maxima very close to the clean image that result in ineffective attacks. We note this was already reported by Tramèr et al. (2018) which proposed to perform a random step to escape those maxima. We argue that adding noise to the random step, when properly implemented, actually prevents those maxima to appear in the first place.

heading: K COMPARISON OF ADVERSARIAL TRAINING COST
text: In this section we describe how we compute the relative training cost for single-step methods shown in Figure 1 (right). We approximate the cost based on the number of forward/backward passes each method uses, disregarding the cost of other additional operations such as adding a random step for RS-FGSM or N-FGSM. We understand these operations have a negligible cost compared to a full forward or backward pass.
FGSM: FGSM is the cheapest of all methods since it only uses one forward/backward to compute the attack and an additional forward/backward to compute the weight update. Hence, Cost FGSM = 2 F/B.
RS-FGSM: As previously mentioned, we do not take into account the cost of random steps or clipping, hence we consider RS-FGSM to have the same cost as standard FGSM. Cost RS-FGSM = 2 F/B.
N-FGSM: Idem as before, cost of N-FGSM = 2 F/B.
ZeroGrad: For ZeroGrad they need to do an additional sorting operation to find the smallest gradient components. This could be potentially expensive, however, since the size of the input image is
several orders of magnitude smaller than that of the network, we also ignore this cost. Cost ZeroGrad = 2 F/B.
MultiGrad: MultiGrad computes 3 random steps and evaluates the gradient in all of them. Therefore, it needs to do 3 F/B to compute the attack and an additional one to update the weights. Cost MultiGrad = 4 F/B.
Kim et al. (2021): Kim et al. (2021) compute the RS-FGSM perturbation and evaluate the model on c points along this direction. Therefore, they will spend 1F/B on the RS-FGSM attack, c−1 F on the evaluations since the clean image has already been evaluated; and 1 F/B for the weight update. In our plot, we used c = 3 since it was the most chosen setting. Kim et al. (2021) assume the cost of a forward is similar to that of a backward pass, following this assumption, cost of Kim et al. (2021) is 1 F/B + 2 F + 1 F/B = 3 F/B
Free-AT: Shafahi et al. (2019) re-use the gradient from the previous backward pass to compute the FGSM perturbation of the current iteration. Hence, the cost of their training is only 1 F/B per iteration. However, Wong et al. (2020) observed they needed a longer training schedule to produce comparable results. Therefore, the total training cost per iteration (1 F/B) is scaled by 96 in the case of Free-AT, while it is only scaled by 30 for other methods. Relative cost Free = (96 · 1 F/B) / (30 · 2 F/B).
GradAlign: Finally, GradAlign uses FGSM with a regularizer. However, this regularizer needs to compute second-order derivatives via double backpropagation, which does not have the same cost
as regular backpropagation. Andriushchenko & Flammarion (2020) report that the cost of using GradAlign regularizer increased the cost of FGSM by 3.

heading: L DETAILED RESULTS FOR SECTION 5.1 AND SECTION 5.6
text: In this section we present the tables with the exact numbers used in plots comparing adversarial training methods. For each method and − l∞ radius, the top number is the clean accuracy while the bottom number is the PGD-50-10 accuracy. We separate single-step from multi-step methods with a double line.
PreActResNet18 – CIFAR-10 Dataset
= 2/255 = 4/255 = 6/255 = 8/255 = 10/255 = 12/255 = 14/255 = 16/255
N-FGSM 91.48 ± 0.17
79.43 ± 0.21
88.44 ± 0.09
67.09 ± 0.31
84.72 ± 0.04
56.62 ± 0.26
80.58 ± 0.22
48.12 ± 0.07
75.98 ± 0.1
41.56 ± 0.16
71.46 ± 0.14
36.43 ± 0.16
67.11 ± 0.37
32.11 ± 0.2
63.18 ± 0.49
27.67 ± 0.93
Grad Align 91.73 ± 0.04
79.16 ± 0.03
88.76 ± 0.0
67.13 ± 0.26
85.67 ± 0.02
56.27 ± 0.31
81.9 ± 0.22
48.14 ± 0.15
77.54 ± 0.06
40.75 ± 0.28
73.29 ± 0.23
34.51 ± 0.63
68.01 ± 0.32
30.36 ± 0.27
61.3 ± 0.15
26.64 ± 0.27
FGSM 91.6 ± 0.1
79.35 ± 0.06
88.77 ± 0.04
67.11 ± 0.09
85.58 ± 0.11
56.33 ± 0.41
86.41 ± 0.7
0.0 ± 0.0
82.08 ± 1.62
0.0 ± 0.0
80.6 ± 2.59
0.0 ± 0.0
76.04 ± 2.37
0.0 ± 0.0
77.14 ± 2.46
0.0 ± 0.0
RS-FGSM 92.09 ± 0.05
78.64 ± 0.08
89.69 ± 0.01
66.12 ± 0.22
87.0 ± 0.12
54.87 ± 0.22
84.05 ± 0.13
46.08 ± 0.18
85.21 ± 0.51
0.0 ± 0.0
65.22 ± 23.23
0.0 ± 0.0
43.59 ± 25.01
0.0 ± 0.0
76.66 ± 0.38
0.0 ± 0.0
Kim et. al. 92.85 ± 0.11
74.74 ± 0.35
91.1 ± 0.04
60.51 ± 0.4
89.34 ± 0.05
48.95 ± 0.45
89.02 ± 0.1
33.01 ± 0.09
88.27 ± 0.14
24.43 ± 0.84
88.35 ± 0.31
13.11 ± 0.63
90.01 ± 0.25
5.86 ± 0.57
90.45 ± 0.08
1.88 ± 0.05
AT Free 87.99 ± 0.16
74.27 ± 0.33
84.98 ± 0.13
62.47 ± 0.25
81.77 ± 0.11
53.18 ± 0.15
78.41 ± 0.18
46.03 ± 0.36
74.79 ± 0.22
39.87 ± 0.07
73.91 ± 4.19
22.99 ± 16.26
61.92 ± 14.94
0.0 ± 0.0
71.64 ± 3.89
0.0 ± 0.0
ZeroGrad 91.71 ± 0.08
79.36 ± 0.05
88.8 ± 0.11
67.32 ± 0.02
85.71 ± 0.1
56.14 ± 0.21
82.62 ± 0.05
47.08 ± 0.1
79.91 ± 0.12
37.58 ± 0.2
78.11 ± 0.2
27.41 ± 0.27
75.66 ± 0.46
21.29 ± 0.97
75.42 ± 0.13
13.06 ± 0.22
MultiGrad 91.57 ± 0.16
79.34 ± 0.02
88.74 ± 0.12
66.81 ± 0.02
85.75 ± 0.05
56.02 ± 0.3
82.33 ± 0.14
47.29 ± 0.07
78.73 ± 0.16
40.11 ± 0.24
75.28 ± 0.2
33.87 ± 0.17
80.94 ± 5.94
9.55 ± 13.5
71.42 ± 5.63
16.35 ± 11.57
PGD-2 91.4 ± 0.07
79.55 ± 0.15
88.46 ± 0.13
67.62 ± 0.03
85.14 ± 0.13
57.39 ± 0.13
81.41 ± 0.05
49.58 ± 0.08
77.18 ± 0.15
43.3 ± 0.11
72.9 ± 0.26
38.13 ± 0.15
70.39 ± 2.71
22.89 ± 15.26
64.81 ± 11.58
9.6 ± 13.37
PGD-10 91.25 ± 0.04
79.47 ± 0.13
88.34 ± 0.11
68.29 ± 0.24
84.79 ± 0.11
58.85 ± 0.18
80.71 ± 0.14
51.33 ± 0.31
76.13 ± 0.35
45.02 ± 0.49
71.24 ± 0.3
39.93 ± 0.5
66.7 ± 0.39
36.02 ± 0.67
62.11 ± 0.62
32.22 ± 0.64
PreActResNet18 – CIFAR-100 Dataset
= 2/255 = 4/255 = 6/255 = 8/255 = 10/255 = 12/255 = 14/255 = 16/255
N-FGSM 69.12 ± 0.27
51.02 ± 0.34
64.0 ± 0.06
39.5 ± 0.12
59.53 ± 0.02
32.06 ± 0.37
54.9 ± 0.2
26.46 ± 0.22
50.6 ± 0.16
22.23 ± 0.17
46.06 ± 0.14
18.95 ± 0.15
41.67 ± 0.25
16.33 ± 0.15
37.91 ± 0.11
14.34 ± 0.07
Grad Align 68.96 ± 0.15
51.31 ± 0.12
64.71 ± 0.16
39.37 ± 0.25
60.42 ± 0.23
31.91 ± 0.28
56.53 ± 0.31
25.8 ± 0.14
54.06 ± 0.44
18.7 ± 1.92
48.87 ± 0.32
17.86 ± 0.04
43.84 ± 0.14
15.51 ± 0.16
38.93 ± 0.21
13.62 ± 0.19
FGSM 69.01 ± 0.13
51.3 ± 0.19
64.47 ± 0.15
39.7 ± 0.16
63.85 ± 2.18
10.93 ± 14.64
53.42 ± 0.65
0.0 ± 0.0
45.06 ± 2.29
0.0 ± 0.0
46.14 ± 2.58
0.0 ± 0.0
41.66 ± 0.88
0.0 ± 0.0
44.68 ± 1.74
0.0 ± 0.0
RS-FGSM 69.83 ± 0.29
50.13 ± 0.32
65.9 ± 0.36
38.36 ± 0.19
62.15 ± 0.23
30.82 ± 0.08
55.26 ± 6.86
0.01 ± 0.01
32.33 ± 12.12
0.0 ± 0.0
36.07 ± 2.59
0.0 ± 0.0
21.52 ± 5.56
0.0 ± 0.0
20.38 ± 6.15
0.0 ± 0.0
Kim et. al. 72.92 ± 0.41
44.19 ± 0.25
70.16 ± 0.07
30.63 ± 0.28
67.98 ± 0.19
22.0 ± 0.02
68.07 ± 0.1
12.75 ± 0.21
68.37 ± 0.21
6.98 ± 0.23
74.09 ± 0.06
0.0 ± 0.0
74.06 ± 0.34
0.0 ± 0.0
74.01 ± 0.36
0.0 ± 0.0
AT Free 63.01 ± 0.19
45.7 ± 0.33
59.41 ± 0.27
35.95 ± 0.09
55.43 ± 0.37
29.37 ± 0.21
51.91 ± 0.08
24.32 ± 0.4
48.11 ± 0.09
20.64 ± 0.22
43.48 ± 1.25
5.71 ± 8.05
18.33 ± 4.86
0.0 ± 0.0
20.43 ± 11.25
0.0 ± 0.0
ZeroGrad 69.35 ± 0.36
51.1 ± 0.09
64.59 ± 0.32
39.38 ± 0.15
60.69 ± 0.09
31.72 ± 0.21
56.94 ± 0.13
25.87 ± 0.09
54.55 ± 0.17
19.49 ± 0.08
52.97 ± 0.34
14.32 ± 0.08
50.87 ± 0.26
10.92 ± 0.59
50.73 ± 0.3
7.3 ± 0.16
MultiGrad 69.01 ± 0.16
51.15 ± 0.03
64.44 ± 0.11
39.16 ± 0.03
60.65 ± 0.26
31.73 ± 0.09
56.84 ± 0.2
25.96 ± 0.11
53.62 ± 0.25
21.37 ± 0.16
53.05 ± 1.85
9.57 ± 7.32
48.28 ± 0.66
3.2 ± 4.49
45.28 ± 11.14
0.0 ± 0.0
PGD-2 69.18 ± 0.1
51.36 ± 0.03
64.32 ± 0.14
40.06 ± 0.14
60.21 ± 0.13
32.99 ± 0.24
55.8 ± 0.16
27.38 ± 0.16
51.68 ± 0.1
23.39 ± 0.19
48.2 ± 0.1
19.83 ± 0.29
46.14 ± 1.24
10.55 ± 7.51
37.97 ± 10.52
4.79 ± 6.75
PGD-10 68.83 ± 0.07
51.51 ± 0.27
63.87 ± 0.09
40.59 ± 0.36
59.37 ± 0.07
33.65 ± 0.02
54.79 ± 0.38
28.55 ± 0.27
50.53 ± 0.15
24.17 ± 0.12
46.05 ± 0.21
21.2 ± 0.12
41.76 ± 0.07
18.72 ± 0.06
37.81 ± 0.14
16.59 ± 0.16
PreActResNet18 – SVHN Dataset
= 2/255 = 4/255 = 6/255 = 8/255 = 10/255 = 12/255
N-FGSM 96.01 ± 0.04
86.44 ± 0.1
94.54 ± 0.15
72.53 ± 0.19
92.25 ± 0.33
58.42 ± 0.14
89.56 ± 0.49
45.63 ± 0.11
86.74 ± 0.86
33.96 ± 0.49
81.48 ± 1.64
26.13 ± 0.81
Grad Align 96.02 ± 0.05
86.43 ± 0.1
94.56 ± 0.21
72.12 ± 0.19
92.53 ± 0.24
57.34 ± 0.24
90.1 ± 0.34
43.85 ± 0.14
87.23 ± 0.75
32.87 ± 0.33
84.01 ± 0.46
23.62 ± 0.41
FGSM 96.04 ± 0.07
86.5 ± 0.05
95.67 ± 0.07
13.61 ± 5.83
93.73 ± 0.68
0.56 ± 0.72
91.74 ± 0.86
0.26 ± 0.36
90.76 ± 0.63
0.07 ± 0.1
87.17 ± 0.43
0.0 ± 0.0
RS-FGSM 96.18 ± 0.11
86.16 ± 0.14
95.09 ± 0.09
71.28 ± 0.4
95.11 ± 0.44
0.11 ± 0.08
94.46 ± 0.16
0.0 ± 0.0
93.88 ± 0.24
0.0 ± 0.0
92.74 ± 0.5
0.0 ± 0.0
Kim et. al. 96.35 ± 0.02
83.26 ± 0.24
95.25 ± 0.08
66.32 ± 0.63
94.83 ± 0.02
48.27 ± 0.52
94.88 ± 0.29
31.8 ± 1.1
96.61 ± 0.09
0.18 ± 0.21
96.61 ± 0.01
0.0 ± 0.0
AT Free 95.01 ± 0.09
84.55 ± 0.27
93.66 ± 0.12
71.61 ± 0.75
91.72 ± 0.29
59.31 ± 1.0
91.29 ± 4.07
0.01 ± 0.0
91.86 ± 3.66
0.0 ± 0.0
92.36 ± 1.0
0.0 ± 0.0
ZeroGrad 96.06 ± 0.03
86.43 ± 0.1
94.81 ± 0.16
71.59 ± 0.22
93.53 ± 0.26
51.72 ± 0.53
92.42 ± 1.29
35.93 ± 2.73
90.34 ± 0.32
21.34 ± 0.31
88.09 ± 0.4
14.14 ± 0.32
MultiGrad 96.01 ± 0.08
86.4 ± 0.08
94.71 ± 0.17
71.98 ± 0.26
95.75 ± 0.58
28.1 ± 18.85
94.86 ± 0.97
11.49 ± 16.19
94.7 ± 0.12
0.0 ± 0.0
94.48 ± 0.19
0.0 ± 0.0
PGD-2 96.03 ± 0.14
86.72 ± 0.06
94.66 ± 0.1
73.29 ± 0.29
93.77 ± 0.61
60.53 ± 0.73
94.63 ± 1.29
20.68 ± 18.56
84.09 ± 14.99
0.41 ± 0.29
94.16 ± 0.54
0.02 ± 0.03
PGD-10 95.92 ± 0.08
86.94 ± 0.14
94.37 ± 0.13
74.76 ± 0.19
92.46 ± 0.25
63.9 ± 0.48
89.67 ± 0.34
53.95 ± 0.55
85.75 ± 0.65
44.91 ± 0.45
80.08 ± 0.93
37.65 ± 0.53
WideResNet28-10 – CIFAR-10 Dataset
= 2/255 = 4/255 = 6/255 = 8/255 = 10/255 = 12/255 = 14/255 = 16/255
N-FGSM 92.51 ± 0.11
81.43 ± 0.3
89.65 ± 0.09
69.11 ± 0.24
85.8 ± 0.23
58.29 ± 0.14
81.59 ± 0.32
49.53 ± 0.25
76.92 ± 0.04
42.37 ± 0.36
72.13 ± 0.15
36.85 ± 0.2
67.82 ± 0.43
31.66 ± 0.6
56.73 ± 0.42
25.01 ± 0.23
Grad Align 92.59 ± 0.05
81.33 ± 0.4
89.95 ± 0.3
69.81 ± 0.47
86.98 ± 0.06
59.0 ± 0.13
83.19 ± 0.26
50.0 ± 0.05
79.35 ± 0.26
41.48 ± 0.51
73.79 ± 0.72
35.06 ± 0.74
66.38 ± 0.53
30.83 ± 0.39
57.75 ± 0.75
26.26 ± 0.13
FGSM 92.65 ± 0.17
81.38 ± 0.22
90.06 ± 0.18
69.59 ± 0.25
87.99 ± 1.3
38.69 ± 26.54
86.46 ± 0.45
0.0 ± 0.0
82.67 ± 1.78
0.0 ± 0.0
80.14 ± 1.2
0.0 ± 0.0
74.54 ± 4.01
0.0 ± 0.0
71.56 ± 3.78
0.0 ± 0.0
RS-FGSM 92.85 ± 0.1
80.9 ± 0.13
90.73 ± 0.2
68.23 ± 0.17
88.24 ± 0.19
57.21 ± 0.17
83.64 ± 1.74
0.0 ± 0.0
82.1 ± 1.45
0.0 ± 0.0
78.62 ± 0.7
0.0 ± 0.0
73.25 ± 8.16
0.0 ± 0.0
68.64 ± 4.3
0.0 ± 0.0
RandAlpha 93.37 ± 0.22
77.67 ± 0.66
92.17 ± 0.21
63.73 ± 0.31
90.71 ± 0.14
50.4 ± 0.14
89.16 ± 0.19
39.37 ± 0.42
87.44 ± 0.31
30.13 ± 0.9
85.69 ± 0.28
23.13 ± 0.33
83.98 ± 0.24
16.0 ± 0.22
83.23 ± 0.46
8.47 ± 0.66
AT Free 90.66 ± 0.25
77.0 ± 0.27
88.37 ± 0.15
64.25 ± 0.33
86.11 ± 0.29
53.76 ± 0.48
83.5 ± 0.27
44.85 ± 0.39
80.52 ± 0.32
31.87 ± 5.53
83.59 ± 1.35
0.0 ± 0.0
39.58 ± 15.8
0.0 ± 0.0
42.59 ± 27.96
0.0 ± 0.0
ZeroGrad 92.62 ± 0.11
81.42 ± 0.28
90.17 ± 0.05
69.28 ± 0.29
86.98 ± 0.28
58.4 ± 0.14
84.25 ± 0.28
48.29 ± 0.16
81.72 ± 0.29
36.08 ± 0.29
79.24 ± 0.82
28.24 ± 1.79
78.14 ± 0.46
18.54 ± 0.31
75.34 ± 0.12
14.6 ± 0.12
MultiGrad 92.64 ± 0.1
81.19 ± 0.28
90.18 ± 0.13
69.3 ± 0.2
87.11 ± 0.36
57.98 ± 0.08
83.87 ± 0.46
48.74 ± 0.09
80.89 ± 0.14
41.22 ± 0.57
82.88 ± 2.85
4.46 ± 6.09
86.6 ± 1.52
0.0 ± 0.0
85.46 ± 3.73
0.0 ± 0.0
PGD-2 92.69 ± 0.14
81.54 ± 0.18
90.18 ± 0.19
69.87 ± 0.26
86.87 ± 0.18
59.4 ± 0.19
83.31 ± 0.16
50.88 ± 0.16
79.61 ± 0.47
43.94 ± 0.24
75.81 ± 0.24
37.77 ± 0.57
71.41 ± 1.38
21.06 ± 13.39
67.2 ± 14.94
0.0 ± 0.0
PGD-10 92.24 ± 0.31
81.18 ± 0.57
89.65 ± 0.33
70.34 ± 0.26
86.91 ± 0.51
60.59 ± 0.21
82.82 ± 0.7
52.58 ± 0.2
78.63 ± 0.66
45.92 ± 0.38
74.0 ± 0.67
40.44 ± 0.17
68.6 ± 0.58
35.98 ± 0.56
64.17 ± 0.72
32.5 ± 0.61
WideResNet28-10 – CIFAR-100 Dataset
= 2/255 = 5/255 = 6/255 = 8/255 = 10/255 = 12/255 = 14/255 = 16/255
N-FGSM 71.56 ± 0.13
52.23 ± 0.33
66.49 ± 0.46
39.93 ± 0.37
61.38 ± 0.68
30.97 ± 0.21
56.23 ± 0.59
26.77 ± 0.65
51.54 ± 0.63
23.03 ± 0.54
46.43 ± 0.61
19.3 ± 0.59
42.11 ± 0.32
16.67 ± 0.4
38.34 ± 0.47
14.27 ± 0.33
Grad Align 71.68 ± 0.33
51.5 ± 0.45
67.09 ± 0.19
39.9 ± 0.42
62.86 ± 0.1
32.0 ± 0.22
58.55 ± 0.41
26.9 ± 0.62
53.85 ± 0.73
22.63 ± 0.62
46.94 ± 0.86
19.9 ± 0.65
42.63 ± 0.5
16.93 ± 0.12
36.17 ± 0.45
14.03 ± 0.24
FGSM 71.92 ± 0.33
52.83 ± 0.37
67.34 ± 0.36
39.83 ± 0.31
64.72 ± 1.12
0.0 ± 0.0
56.87 ± 1.24
0.03 ± 0.05
52.31 ± 2.11
0.0 ± 0.0
48.99 ± 1.17
0.0 ± 0.0
44.27 ± 1.4
0.0 ± 0.0
42.05 ± 1.03
0.0 ± 0.0
RS-FGSM 72.65 ± 0.28
51.63 ± 0.52
68.26 ± 0.2
39.57 ± 0.09
65.58 ± 0.69
26.63 ± 2.8
54.25 ± 5.85
0.0 ± 0.0
46.08 ± 4.87
0.0 ± 0.0
35.84 ± 0.17
0.0 ± 0.0
24.4 ± 1.25
0.0 ± 0.0
21.37 ± 5.04
0.0 ± 0.0
RandAlpha 73.9 ± 0.15
49.13 ± 0.91
71.17 ± 0.12
34.3 ± 0.54
68.65 ± 0.22
25.5 ± 0.33
66.42 ± 0.13
20.27 ± 0.98
64.05 ± 0.5
16.3 ± 0.14
61.99 ± 0.6
12.4 ± 0.29
59.74 ± 0.57
6.93 ± 0.19
58.9 ± 0.78
3.63 ± 0.12
AT Free 67.62 ± 0.24
48.07 ± 0.31
63.27 ± 0.72
37.93 ± 0.69
59.53 ± 0.31
29.7 ± 0.51
55.77 ± 0.28
24.43 ± 0.37
47.02 ± 3.83
3.23 ± 4.43
33.52 ± 9.24
0.0 ± 0.0
7.87 ± 1.78
0.0 ± 0.0
20.92 ± 21.48
0.0 ± 0.0
ZeroGrad 71.68 ± 0.07
52.63 ± 0.61
67.2 ± 0.14
39.57 ± 0.33
63.69 ± 0.14
30.27 ± 0.54
60.77 ± 0.26
23.7 ± 0.08
61.05 ± 0.38
15.1 ± 0.49
58.39 ± 0.16
11.13 ± 0.68
56.19 ± 0.11
8.8 ± 0.36
56.38 ± 0.18
4.9 ± 0.36
MultiGrad 71.8 ± 0.15
51.9 ± 0.29
67.73 ± 0.48
39.7 ± 0.37
63.24 ± 0.33
31.5 ± 0.62
60.05 ± 0.79
26.03 ± 0.09
56.39 ± 0.49
20.8 ± 0.29
56.79 ± 8.27
0.0 ± 0.0
59.8 ± 3.77
0.0 ± 0.0
52.96 ± 5.58
0.0 ± 0.0
PGD-2 71.62 ± 0.15
51.73 ± 0.48
67.25 ± 0.43
40.27 ± 0.7
63.18 ± 0.36
32.23 ± 0.19
59.02 ± 0.4
27.13 ± 0.37
54.47 ± 0.45
23.43 ± 0.31
50.91 ± 0.35
20.23 ± 0.39
41.03 ± 3.18
0.03 ± 0.05
40.13 ± 3.66
0.0 ± 0.0
PGD-10 71.11 ± 0.62
52.5 ± 0.59
66.9 ± 0.57
40.73 ± 0.56
62.05 ± 0.47
32.8 ± 0.29
57.64 ± 0.81
27.97 ± 0.59
52.84 ± 0.88
24.7 ± 0.36
48.14 ± 0.73
21.8 ± 0.57
43.14 ± 0.87
18.87 ± 0.6
39.2 ± 0.62
16.8 ± 0.57
WideResNet28-10 – SVHN Dataset
= 2/255 = 4/255 = 6/255 = 8/255 = 10/255 = 12/255
N-FGSM 95.64 ± 0.09
84.1 ± 0.73
93.66 ± 0.41
66.9 ± 0.86
91.77 ± 0.42
53.0 ± 0.36
88.89 ± 0.58
40.5 ± 0.37
88.07 ± 0.59
30.47 ± 0.76
87.52 ± 0.49
22.43 ± 0.53
Grad Align 95.41 ± 0.06
84.57 ± 0.56
93.9 ± 0.48
67.27 ± 0.54
68.36 ± 34.49
39.53 ± 14.89
42.62 ± 32.73
24.7 ± 9.34
19.3 ± 0.21
17.63 ± 0.62
19.53 ± 0.08
18.13 ± 0.52
FGSM 95.83 ± 0.1
85.03 ± 0.37
95.0 ± 0.24
31.53 ± 6.57
94.23 ± 0.79
1.7 ± 1.36
91.11 ± 1.36
0.13 ± 0.19
88.83 ± 1.71
0.0 ± 0.0
86.74 ± 0.7
0.0 ± 0.0
RS-FGSM 95.81 ± 0.25
83.8 ± 0.43
94.53 ± 0.4
66.67 ± 0.65
95.23 ± 0.26
0.53 ± 0.26
94.68 ± 0.62
0.0 ± 0.0
93.9 ± 0.52
0.0 ± 0.0
91.64 ± 2.98
0.0 ± 0.0
RandAlpha 96.02 ± 0.23
82.5 ± 0.45
95.47 ± 0.18
63.33 ± 0.53
94.69 ± 0.26
47.7 ± 0.99
93.72 ± 0.44
35.73 ± 0.34
93.08 ± 1.45
23.17 ± 1.97
93.96 ± 0.68
11.1 ± 3.05
AT Free 94.85 ± 0.39
83.13 ± 0.17
92.95 ± 0.65
68.67 ± 0.53
91.62 ± 1.93
54.93 ± 2.58
93.74 ± 0.69
0.03 ± 0.05
92.47 ± 0.97
0.0 ± 0.0
90.5 ± 1.41
0.0 ± 0.0
ZeroGrad 95.78 ± 0.21
84.47 ± 0.83
94.06 ± 0.52
66.1 ± 0.37
92.13 ± 0.98
47.3 ± 0.62
91.04 ± 0.4
29.33 ± 0.56
88.85 ± 0.92
20.77 ± 0.63
89.8 ± 1.36
9.33 ± 0.76
MultiGrad 95.63 ± 0.16
84.37 ± 0.59
94.27 ± 0.38
67.27 ± 0.31
93.64 ± 1.21
50.1 ± 0.9
94.83 ± 1.55
1.77 ± 1.72
95.26 ± 0.34
0.0 ± 0.0
95.22 ± 0.15
0.0 ± 0.0
PGD-2 95.88 ± 0.35
86.25 ± 0.7
94.66 ± 0.1
73.29 ± 0.25
93.77 ± 0.61
60.53 ± 0.72
92.99 ± 1.11
40.77 ± 4.39
88.81 ± 0.93
34.33 ± 2.76
83.17 ± 4.78
26.8 ± 3.31
PGD-10 95.92 ± 0.08
86.94 ± 0.13
94.36 ± 0.13
74.46 ± 0.54
92.46 ± 0.25
63.87 ± 0.49
89.67 ± 0.34
53.95 ± 0.55
85.98 ± 0.59
44.59 ± 0.14
80.08 ± 0.93
37.64 ± 0.49

Review summaries:
This paper methodically studied the catastrophic overfitting in fast adversarial training (Fast-AT), and revisited the role of noise and clipping operation in Fast-AT. Based on the empirical findings, this paper discovered that the absence of clipping as well as using stronger noise could help avoid catastrophic overfitting. The author further proposed Noise-FGSM, utilizing single-step FGSM and noise-augmented samples to generate adversarial examples for training. Empirical studies showed the superiority of N-FGSM both in terms of performance and speed. 

This paper aims to address a failure mode in the traditional single-step adversarial training known as catastrophic overfitting. They show that compared to the common practice of generating the adversarial perturbation, adopting larger random initialization and avoiding clipping the perturbation can effectively mitigate catastrophic overfitting. The effects of these two techniques are analyzed empirically in detail, followed by a comprehensive comparison to other methods.