Eliminating Catastrophic Overfitting Via Abnormal Adversarial Examples Regularization
Authors: Runqi Lin
Chaojian Yu
Tongliang Liu
Venue: Submitted to ICLR 2023 (2023)

Abstract: Single-step adversarial training (SSAT) is shown to be able to defend against iterative-step adversarial attacks to achieve both efficiency and robustness. However, SSAT suffers from catastrophic overfitting (CO) with strong adversaries, showing that the classifier decision boundaries are highly distorted and robust accuracy against iterative-step adversarial attacks suddenly drops from peak to nearly 0% in a few epochs. In this work, we find that some adversarial examples generated on the network trained by SSAT exhibit anomalous behaviour, that is, although the training data is generated by the inner maximization process, the loss of some adversarial examples decreases instead, which we called abnormal adversarial examples. Furthermore, network optimization on these abnormal adversarial examples will further accelerate the model decision boundaries distortion, and correspondingly, the number of abnormal adversarial examples will sharply increase with CO. These observations motivate us to prevent CO by hindering the generation of abnormal adversarial examples. Specifically, we design a novel method, Abnormal Adversarial Examples Regularization (AAER), which explicitly regularizes the number and logits variation of abnormal adversarial examples to hinder the model from generating abnormal adversarial examples. Extensive experiments demonstrate that our method can prevent CO and further boost adversarial robustness with strong adversaries.

Under review as a conference paper at ICLR 2023
ELIMINATING CATASTROPHIC OVERFITTING VIA AB- NORMAL ADVERSARIAL EXAMPLES REGULARIZA- TION
Anonymous authors Paper under double-bling review
ABSTRACT
Single-step adversarial training (SSAT) is shown to be able to defend against iterative-step adversarial attacks to achieve both efficiency and robustness. How- ever, SSAT suffers from catastrophic overfitting (CO) with strong adversaries, showing that the classifier decision boundaries are highly distorted and robust accuracy against iterative-step adversarial attacks suddenly drops from peak nearly 0% in a few epochs. In this work, we find that some adversarial examples generated on the network trained by SSAT exhibit anomalous behaviour, that is, although the training data is generated by the inner maximization process, the loss of some adversarial examples decreases instead, which we called abnormal ad- versarial examples. Furthermore, network optimization on these abnormal adver- sarial examples will further accelerate the model decision boundaries distortion, and correspondingly, the number of abnormal adversarial examples will sharply increase with CO. These observations motivate us eliminate CO by hinder- ing the generation of abnormal adversarial examples. Specifically, we design a novel method, Abnormal Adversarial Examples Regularization (AAER), which explicitly regularizes the number and outputs variation of abnormal adversarial examples to hinder the model from generating abnormal adversarial examples. Extensive experiments demonstrate that our method eliminate CO and further boost adversarial robustness with strong adversaries.
INTRODUCTION
In recent years, Deep Neural Networks (DNNs) have performed impressively in various fields, such as autonomous driving (Litman 2017). face recognition (Sharif al.) 2016) and medical imaging diagnosis (Buch et al. 2018). However, DNNs were found to be vulnerable to adversarial examples (Szegedy et al.] 2013). Although these adversarial examples are imperceptible to the human eyes, they can lead to completely different prediction in DNNs. To this end, many adversarial defense methods have been proposed, such as verification and provable defense (Katz et al.] 2017), pre- processing techniques (Guo et al. 2017). detection algorithms (Metzen et al. [217]) and adversarial training (AT) (Goodfellow et al. 2014). Among them, AT is considered to be one of the most effective methods against adversarial attacks (Athalye al. 2018). However, standard iterative- step AT significantly increases computational overhead due to multiple steps forward and backward propagation.
Therefore, some works attempt to improve the vanilla single-step adversarial training (SSAT) to defend against iterative-step adversarial attacks while maintaining efficiency and robustness. Unfor- tunately, a serious problem catastrophic overfitting (CO) occurs with stronger adversaries (Wong al. 2020). This strange phenomenon means that the robust accuracy of the model against the iterative-step adversarial attack suddenly from peak drops to nearly zero during a few epochs, as shown in Figure [] This intriguing phenomenon has been widely investigated and led to many works to resolve CO. Recently, Kim et al. (2021) points out that networks in which CO occurs are generally accompanied by highly distorted decision boundaries. However, the interaction between distorted decision boundaries and CO remains unclear. In this work, we delve into the dynamic effects between CO and decision boundaries distortion. Specifically, we find some adversarial ex- amples generated on the network with distorted decision boundaries exhibit anomalous behavior,

---

Under review as a conference paper at ICLR 2023
Single-step Adversarial Attack
Iterative-step Adversarial Attack
80%
Robustness
60%
40%
20%
0%
5
10
15 Epoch
20
25
30
Figure 1: Model robust test accuracy with different noise magnitudes. The red and green lines are defence against FGSM and PGD-7-1 adversarial attack, respectively. The dashed line and solid line noise magnitude are 8/255 and 16/255, respectively. Dashed black lines correspond to the 10th epoch, which the point that model occurs CO.
that is, although all training samples are generated by the inner maximization process, the loss of some adversarial examples decreases instead. We refer to these training samples abnormal adver- sarial examples. To make matters worse, the decision boundaries distortion will further exacerbate by optimising the classifier directly these abnormal adversarial examples, and the number of abnormal adversarial examples will sharply increase as a result, which leads to a vicious circle be- tween the number of abnormal adversarial example and the decision boundaries distortion. All these atypical findings raise a question:
Can CO be prevented by hindering the generation abnormal adversarial examples?
To answer the above question, we design a novel method, Abnormal Adversarial Examples Reg- ularization (AAER), which incorporates regularizer that prevents CO by suppressing generated abnormal adversarial examples. Specifically, AAER consists of two key components: (i) the num- ber and (ii) outputs variation of abnormal adversarial examples. The first part (i) counts the sample number by dividing the training samples into groups of normal and abnormal adversarial examples through anomalous loss decrease behavior. The second part (ii) contains prediction confidence and logits variation, and calculates these two variations differences between the two groups of samples by cross-entropy and Euclidean distance, respectively. Then, AAER explicitly regularizes the num- ber and outputs variation of abnormal adversarial examples by these two parts to hinder the model from generating abnormal adversarial examples. Extensive experiments show that our method can well eliminate CO and further improve the adversarial robustness. It is worth noting that our method does not involve the extra generation and backward propagation process, which brings us unparal- leled convenience in computational overhead. Our major contributions are summarized as follows:
We found some training samples exhibit anomalous loss variation during the inner maximization process. Besides, the number of abnormal adversarial examples will sharply increase with CO, and the model will further exacerbate by optimising the classifier directly on these abnormal adversarial examples.
Based on the observed effect, we propose a novel method Abnormal Adversarial Examples Reg- ularization (AAER), which explicitly regularizes the number of abnormal adversarial examples and their anomalous outputs variation to hinder the generation of abnormal adversarial examples. Extensive experiments demonstrate that our method can prevent CO and automatically adapt to different noise magnitudes without hyperparameter tuning.
We evaluate the effectiveness our method across different adversarial budgets, adversarial attacks, datasets and network architectures, showing that proposed method consistently achieves -tate-of-the-art robust accuracy in SSAT and can obtain comparable robustness to stan- dard sterative-step AT with only negligible computational overhead.
2

---

Under review as a conference paper at ICLR 2023
2
RELATED WORK
2.1 ADVERSARIAL TRAINING
Adversarial training has been demonstrated be the most effective method for defending against adversarial attacks (Athalye et al. 2018). AT is generally formulated as min-max optimization problem (Madry et al. (017)) the inner maximization problem tries to generate the strongest ad- versarial examples to maximise the loss, and the outer minimization problem tries to optimize the network to minimize the loss on adversarial examples. However, the inner maximization problem a NP-hard problem. Therefore, AT uses a simple gradient ascent generate perturbations to find local approximate solution, and can formalized as a min-max optimization problem as follows:
min L (D
max E(x + y2: BEA
where (z., y) is the training dataset from the distribution D. ((x, y; 0) the loss function parameter- ized by 8, 6 is the perturbation confined within the boundary with Ly- norm distance, shown as: A (6 ||6||p S <). The common threat models are I,,, L2 and Troe this work we chose Tooe for our threat model.
Fast Gradient Sign Method (FGSM) (Goodfellow et al. 2014 single-step adversarial attack method, which uses the sign of the gradient find the perturbation, shown in Eq. (
8FGSM
sign (V (1, y; 0)))
(2)
Fast Training (RS-FGSM) (Wong et al., (220) adds uniform random initialization r) before gener- ating the perturbation, and uses the over-perturbation step size 1.25
7 Uniform(-Ce cc,
8RS-PGSM = & sign (Vztn + 9/ (,39)) 0))
Iterative Fast Gradient Sign Method (I-FGSM) (Kurakin et al. 2018) is an iterative-step version of FGSM that uses multiple gradients to find stronger perturbations. With smaller step size a c/N and the number of iterations T, I-FGSM can be formulated as follows:
FGFSM = a sign (V V2+57-1((2 + 8"' ,):)))
(4)
Projected Gradient Descent (PGD) Madry et al. 2017) adds uniform random initialization on the basis of I-FGSM.
2.2 CATASTROPHIC OVERHITTING
Since Wong et al. 2020) found the CO phenomenon, there has been line of work trying to explore and mitigate this problem. Vivek & Babu (020b)) empirically showed that adding a dropout layer after all non-linear layers avoid early overfitting to FGSM. de Jorge al. (2022) found that augmenting the perturbations by increasing the noise initialization magnitude and removing the perturbation boundaries can eliminate CO. C...Liet al. (2022) successfully prevents CO by constraining training samples in carefully extracted subspace to avoid the abrupt growth of gradient.
Other works attempt to prevent CO by strengthening the inner maximization processes, Kim et al. (222)) assumed that CO is caused by fixed FGSM perturbation magnitude and reduces the perturbed step size for misclassified adversarial examples. Golgooni et al. (202T) argued that small gradients play key role in CO and ignore small gradient information to avoid huge weight updates. Huang al. (022) discovered that fitting instances with a larger gradient norm are more likely to cause CO and learning an instance-adaptive step size is inversely proportional to its gradient norm. Park Lee (202T) leverages the gradients of latent representation as the latent adversarial perturbation to compensate for local linearity.
Similar to our work, some works add regularization term on the loss value to explicitly prevent CO. Andriushchenko & Flammarion (200 found that PGD and FGSM perturbations are orthogo- nal when CO occurs, hence they proposed regularization term to encourage the gradient alignment. Vivek & Babu (2020a) proposed regularization term to mitigate the CO by harnessing properties that differentiate a robust model from that of a pseudo-robust model. Sriramanan et al. (2021) intro- duced relaxation term find more suitable gradient directions for the attack by smoothing the loss
3

---

Under review as a conference paper at ICLR 2023
Class A
Class B
Normal adversarial example
Abnormal adversarial example
Decision boundary
Adversary direction
Normal decision boundary Distorted decision boundary
Figure 2: Visualization of classifier decision boundary and training samples. The left panel shows that the training samples generated on the normal decision boundary are all belong to the normal adversarial example (blue) which can mislead the classifier. The middle panel shows that some training samples generate on the distorted decision boundary that cannot mislead the classifier, which we called abnormal adversarial example (red).
surface. Chen et al. (2021) demonstrated that the negative high-order terms lead to a perturbation loss distortion phenomenon that will cause CO, and they proposed a regularization term to make the loss surface flat.
3 PROPOSED APPROACH
In this section, we first define abnormal adversarial example and show how their number change dur- ing CO (Section 3.1] We further analyse the outputs variation normal and abnormal adversarial examples and find that they exhibit significantly different magnitudes of outputs variation after CO (Section 3.2). Based on our observations, we propose a novel regularization term, AAER, using the number and outputs variation of abnormal adversarial examples to explicitly suppress the generation of these anomalous training samples to eliminate CO (Section 3.3
3.1 DEFINITION AND COUNTING THE ABNORMAL ADVERSARIAL EXAMPLE
Adversarial training employs the most adversarial data to reduce the sensitivity of the network's out- put w.r.t. adversarial perturbation of the natural data. Therefore, expect the inner maximization process can generate effective adversarial examples that can maximize the classification loss. How- ever, Kim et al.] (202T) shows that the decision boundaries of the classifier will be highly distorted accompanied by the occurrence of CO. After adding the adversarial perturbation which is generated on this distorted classifier, the classification loss of some training samples is atypically reduced. As shown in Figure [ it can be seen that, for some samples (blue), they will misclassify the model or be closer to the decision boundary after the inner maximization process, and for some other sam- ples (red), they are farther to the decision boundary after adding the perturbation generated by the distorted classifier, which we called abnormal adversarial example. These abnormal adversarial ex- amples generally fail to mislead the classifier. Thus, we can define abnormal adversarial examples using the following formula: = sign (V V 11,3; 0)),
(5)
8 Abnormal def (x E (x+ 8.y00)
We further observe the changes in the number of abnormal adversarial examples during model train- ing, and the statistical results are shown F can be observed that before CO occurs, the number of abnormal adversarial examples is very small, almost close to 0. During the occurrence of CO, their number increases sharply. For example, the number of abnormal adversarial examples surged 39 times (red line) at the 10th epoch. Note that the rapid increase in the number of abnormal adversarial examples also implies that the classifier boundaries are continuously deteriorating, which also leads further rise in the number of abnormal adversarial examples and peaks the 13th epoch, which is approximately 55 times the number of abnormal adversarial examples before CO occurred. After the occurrence of CO, the abnormal adversarial examples are basically maintained
4

---

Under review as a conference paper at ICLR 2023
50300
Aunatul Abrtran Paanee huregg Sordie
-0.25
+0.50
20020
-0.15
-1.00
10350
13 Epoch
Epoch
Epoch
Figure 3: The number, prediction confidence and logits variation of normal/abnormal adversarial examples and training samples. The left, middle and right panel shows the number, prediction con- fidence and logits variation, respectively. The green/red and blue lines represent normal/abnormal adversarial examples and training samples, respectively. Dashed black lines correspond to the 10th epoch, which the point that model occurs CO. The yellow line represents the number, prediction confidence and the logits variation of abnormal adversarial examples under the AAER method.
at a very large number. Given this observation, we can infer that there is a close correlation be- tween the number of abnormal adversarial examples and the CO phenomenon, which also prompts us to wonder (Q1): whether CO can be mitigated by reducing the number of abnormal adversarial examples.
3.2 OUTPUTS VARIATION OF NORMAL AND ABNORMAL ADVERSARIAL EXAMPLE
The above observations indicate that CO and the number of abnormal adversarial examples are closely related. In this part, we further analyze the outputs variation of normal and abnormal ad- versarial examples during CO. Specifically, confidence and logits, and use the cross-entropy to calculate the prediction confidence variation during the inner maximization process, which is formulated
analyze the outputs into two categories: prediction
follows:
8 (x + + y, 0) E (ar + 7.35 0).
From Figure 3 (middle), we can observe that the change in the prediction confidence of normal adversarial samples is greater than indicating that the perturbation leads to worse predictions. However, the variation of abnormal adversarial examples is atypical negative, meaning that the per- turbation has the opposite effect we expected. Furthermore, we analyze the prediction confidence variation of abnormal adversarial examples during training. Before the occurrence of CO, we can observe that the prediction confidence variation abnormal adversarial examples slightly less than zero, and the negative impact all training samples (blue line) is not significant. During the occurrence of CO, their prediction confidence variation decreases rapidly slumped 27 times at the 10th epoch, and deterministically effect the prediction confidence of all training samples.
Furthermore, we compare the magnitude of the outputs change between normal and abnormal adver- sarial examples, and use the Euclidean distance (I.2 distance) calculate the sample logits variation during the inner maximization process, which is formulated as follows:
[o (x 17 + 8) fo (x 1r++00 11) 12.
where fo is the DNN classifier parameterized by and || [/ the L2 distance.
The magnitude of the logits variation of normal and abnormal adversarial examples is shown in Fig- ure 3 (right). We can observe that the logits variation magnitude of abnormal adversarial examples increases dramatically during CO, which is 16 times larger than that before CO. A single-step gradi- ent ascent can bring an earth-shaking change the output logits, which generally happens on highly distorted decision boundaries. Additionally, we observed that the logits variation magnitude of the normal adversarial examples (green line) increases one epoch later than the abnormal ones, which indicates that the model boundary distortion mainly lies in the abnormal adversarial examples. in other words, directly optimizing the network with these abnormal adversarial examples will further exacerbate the model boundary distortion. Moreover, further compare the magnitude of logits
5

---

Under review as a conference paper at ICLR 2023
variation for normal and abnormal adversarial examples. From Figure 3 (right), (right), we can observe that the logits variation magnitude on normal and abnormal adversarial examples is similar before CO. However, there is significant difference in the logits variation magnitude between these two types of examples after CO. It observed that the logits variation magnitude abnormal adversarial examples is 4 times that in normal ones the 10th epoch. There significant differences in the magnitude of both prediction confidence and logits variation between normal and abnormal adver- sarial examples, which inspires the outputs variation of abnormal adversarial examples.
wonder (Q2): whether CO can be mitigated by constraining
3.3 ABNORMAL ADVERSARIAL EXAMPLES REGULARIZATION TERM
We answer these two questions through three optimization objectives. To answer the QI, the first part (i) uses the Eq. to divide the training samples into normal and abnormal adversarial examples, and then penalize the number of abnormal adversarial examples. To answer the Q2, the second part (ii) and the third part (iii) constrain the outputs variation of abnormal adversarial examples. Specifically, the second part (ii) calculates the prediction confidence variation of abnormal adversarial examples. and then penalizes this variation that should not decrease during the inner maximization process, which is formalized as follows:
Σ (e (1 (nnlmormal U. (1 +1+ +.
where n is the number of abnormal adversarial examples.
The third part (iii) calculates the logits variation of normal and abnormal adversarial examples. Since the logits variation is representation of the change magnitude, which is not related to the label, there is no clear target value for the optimization standard. Therefore, we use the logits variation of normal adversarial examples as the standard and explicitly make them logits variation closer. In order to avoid the network only focusing on increasing the logits variation of abnormal adversarial examples instead of reducing the abnormal ones, use the max function to limit the minimum value to O, which is formalized as follows:
(ilfo (a Aannormal +1+ - fo (1. LnAbborraa 17) 112)
(llf (x Normal
+n+ 6) Normal 17) 112)
72
where is the number of training samples and max(,) is the max function.
Based on the above analysis, we design a novel regularization term, AAER, which aims to suppress the abnormal adversarial examples by (i) the number, (ii) the prediction confidence variation and (iii) the logits variation, ultimately achieving the purpose of preventing CO, which is shown in the following formula:
AAER = n
(Eq.
8
A1 Eq. X2).
(10)
m
where A1 and X2 is the hyperparameter to control the strength of the regularization term.
AAER can effectively hinder the generation of abnormal adversarial examples which are highly cor- related with distorted classifier, thereby encouraging training of smoother classifiers that can better defend against adversarial attacks. Furthermore, the strength of AAER depends on the product of the number and outputs variation of abnormal adversarial examples, which can more comprehen- sively and flexibly reflect the degree classifier distortion. The algorithm realization is summarized in Algorithm IT Note that employ increasing to stabilize the optimization objective and avoid model training to crash in the early stages.
EXPERIMENT
In this section, conduct extensive experiments to verify the effectiveness of AAER including periment settings (Section[4.I) performance evaluations (Section 4.2), ablation studies (Section 4.3) and time complexity study (Section 4.4).

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Abnormal Adversarial Examples Regularization (AAER)
Input: network fa, epochs T, mini- M, perturbation radius c, step size a, initialization term 1).
1: for /- 1...7 .....Tuu do
2: for = 1...... M to do
3:
a= =t/T-a
4:
8 == a. sign (V Vatn + 77 0)) L [1) + 17 + 6, y::
5: CEloss 1
6:
AAERloss Eq.
7: 0 0= Vo (C + L/T AAERIoss,)
8: end for
9: end for
4.1 EXPERIMENT SETTING
Baselines.
We compare method with other SSAT methods including RS-FGSM (Wong al.
2020). ATTA (Zheng et al. 2020). FreeAT (Shafahi et al. 2019). N-FGSM (de Jorge et al. 2022).
Grad Align (Andriushchenko & Flammarion 2020). ZeroGrad and MultiGrad (Golgooni et al.
2021 We also compare our method with iterative-step AT PGD-2 and PGD-10 (Madry et al. (017]
providing reference for the ideal performance. To accommodate different adversarial budgets, we use PGD-10 with two step size of 2/255 and e/10. We will show natural and robust accuracy results using the hyperparameters reported in their official repository (except for FreeAT, we do not divide the number of epochs by to keep the same training epochs). It is worth noting that we do not use early stopping (Wong et al. 2020) as this technique can restore the robustness of all methods.
Datasets and Model Architectures. We will show the results on the benchmark datasets Cifar- 10/100 (Krizhevsky et al. 2009) and use random cropping and horizontal flipping for data argumen- tation. We use the PreactResNet-L1 (He et al. 2016) and WideResNet-34 (Zagoruyko & Komodakis 2016) architectures on these datasets to evaluate results. The training results of WideResNet-34 also available in the Appendix B. We also report the settings and results of our method on SVHN (Netzer et al. 2011) and Tiny-imagened (Netzer et al., 2011) in the Appendix
Attack Methods and Learning Rate Schedule. To report the robust accuracy of models, attack these methods using the standard PGD adversarial attack with a = c/4, 10 restarts and 50 attack steps. We also evaluate methods based Auto Attack in the Appendix [ We the cyclical learning rate schedule (Smith) 2017) with 30 epochs that reaches its maximum learning rate (0.2 in our experiments) when half of the epochs (15) are passed Cifar-10/100.
Setup for Our Proposed Method. In this work, we use the SGD optimizer with momentum of 0.9 and weight decay of 10-4, We chose Lee as the threat model and set gradient ascent step size = 1.5 e. We set 17 = Uniform(-e, e) for random initialization, and the 7) setting for previous initialization can found the Appendix A We will show the best À settings in the Appendix [D] It is worth noting that our method can also achieve robustness without tuning hyperparameters with different adversarial budgets, the results on universal À in the Appendix
4.2 PERFORMANCE EVALUATION
In this part, we report the experimental results of our method under four different settings AAERReE AAER with random initialization and clipped perturbations and AAERRUC: AAER with random initialization and unclipped perturbations. The unclipped technique was proposed by de Jorge et al. (002),, who claimed that clipping is performed after taking a gradient ascent step, which may make adversarial samples no longer effective. The AAER based on the previous initialization is available in the Appendix[A] A
CIFAR1O Results. In Table I present evaluation of the proposed methods with the compet- ing baselines on the CIFAR- I0 dataset. First, we can observe that RS-FGSM, ATTA and FreeAT suffer from CO with strong adversaries. We can also observe interesting phenomenon that some weakly robust methods will recover partial robustness with large noise magnitude 32/255. Table shows that proposed methods can significantly improve the robust accuracy, achieve superior

---

Under review as a conference paper at ICLR 2023
Table 1: CIFARI0/100: Accuracy of different methods and different noise magnitudes using PreActResNet-Ie 18 under Lo threat model. The left and right panel are the CIFARIO and CIFAR100 results, respectively. The top number the natural accuracy while the bottom number is the PGD- 50- 10 accuracy. The results are averaged over random seeds and reported with the standard devi- ation.
dataset
CIFARIO
CIFARI00
noise magnitude
8/255
12/255
16/255
32/255
8/255
12/255
16/255
32/255
83,91 0.21
66.46 22.80
66.54
12.25
36.43 7.86
60.29 + 1,51
11.03 5.24
11,40 8.60
RS-FGSM
46.01 = 0.18
+.00000000
0.00
0.00
0.00 + 0.00
10.58 +: 13.10
0.00 a 0.00
0.00 = 0.00
0.00 + 0.00
86.41
81.15 0.94
0.71
4.48
0.25
37.40 16.34
11.12
21.71
ATTA
42.15 a 0.42
19.55 1: 15.20
0.00 0.00
7.84 2 6.80
22.78 4: 0.19
0.00 a 0.00
0.00 = 0.00
0.00 1: 0.00
76.20 1.09
19.07
8.41
0.30
39.84 0,40
2.48 0.00
FreeAT
0.62
0.33
0.20
0.00
81.60 0.16
77.52 0.21
79:65 0.17
65.48 6.26
53.83 a 0.22
49.07 0.14
50.76 0.02
47.75
ZeroGrad
47.56 0.16
27:34
25.02 0.24
14.76 0.26
5.23 0.09
0.01 0.02
81.65 = 0.16
81,09 1 4.67
82.98 1 3.30
41.90 30.53
53.11 + 0.34
46.81 + 0.51
46.05 = 8.68
16,25 10.48
MultiGrad
47,93
9.95 + 16.97
0.00
5.69 5.14
25.68 0.21
16.56 0.56
0.00 0.00
0.53 +
80.48 + 0.21
71.30 + 0.12
62.96 + 0.74
32,44 = 2.79
54.87 + 0.28
46 46.16 0.13
37.76 0.16
15.56 4.55
N-FGSM
47,91 0.29
36.23 0.10
27.14 a 1.44
10.58 0.89
26.51 a 0.38
18.75 a 0.19
14.14 0.05
1.61 2.77
82.10 0.78
74.17 + 0.55
60.37 a: 0.95
25.23 = 3.41
54.00 2: 0.44
45.83 a 0.72
36.80 0,10
15:05 a: 0.07
Grad Align
47,77 + 0.58
34.87 + 1,00
27.90 + 1.01
11,53 = 3:23
25.27 + 0.68
18.13 0,71
13.77 0.76
2,85 1.34
84.33 = 0.08
76.07 + 0.78
65.88 + 0.60
26.14 1.05
58.94 0.53
49.09 + 0.16
39.44 0.88
20.03 = 1.08
AABRaC
46.42 0.19
32,94 + 0.31
24.67 = 0.52
14.70 = 0.39
25.34 0.21
16.94 = 0.24
12,33 = 0.14
5,20 + 0.17
77.41 = 0.13
66.24 0.60
55.32 + 0.55
20.57 = 2.61
$3.60 0.19
40.56 + 0.33
29.27 = 0.51
12.47 + 1.26
AAERRuC
51,51 = 0.22
38.66 + 0.38
30,31 # 0.15
12.72 = 0.52
28.50 0.15
19.69 a 0.23
13.96 0.07
4.61 0.09
85.07 a 0.12
78.97 a: 0.23
72.31 A 0.40
48.45 = 0.71
60.09 4 0.20
53.46 a 0.27
47,50 a 0.28
31.89 & 0.69
PGD-2
45.27 + 0.07
32.99 + 0.46
24.32 + 0.64
11:24 = 0.40
24.58 + 0.12
17.16 0.21
12.69 0.06
4,51 + 0.21
80.55 0.37
72.37 + 0.31
67.20 0.69
= 0.84
55.05 + 0.25
47.42 0.29
42.39 0.17
34, 0.23
PGD-10 (2/255)
50.67 = 0.40
38.60 a: 0.39
29.34 #: 0.18
5.80 a 0:23
27.87 + 0.12
20.29 # 0.18
15.01 # 0.21
3.81 # 0.12
84.74
0.57
0.46
0.78
0.37
52.74
0.32
26.07
(GD-10(e/10)
46.06 = 0.32
34.13 0.62
26.07 = 0.69
15.16 0.34
24.71 2.11
17.62 0.13
13.39 0.10
6,46 : 0.22
robustness to other SSAT methods and even have comparable robustness to PGD AT. Interestingly, unclipped perturbations methods always have better performance, except for 32/255 noise mag- nitude, where we conjecture that the unclipped perturbations are too large to disturb the original features of the inputs.
CIFAR100 Results. We also conduct experiments on the CIFAR100 dataset. Note that CIFAR100 is more challenging than CIFAR10 the number of classes/training images per class is ten times larger/smaller than that of CIFAR10. As shown by the results in Table [ the proposed methods are still able prevent CO and improve robust accuracy. It verifies that the AAER can reliably prevent CO and is general across different datasets.
4.3 ABLATION STUDY
In this part, investigate the impacts of AAERRC with 16/255 noise magnitude using PreactResNet-188 on CIFARIO dataset under Loo threat model.
Optimization Objectives. To verify the effectiveness of our proposed method, we show the change in the three optimization objectives during training in Figure 3 We can observe that the number, prediction confidence and logits variation of abnormal adversarial examples are well constrained by AAER throughout the training. We also try to simply ignore abnormal adversarial examples and train only on normal ones. Unfortunately, this method does not work due the abnormal
8

---

Under review as a conference paper at ICLR 2023
80%
60%
70%
60%
Robustness
Robustness
50%
Natural Accuracy Robust Accuracy
40%
40%
30%
20%
30%
10%
0.0
4.0
8.0
12.0
14.0
0.0
2.5
5.0
7.5
10.0
X1 Size
Size
Figure 4: Ablation Study. The red and green line are the natural and robust test accuracy, respec- tively. Left panel: Effect of different sizes X1, and we fix X2 as 3.0 at this experiment. Right panel: Effect different sizes X2, and we fix as 8.0 this experiment.
adversarial example is not the cause of decision boundary distortion but rather co-occurs. Thus, ignoring abnormal adversarial examples cannot repair existing decision boundary distortion.
Selection. To verify the effectiveness of proposed method, investigate the effect of dif- ferent sizes Al and X2 on natural and robustness performance. From the figure 4 (left), we can observe that the effect of Al does seem to be significant. However, acts buffer prevent the AAER from changing too drastically, and we chose À1 of 8.0 for optimal preference. From the Figure (right), we can observe that the model can successfully prevent CO when A2 is not O, which proves that our method can effectively eliminate CO. Under the same experimental setting mentioned before, with the value of À varying from 10.0, we can observe that choosing A2 of 5.0 can achieve the best robustness.
4.4 TIME COMPLEXITY STUDY
We show the time complexity different AT methods in Table 2, we can observe that the running time for one epoch of AAER is almost equal to the RS-FGSM method. In contrast, Grad Align and PGD-10 are 2.3 and 4.6 times slower than our method, respectively.
Table CIFAR1O training time on a single NVIDIA Tesla V100 GPU using PreactResNet-11 18. The results are averaged over 30 epochs.
Method
RS FGSM
ATTA
FreeAT
ZeroGrad
MultiGrad
Second Epoch
26.1S
41.5S
106.6S
28.7S
52.15
Method
N FGSM
Grad Align
AAER (Our)
PGD3-2
PGD-10
Second Epoch
25.9S
69.4S
30.55
39.1S
140.75
CONCLUSION
In this paper, we find that the abnormal adversarial examples exhibit anomalous behaviour, i.c. they are further to the decision boundaries after adding perturbations generated by the inner max- imization process. We empirically show that the catastrophic overfitting is closely related to the abnormal adversarial examples by analyzing their number and outputs variation during model train- ing. Motivated by this, we propose novel and effective method, Abnormal Adversarial Examples Regularization (AAER), through a regularizer to eliminate catastrophic overfitting by suppressing generated abnormal adversarial examples. Our approach successfully resolve the catastrophic overfitting with different noise magnitudes and achieve state-of-the the preference with computa- tional convenience in various settings.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Maksym Andriushchenko and Nicolas Flammarion. Understanding and improving fast adversarial training. Advances in Neural Information Processing Systems, 33: 13:16088-16059, 2020.
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of se- curity: Circumventing defenses adversarial examples. In International conference on machine learning, pp. 274-283, PMLR, 2018.
Varun H Buch, Irfan Ahmed, and Mahiben Maruthappu. Artificial intelligence in medicine: current trends and future possibilities. British Journal of General Practice, 68(6688:143-144, 2018.
Renjie Chen, Yuan Luo, and Yisen Wang. Towards understanding catastrophic overfitting fast adversarial training. 2021.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning, pp. 2206- 2216. PMLR, 2020.
Pau de Jorge, Adel Bibi, Riccardo Volpi, Amartya Sanyal, Philip HS Torr, Grégory Rogez, and Puneet K Dokania. Make some noise: Reliable and efficient single-step adversarial training. arXiv preprint 1821v:2202.01181, 2022.
Zeinab Golgooni, Mehrdad Saberi, Masih Eskandar, and Mohammad Hossein Rohban. Zerograd:
Mitigating and explaining catastrophic overfitting in fgsm adversarial training. arXiv preprint MrXiv:2103.15400 2021.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint 111iv:1412.6572, 2014.
Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens Van Der Maaten. Countering adversarial images using input transformations. arXiv preprint ar 1.000// 17, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645. Springer. 2016.
Zhichao Huang, Yanbo Fan, Chen Liu, Weizhong Zhang, Yong Zhang, Mathieu Salzmann, Sabine Süsstrunk, and Jue Wang. Fast adversarial training with adaptive step size. arXiv preprint arXiv:2206.02411 2022.
Guy Katz, Clark Barrett, David L Dill, Kyle Julian, and Mykel Kochenderfer. Reluplex: An efficient smt solver for verifying deep neural networks. In International conference on computer aided verification, pp. 97-117. Springer, 2017.
Hoki Kim, Woojin Lee, and Jaewook Lee. Understanding catastrophic overfitting in single-step adversarial training. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 8119-8127, 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alexey Kurakin, Ian J Goodfellow, and Samy Bengio. Adversarial examples the physical world. In Artificial intelligence safety and security, pp. 99-112. Chapman and Hall/CRC, 2018.
Tao Li, Yingwen Wu, Sizhe Chen, Kun Fang, and Xiaolin Huang. Subspace adversarial training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 13409-13418, 2022.
Todd Litman. Autonomous vehicle implementation predictions. Victoria Transport Policy Institute Victoria, BC, Canada, 2017.
Guanxiong Liu, Issa Khalil, and Abdallah Khreishah. Using single-step adversarial training to defend iterative adversarial examples. In Proceedings of the Eleventh ACM Conference on Data and Application Security and Privacy, pp. 17-27, 2021.
10

---

Under review as a conference paper at ICLR 2023
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.060S. 2017.
Jan Hendrik Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff. On detecting adversarial perturbations. arXiv preprini urXiv:/702.042000 2017.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Geon Yeong Park and Sang Wan Lee. Reliably fast adversarial training via latent adversarial per- turbation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp- 7758-7767, 2021. 2021.
Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! Ad- vances in Neural Information Processing Systems, 32, 2019.
Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K Reiter. Accessorize to crime: Real and stealthy attacks on state-of-the-art face recognition. Proceedings of the 2016 acm sigsac conference computer and communications security, pp. 1528-1544 2016.
Leslie N Smith. Cyclical learning rates for training neural networks. In 2017 IEEE winter conference on applications of computer vision (WACV), pp. 464-472. IEEE, 2017.
Gaurang Sriramanan, Sravanti Addepalli, Arya Baburaj, et al. Towards efficient and effective adver- 1121--11333,, 2021.
sarial training. Advances in Neural Information Processing Systems,
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruma, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6799 2013.
BS Vivek and Venkatesh Babu. Regularizers for single-step adversarial training. arXiv preprint arXiv:2002.00614 2020a.
BS Vivek and R Venkatesh Babu. Single-step adversarial training with dropout scheduling. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 947-956. IEEE, 2020b.
Eric Wong, Leslie Rice, and J Zico Kolter. Fast is better than free: Revisiting adversarial training. arXiv preprint urXiv:2001.0399) 2020.
Sergey Zagoruyko and Nikos Komodakis.
Wide residual networks.
arXiv preprint
arXiv:1605.07146. 2016.
Haizhong Zheng, Ziqi Zhang, Juncheng Gu, Honglak Lee, and Atul Prakash. Efficient adversarial training with transferable adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1181-1190, 2020.
11

---

Under review as a conference paper at ICLR 2023
A EXPERIMENT WITH PREVIOUS INITIALIZATION
Most works build perturbations based on zero or random initialization, but Zheng et al. (2020) and Liu et al. (202I) found that perturbations are highly transferable between models from adjacent epochs, so they used perturbations from adjacent epochs to intensify the effect of perturbations,
which is formalized as follows:
T (Mh-1 5(-1). 3,
(11)
where is the epoch, Th- and 6,-1 saved from the adjacent epoch and 3 is the hyperparameter to control the strength of the initialization.
In this part, we will show the effect of our method by using previous initialization by AAERpC: AAER with previous initialization and clipped perturbations and AAERpuC: AAER with previous initialization and unclipped perturbations. We set 3 = 0.5 for the previous initialization experi- ments, and report the results on Cifar in Table 3 and Table
Table 3: CIFARIO: Accuracy of different methods and different noise magnitudes using PreActResNet-I 18 under Lo threat model. The top number is the natural accuracy while the bottom number is the PGD-50-10 10 accuracy. The results are averaged over 3 random seeds and reported with the standard deviation.
mugnitude
8/255
12/255
16/255
32/255
AAERPC
83.68 = 0.14
73.03 1,90
62.67 1.32
28.39 1,55
with ATTA
47.01 = 0.29
33.56 + 0.67
25.01 = 0.50
13.93 = 0.60
0.25
74.29 0.63
63.52 1.41
26.90 1.64
AAEReC
4..14
33.38 + 0.71
24.83 + 0.75
14.21 = 0.38
AAERruc
#7.25
63.54 0.63
45,93 3.66
19.28 1.08
with ATTA
51.25
38,39 = 0.70
27.34 1.91
9.96 = 0.96
77.23
63.68
49.45
23.20
AABReuC
50.66
38.51 + 0.29
28.85 + 0.31
13.11 = 0.68
Table 4: CIFAR100: Accuracy of different methods and different noise magnitudes using PreActResNet-L 18 under Lx threat model. The top number is the natural accuracy while the bottom number is the PGD-50-10 10 accuracy. The results are averaged over 3 random seeds and reported with the standard deviation.
noise magnitude
8/255
12/255
16/255
32/255
AAERPC
$7.66 0.28
47.44 + 0.48
37.10 0.43
20.00 0.22
with ATTA
0.13
16.97 +
12.00
4.68
$7.66 0.82
45.59 = 0.57
35.30 0.23
17.41 0.53
AAEReG
0.41
16.53 + 0.16
12.03 +
5.05
AAERpuC
50.91 1.08
37.93 1 0.75
25.98
9.62 0.51
with ATTA
27.43 = 1.00
18.86 + 0.48
12.81 +: 0.56
4.14
$2.52
36,97 0.87
24.90 0.70
11.08 1.48
AAEReuC
27.94 1 0.34
19.19 a 0.62
13.32 1: 0.36
4.43 0.21
From Table and Table 4 we can observe that our method with previous initialization can still successfully achieve high robustness, even achieving better robustness in settings compared to random initialization. However, using previous initialization has some negative effects on natural accuracy.
The hyperparameter 3 determines the strength the previous initialization perturba- tions, and the effect of different 33 test accuracy is shown in Figure When / is 0 which is
Selection.
12

---

Under review as a conference paper at ICLR 2023
70%
60%
Robustness
50%
Standard Accuracy
Robust Accuracy
40%
30%
20%
0.0
0.25
0.5
0.75
1.0
B Size
Figure 5: Effect of different size B. The red and green line are the natural and robust test accuracy, respectively. We do this experiment based on AAERpC with 16/255 noise magnitude.
equivalent to using initialization, increasing / leads higher natural accuracy. When B is greater than 0.5, it is observed increasing 3 makes model robustness decrease. Therefore, we set B 0.5 to achieve the best trade-off between natural and robust test accuracy.
Data Argumentation Technique. We notice that Zheng et al. 22200) proposed a data argumenta- tion technique ATTA, which adds different arguments each epoch. We add this data argumentation technique on AAERec shown in Table 3 and Table 4 We can observe that the ATTA does not or slightly improves our method accuracy, but the training time will significantly increase from 30.5S to 43.1S. Therefore, our method AAER do not use the data argumentation technique ATTA.
B EXPERIMENT WITH WIDERESNET ARCHITECTURE
We also compare the performance of our method using WideResNet, which is more complex than Nett The settings are the same as PreActResNet-18, and we report the results on Ci- far10/100 in Table Table
Table CIFARIO: Accuracy of different methods with 8/255 noise magnitude using WideResNet- 34 under Loe threat model. The results are averaged over 3 random seeds and reported with the standard deviation.
method
AAERaC
AAERaue
AAERec
AAERoue
PGD-2
PGD-10 (2/255)
PGD-10 (c/10)
natural accuracy
87.83 = 0.14
82.34=0.23
86.72
00.22
= 0.14
85.53 0.22
88.51 0.30
robust accuracy
47.54
=
4..8
5.44
47.32
0.50
53.70 0.53
47.72 0.84
training time
227.58
281.68
1031.58
From Table 5 and Table 6 we can observe that method can still successfully achieve high robustness in other architectures. Although, the PGD-10 AT seems to better utilize the complex network to achieve higher natural and robust accuracy. However, it is worth noting that complex networks can better reflect the efficiency of our method in terms of training time, while our method achieve comparable robustness.
13

---

Under review as a conference paper at ICLR 2023
Table CIFARI00: Accuracy of different methods with 8/255 noise magnitude using WideResNet- 34 under Loe threat model. The results are averaged over 3 random seeds and reported with the standard deviation.
method
AAERaC
AAERRUC
AAERPC
AAERpuC
PGD-2
PGD-10 (2D-50(2/255)
PGD-10 (c/10)
accuracy
61.75
61.23
0:27
robust accuracy
2
29.89 0.66
2011
26.47 3.28
0.10
30.02 0.09
26.45 0.30
traning time
228.58
285.75
1036.75
C EVALUATION BASED ON AUTO ATTACK
Auto Attack Croce Hein 2020) is regarded as most reliable robustness evaluation to date, which is ensemble of complementary attacks, consisting of three white-box attacks APGD-CE, APGD-DLR, and FAB and a black-box box attack Square Attack. We report the results on Cifar10/100 in Table and Table
Table CIFARIO: Accuracy different methods and different noise magnitudes using PreactResNet- 18 under Log threat model. The number is the Auto Attack accuracy while the natural accuracy is same as Table [ The results are averaged over random seeds and reported with the standard deviation.
noise magnitude
8/255
12/255
16/255
32/255
RS-FGSM
43.17 0.34
0.00 0.00
0.00 0.00
0.00 0.00
ATTA
40.09 0.39
16.18 14.03
0.00 0.00
4,90 4.224
FreeAT
40.23 0.33
28.04 0.73
0.00 0.00
0.00 0.00
ZeroGrad
43.48
MulitGrad
44.39
N-FGSM
44.43 0.24
30.32 0.08
19.06 1.81
6.78 0.75
Grad Align
44.82 0.09
30.05 0.17
19.60 0.47
7.89 2.62
AAERgC
43.22 + 0.24
27.20 0.35
16.91 # 0.41
10.54 + 01.67
AAERRuC
46,29 +: 0.23
31.00 4: 0.17
21.16 + 0.15
10.72 # 1.74
AAERPc
43.58 + 0.28
27.22 + 0.45
16.87 + 0.38
9.76 0.46
AAERpuC
45.44 0.44
31.01 0.35
21.33 + 0.22
9.50 0.79
PGD-2
42.97 0.65
0.38
18.52 =
3.77 0.02
PGD-10 (2/255)
46.95 0.54
33.30 0.20
22.29 0.27
2.29 0.10
PGD- PGD-10 (c/10)
43.44 0.45
29.82 0.43
19.92 a 0.63
9.61 0.41
In Table Table we can observe that our method can still achieve boost adversarial robustness in different adversarial attacks. Surprisingly, the unclipped AAER achieves higher robustness with 32/255 noise magnitude under Auto Attack, which is slightly different from the result under PGD- 50- 10 attack.
D EXPERIMENT WITH UNIVERSAL À
It is worth noting that unlike other SSAT methods (such as Grad Align (Andriushchenko & Flam- marion 2020) and ZeroGrad (Golgooni et al.) 202T)), our method can achieve robustness without uning hyperparameters with different adversarial budgets, the universal À settings are shown in Ta-
14

---

Under review as a conference paper at ICLR 2023
Table CIFARI00: Accuracy of different methods and different noise magnitudes using PreactResNet- 18 under Lo threat model. The number is the Auto Attack accuracy while the natural aceuracy is same as Table I
The results are averaged over random seeds and reported with the
standard deviation.
noise magnitude
8/255
12/255
16/255
32/255
RS-FGSM
7.98 11.91
0.00 0.00
0.00 0.00
0.00 + 0.00
ATTA
20.09 + 0.06
0.00 + 0.00
0.00 = 0.00
0.00 + 0.00
FreeAT
18.28 * 0.20
12.37 + 0.14
0.00 0.00
0.00 + 0.00
ZeroGrad
21.15
MulitGrad
21.62
N-FGSM
22.68 0.25
14.57 0.09
10.30 0.14
0.78 1.36
Grad Align
21.87 + 0.13
13.78 0.11
9.64 0.12
1.76 + 0.70
AAERRe
21.79 a: 0.11
12.92 a 0.20
8.81 0.07
2.66 a 0.10
AAERRUC
23.77 4 0.10
14.62 + 0.24
9.86 0.11
2.94 +: 0.14
AAERpe
21.79
12.48 0.25
8.38 0.18
2.54 + 0.13
AAERroc
2.23
14.23 0.27
9.16 0.22
2.59 0.30
PGD-2
21.52
13.69 0.02
9.56 0.07
1.76 0.22
PGD- 10 (2/255)
23.78 0.08
15.61 = 0.09
10.93 + 0.05
2.18 + 0.09
PGD- 10 (s/10)
21.60 + 0.03
13.95 = 0.10
10.18 + 0.08
3.76 + 0.10
ble and Table TO For CIFAR-10, we set À1 8.0 A2 5.0 for AAER with clipped perturbations, and 6.5 X2 5.0 for AAER with unclipped perturbations. For CIFAR- 100, we set À1 = 7.5 X2 = 3.0, and A1 6.5 X2 2.5 for AAER with clipped and unclipped perturbations, respectively.
Table 9: CIFARIO: The best and universal setting for different noise magnitudes. Last panel is universal À setting, other panels are best setting. The top number Àl while the bottom number X2.
Datnset / Method
8/255
12/255
16/255
32/355
Universal
clipped perturbations
3.0
4.5
5.0
7.5
5.0
6.5
unclipped perturbations
1.5
3.0
5.0
10.0
5.0
Table 10: CIFARI00: The best and universal setting for different noise magnitudes. Last panel is universal À setting, other panels are best À setting, The top number is Ål while the bottom number A2.
Dataset / Method
8/255
12/255
16/255
32/355
Universal
6.5
7.0
7.5
9.0
7.5
clipped perturbations
2.0
2.5
3.0
1.5
3.0
4.0
5.0
6.5
7.5
6.5
unclipped perturbations
1.0
2.0
2.5
1.5
2.5
15

---

Under review as a conference paper at ICLR 2023
We report the universal À results Cifar10/100 in Table 11 and Table 12 We can observe that the results using universal À can still achieve high robustness on both datasets. The absence of hyperparameter tuning provides method with unparalleled generality and adaptability.
Table 11: CIFARIO: Accuracy of universal AAER with different noise magnitudes using PreactResNet-11 18 under Lx threat model. The top number is the natural accuracy while the bot- tom number is the PGD-50-10 accuracy. The results are averaged over random seeds and reported with the standard deviation.
noise mugnitude
8/255
12/255
16/255
32/255
84.11 + 0.43
74.73 = 1.19
65,88 +: 0.60
25.86 = 0.57
ACERRC (Al 8.0, X2 5.0)
46.04 + 0.46
32.66 0.33
24.67 0.52
14.64 = 0.27
76.06 0.59
64.26 0.52
55.32 0.55
24.56 + 0.71
AAERRuc (65 =2 5.0)
50.50 a: 0.19
38.03 0.12
30.31 0.15
11.29 1.79
83.39 0.20
72.15 0.55
63,52 + 1,41
27.71 1.65
AAERPC (Al 8.0, A20
46.99 0.31
33,20 0.47
24.83 0.75
13.70 + 0.27
76.03 = 0.27
60.57 1.24
49,45 = 1,14
21.21 1.29
AAERpUC (A1 6.5,, 5.0)
50.60 = 0.08
37.01 0.28
28.85 0.31
12.14
Table 12: CIFAR100: Accuracy of universal AAER with different noise magnitudes using PreactResNet-188 18 under Lxx threat model. The top number is the natural accuracy while the bot- tom number is the PGD-50-10 accuracy. The results are averaged over random seeds and reported with the standard deviation.
moise magnitude
8/255
12/255
16/255
32/255
56.32 0.40
47.37 0.63
39.44 0.88
16.94 0.94
AAEReo (A1 - X2 3.0)
24.86 +
16.41 0.06
12.33 0.14
5.12 + 0.09
49.80 0.47
38.56 0.70
29:27 0.51
10.88 1.22
AAERRue x2
27.47 =
18.40 = 0.68
13,96 + 0.07
4.14 0.23
55.53 1.05
45.20 0.67
35.30 0.23
13.84 L.05
AAERpe (A1 = A2
25.14 0.41
16.17 = 0.44
12,03 + 0.29
4.85 + 0.21
49.04
36.82 1.00
0.70
0.54
AAERPUC (A1 6.5, X2 2.5)
27.09 = 0.17
18.53 0.57
13.32 0.36
3.78 + 0.09
E SETTINGS AND RESULTS ON SVHN AND TINY-IMAGENET
SVHN Settings and Results. For experiments on SVHN, we follow the settings of de Jorge et al. 2022). which use the cyclical learning rate schedule with 15 epochs that reaches maximum learning rate (0.05) when 40% (6) epochs are passed. We show the best X settings on SVHN in the Table 13 In Table 14, we show the AAER performance on the SVHN dataset, and the competing baselines result from de Jorge et al. (2022). We can observe that our method can effectively improve the robust accuracy and prevent CO with different noise magnitudes.
Tiny-imagenet Settings and Results. For experiments on Tiny-imagenet, we use the cyclical learn- ing rate schedule with 30 epochs that reaches its maximum learning rate (0.2) when half of the epochs (15) are passed. We set A1 3.0 X2 1.0 for AAER with clipped perturbations, and AL = 4.0 X2 = 0.5 for AAER with unclipped perturbations. In Table [5] we show the AAER performance on the Tiny-imagener dataset. We can observe that our method can effectively improve the robust accuracy and prevent CO.
16

---

Under review as a conference paper at ICLR 2023
Table 13: SVHN: The best setting for different noise magnitudes. The top number is while the bottom number is X2.
Method
4/255
12/255
3.0
3.5
5.0
clipped perturbations
5.0
unclipped perturbations
Table 14: SVHN: Accuracy of different methods and different noise magnitudes using PreActResNet-Il 18 under Lo threat model. The top number is the natural aceuracy while the bottom number is the PGDD5001B 10 accuracy. The results are averaged over 3 random seeds and reported with the standard deviation.
noise magnitude
4/255
8/255
12/255
95.09 1 0.09
94.46 a: 0.16
92.74 1: 0.5
RS-FGSM
71.28 a 0.4
0.0 0.0
0.0 0.0
93.66 = 0.12
91.29 a 4.07
92.36 + 1.0
FreeAT
71.61 0.75
0.01 a: 0.0
0.0 0.0
94.81 + 0.16
92,42 a: 1.29
88.09 0.4
ZeroGrad
71.59 0.22
35,93 2.73
14.14 0.32
94.71 0.17
94.86 + 0,97
94.48 0.19
MultiGrad
71.98 0.26
11.49 16.19
0.0
94.54 = 0.15
89.56 + 0.49
81,48 1.64
N-FGSM
72.53 0.19
45.63 0.11
26.13 0.81
94.56 0.21
90.1 0.34
84.01 0.46
Grad Align
43.85 0.14
23.62 0.41
94.75 0.70
91.40 a: 0.85
82.18 4.13
AAERac
42.55 0.55
0.51
93.81 0.26
87.11 0.557
78.91 0.84
AAERaue
73.41
0.54
PGD-2
73.29 = 0.29
20.68 18.56
0.02 = 0.03
89.67
PGD-10
74.76 + 0.19
53.95 + 0.55
37.65 0.53
Table 15: Tiny-imagenet: Accuracy of different methods with 8/255 noise magnitude using PreActResNNt-11 under L threat model. The results are averaged over 3 random seeds and re- ported with the standard deviation.
method
AAERaC
AAERRuC
RS-FGSM
N-FGSM
PGD-2
natural accuracy
47.92 0.39
44.38 + 0.27
52.28 42.64
48.16 0.61
46.43 0.35
robust accuracy
19.38 0.14
0.00 0.01
0.00
20.82 = 0.40
20.72
17


Reviews:
[{"review_id": "zWGwZY2t6_", "reviewer": "Reviewer_gJHh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Recently, FGSM-based adversarial training faced an emergency problem of catastrophic overfitting. To mitigate this, the authors first observed that the catastrophic overfitting is related to some abnormal examples, whose loss decreases under the FGSM attack. Based on this observation, in this paper they proposed a new abnormal adversarial example regularization (AAER), which considers two key components: one is about the number of the abnormal examples, while the other is about the logits variation. The experiments on CIFAR-10/100 show that the proposed AAER is effectiveness.\n  ", "summary_of_the_review": "This paper finds some interesting observations, but the proposed technique does not appear to be strongly correlated with these observations. Moreover, the experiments are not very sufficient. ", "strength_and_weaknesses": "**Strength**:\n1. The observation of the abnormal adversarial examples is interesting. \n2. The experiments show their effectiveness on two widely-used CIFAR datasets.\n\n**Weaknesses**:\n1. In section 3, the authors analyzed the number and logit variation of normal and abnormal examples, respectively. Section 3.1 shows that the abnormal adversarial examples dominate the catastrophic overfitting since the number has a significant increase when catastrophic overfitting exists. In section 3.2, the authors found that the logit variations during the inner process will increase after the catastrophic overfitting. Both of these two observations are new and interesting. And the authors claimed that their proposed AAER is inspired by them. However, it's very difficult for me to understand how Eq.7 is derived from these two observations? \n1.1 From the second observation of logit variations, whether it is a normal sample or an abnormal sample, the change of logical distance shows an increasing trend, and the trend of distance change is also the same. Therefore, why does Eq.7 reduce the gap between these two type of adversarial examples? \n1.2 At the same time, Eq.7 is not very rigorous, because the $x_k$ in the second term may contain abnormal examples. \n1.3 The authors wanted to reduce the number of the abnormal examples. So they introduced a term $\\frac{n}{m}$ in Eq.8. But the authors did not analyze the mechanism of this design.\n\n2. The authors conducted experiments on CIFAR-10 and CIFAR-100. I don't think the experiments are enough. As we know, FGSM-based adversarial training is one of the most effective training schemes that can also train the model on a large-scale dataset. So it's better to evaluate the proposed method and analyze the properties of abnormal adversarial examples on a large dataset, like Tiny-ImageNet or ImageNet. \n\n3. Since the proposed AAER is to reduce the impact of abnormal adversarial examples. However, there were no corresponding experiments on this part in the experiment. For example, show the number curve after using AAER (as shown in Figure 3 (a)). Also, as training progresses, it is better to show the loss curve.\n\n4. On page 7, the authors said that they used the unclipped perturbation to achieve better robustness. However, this is not a commonly used technique. This may cause the pixel value of the adversarial sample to exceed the actual value. The authors need to clarify the reasons for using this technique.\n\n5. On page 8, what's the parameter $\\beta$? In previous section, I didn't find this hyper-parameter. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "f-9ZUSYt24", "reviewer": "Reviewer_DBF4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new training objective for single-step adversarial training to reduce catastrophic overfitting (CO). In particular, the paper establishes a correlation between CO and the presence of \"abnormal adversarial examples,\" i.e., instances where taking a single step of FGSM actually decreases the loss rather than increasing it. To this end, the authors propose a regularizer called AAER meant to penalize abnormal adversarial examples, and show that it often improves the accuracy of SSAT.", "summary_of_the_review": "In summary, the work proposes a new training objective for regularizing single-step adversarial training. Some of the motivation is a bit unclear and there is a natural baseline that the work does not explore. Overall, I think the paper as is is slightly below the bar for publication, but I am willing to raise my score.", "strength_and_weaknesses": "Strengths:\n\n- The observation that CO co-occurs with abnormal adversarial examples is both novel and interesting.\n- The attack seems efficient (e.g., not much slower than N-FGSM) while providing better adversarial accuracy.\n- The authors conduct an extensive evaluation of other methods and baselines.\n\nWeaknesses:\n\n- Given the link between CO and abnormal adversarial examples, a natural baseline would be to simply ignore any abnormal examples in the loss and train only on examples where the loss actually increased. Does this not work? If so, why?\n- The method does seem to improve on N-FGSM in terms of robust accuracy, but this often comes at the expense of clean accuracy, so it's unclear if the proposed method truly dominates N-FGSM. There is also some misleading bolding in the Table (i.e., for CIFAR-100 16/255 it doesn't look like the confidence intervals of the two bolded entries actually overlap) that should be fixed.\n- I am confused about the motivation behind Eq. 7---it seems like it is encouraging the network to classify the original and \"normal adversarial\" examples very differently (judging by the negative term)? Please correct me if I am wrong here, but it seems very counterintuitive.\n- Finally, there are many non-standard additions to the training routine (I'm thinking mainly of the linearly increasing $\\alpha$, as the other terms were ablated quite well). Does normal FGSM with linearly increasing alpha still suffer from CO?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "M6m51BpRra", "reviewer": "Reviewer_5TBo", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes Abnormal Adversarial Examples Regularization (AAER), a regularizer to address the known catastrophic overfitting issue observed in fast adversarial training (AT). AAER is based on suppressing the output difference of abnormal adversarial examples (irregular adversarial samples such that their loss is smaller after adding perturbation) and their number, as well as maximizing the difference of model output between unperturbed and perturbed samples. ", "summary_of_the_review": "1. Figure 3 is a very nice motivating example of the challenge and the proposed solution. However, it is unclear whether the emergence of abnormal examples is unique to the fast AT setting or not. To complete the analysis, I suggest running the same analysis on standard AT. If standard AT does not suffer from this issue but fast AT does, then this is a more convincing analysis.\n\n2. How critical is the second term in eq. (7)? In Line 15 of Algorithm 1, I noticed the authors use a zero threshold $\\max${$\\cdot$,0}. Does this suggest the second term in eq. (7), which contributes to negative numbers fo AAER, may not be important? Can the authors run an ablation study of the experiments with and without the second term?\n\n3. In most cases, the AAER_RUC method is the best regularizer. However, it's unclear to me what \"unclipped perturbation\" means in this context. Do the authors mean allowing the perturbed samples to go beyond the pixel values, or the authors did not clip the norm of the perturbation? Both cases make little sense to me, and the result needs more justification.\n\n4. Can AAER show complementary performance gain together with existing methods (e.g., gradient alignment) to mitigate catastrophic forgetting? For instance, any benefit of combining AAER with any of the existing methods in Table 1?\n", "strength_and_weaknesses": "Strength:\n1. The proposed method is simple to implement and has low overhead\n\nWeakness:\n1. The uniqueness of Fig. 3 to fast AT needs more justification\n2. The role of the second term in eq. (7) is unclear\n3. What is \"unclipped perturbation\"? Is it a rational regularizer? Why it gives better performance?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "A0oR-A5VZ_", "reviewer": "Reviewer_qRtw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "\nThis paper studies the robust overfitting of one-step adversarial training and finds that robust overfitting is related to the increase of abnormal adversarial examples generated for training. Abnormal adversarial examples have even lower losses after adding the adversarial perturbations, which can be a reason for causing robust overfitting in one-step adversarial training. Based on this observation, this paper proposes a novel regularization term to reduce the number of abnormal adversarial examples. By comparing the proposed methods with other baselines on CIFAR10 and CIFAR100 datasets, the evaluation results show that the proposed method achieves better robustness against the PGD attack.\n", "summary_of_the_review": "There are several points/places lacking clarity in the paper  (as discussed in the weakness). Further clarification and details may be necessary.\n\nReproducibility: The experiment settings section does not include all experiment details and no code is provided.\n", "strength_and_weaknesses": "Strength: \n+ This paper points out the existence of abnormal adversarial examples when robust overfitting happens and speculates that robust overfitting in single-step adversarial training is caused by abnormal adversarial examples. \n\n+ This paper proposed a novel regularization term to reduce the number of abnormal adversarial examples in training.\n\n+ The evaluation results show that the proposed method outperforms other baseline methods.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject