Eliminating Catastrophic Overfitting Via Abnormal Adversarial Examples Regularization
Authors: Runqi Lin
Chaojian Yu
Tongliang Liu
Venue: NeurIPS 2023 poster (2023)
Keywords: adversarial training
catastrophic overfitting

Abstract: Single-step adversarial training (SSAT) has demonstrated the potential to achieve both efficiency and robustness. However, SSAT suffers from catastrophic overfitting (CO), a phenomenon that leads to a severely distorted classifier, making it vulnerable to multi-step adversarial attacks. In this work, we observe that some adversarial examples generated on the SSAT-trained network exhibit anomalous behaviour, that is, although these training samples are generated by the inner maximization process, their associated loss decreases instead, which we named abnormal adversarial examples (AAEs). Upon further analysis, we discover a close relationship between AAEs and classifier distortion, as both the number and outputs of AAEs undergo a significant variation with the onset of CO. Given this observation, we re-examine the SSAT process and uncover that before the occurrence of CO, the classifier already displayed a slight distortion, indicated by the presence of few AAEs. Furthermore, the classifier directly optimizing these AAEs will accelerate its distortion, and correspondingly, the variation of AAEs will sharply increase as a result. In such a vicious circle, the classifier rapidly becomes highly distorted and manifests as CO within a few iterations. These observations motivate us to eliminate CO by hindering the generation of AAEs. Specifically, we design a novel method, termed Abnormal Adversarial Examples Regularization (AAER), which explicitly regularizes the variation of AAEs to hinder the classifier from becoming distorted. Extensive experiments demonstrate that our method can effectively eliminate CO and further boost adversarial robustness with negligible additional computational overhead. Our implementation can be found at https://github.com/tmllab/2023_NeurIPS_AAER.

Eliminating Catastrophic Overfitting Via Abnormal Adversarial Examples Regularization
Runqi Lin
Chaojian Yu Tongliang Liu"
Sydney Al Centre, The University of Sydney
(rlin0511, chyu8051, tongliang @sydney eeeeddeyy.ddd.e edu. au
Abstract
Single-step adversarial training (SSAT) has demonstrated the potential to achieve both efficiency and robustness. However, SSAT suffers from catastrophic overfit- ting (CO), phenomenon that leads to a severely distorted classifier, making it vulnerable to multi-step adversarial attacks. In this work, we observe that some adversarial examples generated on the SSAT-trained network exhibit anomalous behaviour, that is, although these training samples are generated by the inner maximization process, their associated loss decreases instead, which we named discover abnormal adversarial examples (AAEs). Upon further analysis, close relationship between AAEs and classifier distortion, as both the number and outputs of AAEs undergo significant variation with the onset of CO. Given this observation, we re-examine the SSAT process and uncover that before the occurrence of CO, the classifier already displayed slight distortion, indicated by the presence of few AAEs. Furthermore, the classifier directly optimizing these AAES will accelerate its distortion, and correspondingly, the variation of AAEs will sharply increase as a result. In such a vicious circle, the classifier rapidly becomes highly distorted and manifests CO within few iterations. These observations motivate us to eliminate CO by hindering the generation of AAEs. Specifically, we design a novel method, termed Abnormal Adversarial Examples Regularization (AAER), which explicitly regularizes the variation of AAEs to hinder the classifier from becoming distorted. Extensive experiments demonstrate that our method can effectively eliminate CO and further boost adversarial robustness with neg- ligible additional computational overhead. Our implementation can be found https: / / gi rithub... cww/tmmm//c/ccc http//mlllbbbbbe malmaabb22223 2ttp:/22222__221222
Introduction
In recent years, deep neural networks (DNNs) have demonstrated impressive performance in various decision-critical domains, such as autonomous driving [25, 12], face recognition [31, 3] and medical imaging diagnosis [8]. However, DNNs were found to be vulnerable to adversarial examples [35, 9, 16]. Although these adversarial perturbations are imperceptible to human eyes, they can lead to a completely different prediction in DNNs. To this end, many adversarial defence strategies have been proposed, such as pre-processing techniques [13], detection algorithms [27], verification and provable defence [20, 44], and adversarial training (AT) [11, 26, 46). Among them, AT is considered to be the most effective method against adversarial attacks [2, 4J.
Despite the notable progress in improving model robustness, the standard multi-step AT signifi- cantly increases the computational overhead due to the iterative steps of forward and backward propagation [26, 40, 41. 42, 48, 47]. In light of this, several works have attempted use single-step adversarial training (SSAT) as more efficient alternative to achieve robustness. Unfortunately.
"Coresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

serious problem catastrophic overfitting (CO) has been identified SSAT [39], manifesting as a sharp decline in the model's robust accuracy against multi- step adversarial attacks, plummeting from peak to nearly 0% within a few iterations, shown in Fig-
80%
RS-FGSM
70%
60%
Accuracy
50%
ure 1. This intriguing phenomenon has been widely investigated and prompted numerous efforts to re-
40%
solve it. Recently, Kim [21] pointed out that the SSAT-trained classifiers are typically accompanied by highly distorted decision boundaries, which will lead to the model manifestation CO. However, the underlying process of the classifier's gradual distor- tion, well as the factor inducing rapid distortion, remains unclear.
30%
20%
10%
0%
5
10
20
25
Epoch
Figure 1. The test accuracy of RS-FGSM [39] (red line) and RS-AAER (green line) with 16/255 noise magnitude. The dashed and solid lines denote natural and robust (PGD-7- 1) accuracy, respectively. The dashed black line corresponds to the 9th epoch, which is the point that RS-FGSM occurs CO.
In this study, we identify some adversarial exam- ples generated by the distorted classifier exhibiting anomalous behaviour, wherein the loss associated with them decreases despite being generated by the inner maximization process. We refer to these anoma- lous training samples as abnormal adversarial exam- ples (AAEs). Upon further investigation of the train- ing process, we observe that both the number and outputs of AAES undergo a significant variation during CO. This observation suggests strong correlation between the variation of AAEs and the gradually distorted classifier. By utilizing AAES as the indicator, we re-evaluate the process of SSAT and uncover that the classifier already exhibits slight distortions even before the onset of CO, which is evidenced by the presence of few AAEs. To make matters worse, directly optimizing the model based on these AAEs will further accelerate the distortion of the decision boundaries. Furthermore, in response to this more distorted classifier, the variation in AAE will dramatically increase as result. This interaction leads to a vicious circle between the variation of AAEs and the decision boundaries distortion, ultimately leading to the model rapidly manifesting as CO. All these atypical findings raise a question:
Can CO be prevented by hindering the generation of abnormal adversarial examples?
To answer the above question, design novel method, called Abnormal Adversarial Examples Regularization (AAER), which prevents CO by incorporating a regularizer term designed to suppress the generation of AAEs. Specifically, to achieve this objective, AAER consists of two components: the number and the outputs variation of AAEs. The first component identifies and counts the number of AAES in the training samples through anomalous loss decrease behaviour. The second component calculates the outputs variation of AAES by combining the prediction confidence and logits distribu- tion. Subsequently, AAER explicitly regularizes both the number and the outputs variation of AAFs to prevent the model from being distorted. It is worth noting that our method does not involve any extra example generation or backward propagation processes, making highly efficient in terms of computational overhead. Our major contributions are summarized as follows:
We identify particular behaviour in SSAT, in which some AAES generated by the distorted classifier have an opposite objective to the maximization process, and their number and outputs variation are highly correlated with the classifier distortion.
We discover that the classifier exhibits initial distortion before CO, manifesting as small number of AAEs. Besides, the model decision boundaries will be further exacerbated by directly optimizing the classifier on these AAEs,, leading to further increase in their number, which ultimately manifests as CO within a few iterations.
Based on the observed effect, we propose a novel method Abnormal Adversarial Examples Regularization (AAER). which explicitly regularizes the number and outputs variation of AAEs to hinder the classifier from becoming distorted. We evaluate the effectiveness of our method across different adversarial budgets, adversarial attacks, datasets and network architectures, showing that our proposed method can consistently prevent CO even with extreme adversaries and boost robustness with negligible additional computational overhead.

---

2 Related Work
2.1 Adversarial Training
DNNs are known to be vulnerable to adversarial attacks [35], and AT has been demonstrated to be the most effective defence method [2]. AT is generally formulated as min-max optimization problem [26, 4]. The inner maximization problem tries to generate the strongest adversarial examples maximize the loss, and the outer minimization problem tries to optimize the network to minimize the loss on adversarial examples, which can be formalized as follows:
min (a. max E(x + + BEA
where (z, y) is the training dataset from the distribution D, (z, yı 0) is the loss function parameterized by 0, is the perturbation confined within the boundary shown as: A = {ó: ||8|| S c}.
2.2 Catastrophic Overfitting
Since the intriguing phenomenon of CO was identified [39], there has been line of work trying to explore and mitigate this problem. [39] first suggested using a random initialization and early stopping avoid CO. Furthermore, [36] empirically showed that using dynamic dropout schedule can avoid early overfitting to adversarial examples, and [6, 45] found that incorporating a stronger data augmentation is effective in avoiding CO. Another alternative approach imports partial multi-step AT, for example, [38] periodically trained the model on natural, single-step and multi-step adversarial examples, and [37] built regularization term by comparing with the multi-step adversarial examples.
However, the above methods have not provided deeper insight into the essence of CO. Separate works found that CO is closely related to anomalous gradient updates. [24] constrained the training samples to a carefully extracted subspace to avoid abrupt gradient growth. [10] ignored the small gradient adversarial perturbations mitigate substantial weight updates in the network. [1.5] proposed an instance-adaptive SSAT approach where the perturbation size is inversely proportional to the gradient. [29] leveraged the latent representation of gradients as the adversarial perturbation to compensate for local linearity. [34] introduced a relaxation term to find more suitable gradient directions by smoothing the loss surface. [1] proposed regularization term avoid the non- linear surfaces around the samples. More recently, [21] introduced a new perspective that CO is a manifestation of highly distorted decision boundaries. Accordingly, they proposed reduce the perturbation size for the already misclassified adversarial examples.
Unfortunately, the aforementioned methods tend to either suffer from CO with strong adversaries or significantly increase the computational overhead. In this work, we delve into the interaction between AAES and distorted decision boundaries, revealing close relationship between them. Based on this insight, we propose novel approach, AAER, that eliminates CO by explicitly hindering the generation of AAEs, thereby achieving both efficiency and robustness.
3 Methodology
In this section, we first define the abnormal adversarial example (AAE) and show how their numbers change throughout the training process (Section 3.1). We further compare the outputs variation of normal adversarial examples (NAEs) and AAES and find that their outputs exhibit significantly different behaviour after CO (Section 3.2). Building upon our observations, we propose novel regularization term, Abnormal Adversarial Examples Regularization (AAER), that uses the number and outputs variation AAEs to explicitly suppress the generation of them eliminate catastrophic overfitting (CO) (Section 3.3).
3.1 Definition and Counting of AAE
Adversarial training employs the most adversarial data to reduce the sensitivity of the network's outputs w.r.t. adversarial perturbation of the natural data. Consequently, the inner maximization process expected to generate effective adversarial examples that maximize the classification loss. As empirically demonstrated by [21], the decision boundaries of the classifier become highly distorted
3

---

Chos
Cluss
Noenal advensral euumple
Abnomaal ahveavrd stampl
Uecisin borndayy
Adversay diection
Before optimizing AABS
After optimizing AAEs
Figure 2. A conceptual diagram of the classifier's decision boundary and training samples. The training samples belonging to NAE (blue) can effectively mislead the classifier, while AAE (red) cannot. The left panel shows the decision boundary before optimizing AAEs, which only has a slight distortion. The middle panel shows the decision boundary after optimizing AAEs, which exacerbates the distortion and generates more AAEs.
after the occurrence of CO. In this study, we find that after adding the adversarial perturbation generated by the distorted classifier, the loss of certain training samples unexpectedly decreases. This particular behaviour is illustrated in Figure 2, we observe that the NAEs (blue) can either lead to the model misclassifications or position themselves closer to the decision boundary after the inner maximization process. contrast, the AAEs (red) will be located further away from the decision boundary and fail to mislead the classifier after adding the perturbation generated by the distorted classifier. Therefore, we introduce the following formula define AAEs:
8 == sign (V V2tnb(0 + 13, 2, 0)),
AAE del (x + 11, 1; 8 (x
0)
where 7) is the random initialization.
Next, we observe the variation in the number AAES throughout the model training process, and the corresponding statistical results are presented in Figure (left). It can be observed that before the occurrence of CO, a small number of AAES already existed, indicating the presence of slight initial distortion in the classifier. To further validate this point, we visualize the loss surface of both AAEs and NAEs using the method proposed by [23] as shown in Figure (left and middle). It's evident that before CO, the classifier showcases more nonlinear loss surface around AAEs in comparison to NAEs. This empirical observation strongly suggests that generation AAES directly influenced by the distorted classifier.
Besides, the number of AAEs experiences a dramatic surge during CO occurrences. For example, the number of AAEs (red line) exploded 19 times at the onset of CO (9th epoch), as shown in Figure (left). Importantly, this rapid increase in the AAES number implies a continuous deterioration in the classifier's s boundaries, which in turn leads to a further increase in their number. The number of AAES reaches its peak at the 10th epoch, surging to approximately 66 times than that before CO. To meticulously analyze the interaction between AAEs and CO, we delve into their relationship at the iteration level, as shown Figure (right). We can observe that the robustness accuracy from peak sharply drops to nearly 0% within 18 iterations, simultaneously. the number of AAEs rises from 0 to 70. Remarkably, the trends in robustness accuracy and the number of AAEs display a completely opposite relationship, suggesting vicious cycle between optimizing AAEs and CO. Lastly, the number of AAEs consistently maintains a high level until the end of the training. Given this empirical and statistical observation, and the CO phenomenon, which also prompts us to wonder (QI): whether CO can be mitigated by reducing the number of abnormal adversarial examples.
infer that there is close correlation between the number of AAEs
3.2 Outputs Variation of NAE and AAE
The above observations indicate the close relationship between CO and AAEs. In this part, we further analyze the outputs variation of AAEs during CO. Specifically, we discover that CO has

---

50K
0.5
Prediction Confidence Variation
Logits Distributton Vartation
50
40K
40
Number
-0.3
AAE
-30
Training
20
-1.0
10K
AAE NAE
A4E NAE
-1.5
Sample
Training Sample
Epoch
Epoch
The number, the variation of prediction confidence and logits distribution (from left to right) for NAEs, AAES and training samples in RS-FGSM with 16/255 noise magnitude. The dashed black line corresponds the 9th epoch, which is the point that the model occurs CO.
Figure 3.
sigmificant impact on both the prediction confidence and the logits distribution of AAEs. To quantify the variation in prediction confidence, we utilize the cross-entropy (CE) calculate the change loss during the inner maximization process, which is formulated as follows:
i (x + y: 0) & (x + 7). yr 0).
We investigate the prediction confidence of NAEs and AAEs variation during the model training. From Figure (middle), we can observe that the change in prediction confidence of NAEs is consistently greater than O, indicating that their lead worse prediction in the classifier. On the contrary, this variation in AAES is atypical negative implying that the associated adversarial perturbation has an unexpected opposite effect. Moreover, delve into the impact of CO the variation in prediction confidence. Before CO, we note slight negative variation in the AAEs' prediction confidence, which has an insignificant impact all training samples (blue line). However, during CO, the prediction confidence of AAEs undergoes rapid and substantial drop, reaching a decline of 17 times at the 9th epoch. After CO, the prediction confidence of AAES 43 times (10th epoch) smaller than before and significantly impacts all training samples.
In addition the inability mislead the classifier, the logits distribution of AAE is also disturbed during the CO process. To analyze the variation logits distribution, we employ the Fuclidean (I.2) distance to quantify the impact of adversarial perturbation, which is formulated follows:
fo (x + 5) - fo (x + 17) l2.
where is the DNN classifier parameterized by and ||- 113 the L2 distance.
The logits distribution variation of both NAEs and AAEs are illustrated in Figure 3 (right). Comparing the logits distribution variation between NAEs and AAEs. we can find that their magnitudes are similar before CO. However, it becomes evident that the logits distribution variation of AAEs increases dramatically during CO, being 13 times larger than before. After further optimization on AAEs, the variation in logits distribution reaches the peak, approximately 62 times larger than before. This observation highlights that even a small adversarial perturbation can cause substantial variation in the logits distribution, this phenomenon typically happens the highly distorted decision boundaries. Additionally, it's worth noting that the increase logits distribution variation for NAEs (green line) occurs one epoch later than that of AAEs, indicating that the primary cause of decision boundary distortion lies within the AAEs. In other words, directly optimizing the network using these AAEs exacerbates the distortion of decision boundaries, resulting in significant change in the logits distribution for NAEs. Even after CO, the logits distribution variance AAEs remains twice as large NAEs. The significant difference between NAEs and AAEs in the variation of prediction confidence and logits distribution inspires us to wonder (Q2): whether CO can be mitigated by constraining the outputs variation abnormal adversarial examples.
3.3 Abnormal Adversarial Examples Regularization
Recognizing the strong correlation between CO and AAEs, we first attempt passive approach by removing AAEs and training solely on NAEs. This simple approach demonstrates the capability
5

---

1yy
232
22
210
2:00
231
2.06
20
3.50
32
30
60h
10
30 10
10
35 10 000
230
Figure 4. Left/Middle panel: The visualization of AAEs/NAFS loss surface before CO (8th epoch). Right panel: The number of AAES and the test robustness within each iteration at CO (9th epoch). The green and red lines represent the robust accuracy and number of AAEs, respectively.
to delay the onset of CO, thereby confirming that direct optimization of AAEs will accelerate the classifier's distortion. However, it is important to note that the generation of AAEs is caused by the distorted classifier. Passively removing AAEs cannot provide the necessary constraints to promote smoother classifiers, thereby only delaying the onset of CO but not preventing it.
To truly relieve this problem, we design a novel regularization term, Abnormal Adversarial Examples Regularization (AAER), which aims hinder the classifier from becoming distorted by explicitly reducing the number and constraining the outputs variation of AAEs. Specifically, part (i) categorizes the training samples into NAEs and AAES according to the definition in Eq. 2, and then penalizes the number of AAEs. The AABs' outputs variation is simultaneously constrained by part (ii) prediction confidence and part (iii) logits distribution. In terms of prediction confidence, we penalize the anomalous variation in AAEs that should not be negative during the inner maximization process, which is formalized as follows:
AAE AAE_CE CE 1
Σ (e(xAAF + + 77 0) - (z.1 + + 8,;+,);0))
(5)
where is the number of abnormal adversarial examples. For logits distribution, we first calculate the logits distribution variation of AAEs and NAEs separately, as shown in Eq. (6) and Eq. (7):
AAE L2
(1 (I AAE + 5) fo (x; AAE + +1) 12):
(6)
[m1
NAE
(Ilfo (I + ff (INAE + 112)
m
j=1
where m is the number of training samples.
Then, we use the logits distribution variation of NAEs as reference to constrain the variation in AAEs. It's essential emphasize that our optimization objective make the logits distribution variation of AAES closer to that of NAEs, rather than less. To achieve this, we use the max function limit the minimum value, which is formalized as follows:
Constrained V Vriaaionn = max (AAE 1.2 (6) N2
12 (7),0),
(8)
where mar()) (,) is the max function.
Although Figure (right) illustrates that the logits distribution variation of NAEs will significantly increase and instability after CO. However, that is a natural consequence of the highly distorted classifier which disrupted the logits distribution of NAEs. In contrast, after using the AAER to hinder the classifier from becoming distorted, the NAEs can be used as a stable standard throughout the training, shown in Figure 5 (a:right).
Based on the optimization objectives described above, we can build a novel regularization term AAER, which aims to suppress the AAEs by the number, the variation of prediction confidence

---

Algorithm Abnormal Adversarial Examples Regularization (AAER)
Input: Network Jo. epochs T. mini-batch M. perturbation radius step size initialization term Output: Adversarially robust model
1: for = ........ do
2: for k ....MM do
3:
8 = a. -signn (V 2+ (ax + 17, yk: 0))
4:
CE = 1 Lk & (Ix- + + 6, yk:
5:
AAER Eq. (9)
6:
0 = Vo (CE + AAER
7:
end for
8:
end for
and logits distribution, ultimately achieving the purpose of preventing CO, which is shown in the following formula:
71 AAER . "..)) AAE_CE (5) + X3 Comstruined Variation Variaaaion (8)),
m
where Å1, A2 and λ are the hyperparameters to control the strength of the regularization term.
AAER can effectively hinder the generation of AAEs that are highly correlated with the distorted classifier and CO, thereby encouraging training for smoother classifier that can effectively defend against adversarial attacks. By considering both the number and output variation of AAEs, we establish more adaptable and comprehensive measure of classifier distortion. Importantly, our method does not require any additional generation or backward propagation processes, making it highly convenient in terms computational overhead. The proposed algorithm AAER realization summarized in Algorithm 1.
4 Experiment
In this section, we provide a comprehensive evaluation to verify the effectiveness of AAER, including experiment settings (Section 4.1), performance evaluation (Section 4.2), ablation studies (Section 4.3)
and time complexity study (Section 4.4).
4.1 Experiment Settings
method with other SSAT methods, including RS-FGSM [39], FreeAT [30], N-FGSM [6], Grad Align [1], ZeroGrad and MultiGrad [10]. We also compare method with multi-step AT, PGD-2 and PGD-10 (PGD-22 with 32/255 noise magnitude) [26), providing reference for the ideal performance. The results of other competing baselines, including GAT [33], NuAT [34], PGI-FGSM [17]. SDI-FGSM [18] and Kim [21], can be found in Appendix We report both the natural and robust accuracy results of the final model, which are obtained without early stopping and using the hyperparameters provided in the official repository. Please note that for AT, we did not use the subset of training samples to keep the same training epochs across different methods.
Baselines. We compare
Attack Methods. To report the robust accuracy of models, we attack these methods using the standard PGD adversarial attack with a e/4 step size, 50 attack steps and 10 restarts. We also evaluate our methods on Auto Attack [5] as shown in Appendix C.
Datasets and Model Architectures. We evaluate our method several benchmark datasets, includ- ing Cifar-10/100 10/100 [22], SVHN [28], Tiny-ImageNet [28] and Imagenet 100 [7]. The standard data augmentation random cropping and horizontal flipping are applied for these datasets. The settings and results on SVHN, Tiny-ImageNet and Imagenet-100 100 are provided Appendix E. We use the PreactResNet- 18 [14] and WedeResNet-34 [43] architectures on these datasets to evaluate results. The results of WideResNet-344 can be found in Appendix D.
Setup for Our Proposed Method. In this work, we use the SGD optimizer with momentum of 0.9, weight decay of 5 10-4 and Tope as the threat model. For the learning rate schedule, we use the cyclical learning rate schedule [32] with 30 epochs, which reaches its maximum learning rate
7

---

Table 1. The hyperparameter settings for Cifar-10/100 are divided by slash. The top number represents A2 while the bottom number represents A3- Throughout all settings, X1 is fixed at 1.0.
RS-AAER
N-AAER
CIFARI0/100
8/255
12/255
16/255
32/355
8/255
12/255
16/255
32/355
A2
2.5/3.5
5.5
7.0/6.0
5.75
1.5/
5.0/
8.5
2.75
Às
1.515
2.75
2.75
3.25/2.25
0.77
0.15
0.55
0..5/0..
1.5 0.5
0.75/0.5
Table 2. CIFARIO/100: Accuracy of different methods and different noise magnitudes using PreActResNet-1 18 under Los. threat model. The top number the natural accuracy (%), while the bottom number is the PGD-50-10 accuracy (%). The results are averaged over 3 random seeds and reported with the standard deviation.
dataset
CITARIO
COFAR100
moise magnitude
8/255
12/255
16/255
32/255
8/255
12/255
16/255
32/255
76.30 e: 1.09
68.07 0.38
H 19.07
61.11 8.41
47.41 0.30
0.40
3.30 2.48
26.3
FreeAT
43.74 = 0.41
33.14 s 0.62
+ 0,00
0.000
22.27 0.33
16.57
81,60 0.16
77.52 + 0.21
79.65
65.48 1.48+.6... 6.26
53.83 0.22
49.077 0.14
51.76 + 0.02
49.38 + 1.39
ZaroCia
0.16
27.24 A 0.09
6.37 0.22
0.00 0.00
25.02 0.24
14.76 0.26
5.23 0.09
0,00 = 0.00
81.65
81.09 4.67
3.30
53.11 0.34
46.81 0.51
46.05
28.33 0.48
MuitKirad
47,93 0.18
25.68
16.56
82,10 =: 0.78
14.17
60.37 0.95
25.23 3.41
54.00=0.44
45.83
36,50 = 0.10
15.05 0:07
Grad Align
0.557:00.58
$4.87 1.00
27.90 1.01
31233
25.27 0.68
0.71
13.77 : 0,76
2,85 = 1.34
83.91
66.46 22.800
36,43
1.51
=8.1
11.00
11,40
RS-FGSM
46.01
80.48 0.21
21.20
62.96 0.74
29.79 3:87
54.92
0.116
31.93 =: 0.22
18.18 1.188:44.3 4.55
NGSM
47.91 0.29
36.33 0.10
23.14 1.44
8.30 + 7.85
26.20 0.41
18.75 0.19
14,05 + 0.07
0.00 - 0.00
85.33 0.27
74.40 0.79
57.71 0:29
0.93 16.41
18.50 1.68
RS AS-A
4..14:00.02
32.17
0.36
10.62
25.31 0.01
= 0.17
4,90
80.56 0.35
71.15 0.18
61.84
27.08 a 0.02
54.47 0.45
45.98 0.13
36.80 2:
16.95 0.44
N-AAFR
48.31 = 0.23
36.52 a 0.10
28.20 0.71 0.71
12.97 0.57
26.81 0.13
19.03 a 0.04
14.31 e 0.05
5.45 e 0.14
85.00
78.97
72.31
0.71
0.20
53.46 0.27
47.50
0.69
DGD-2
45.27 0.07
32.99 0.46
24.32
11.24
1721
80.55 0.37
72:37 à: 0.31
67.20 2000::0.00
24.70
55.05 0.25
47.42 2 0.29
42.39 0.17
21.68 0.18
(20)
50.67 a 0.40
38.60 0.39
29.34 0.18
16.10 0.20
27.87 0.12
20.29 0.18
15.01 = 0.21
7,39 = 0.38
(0.2) when half of the epochs (15) are passed. The results obtained by the long training schedule can be found in Appendix G. We superimpose our method on two baseline methods: RS-FGSM and N-FGSM, both of which use the vanilla min-max process. For RS-AAFR we follow the settings accordance with the N-FGSM setting suggested by [6]. we set a 1.0 and v) = Uniform( (-2 r, .26 for N-AAER. The hyperparameter settings for Cifar-10/100 are summarized in the Table 1.
of [39] that set step size a 1.25- and random initialization v) = Uniform(-e,6
4.2 Performance Evaluation
CIFARIO Results. In Table 2 (left), we present a comparison of our proposed methods with the competing basclines. First, we can observe that CO occurs in all baseline methods except for Grad Align. However, Grad Align requires double backward propagation which notably reduces its efficiency. Compared to them, RS-AAER and N-AAER can effectively eliminate CO with all noise magnitudes and only incur negligible computational overhead. objective is to eliminate CO in SSAT, thus AAER will show significant performance enhancements under CO scenarios, exemplified by RS-AAER under 12, and 32 noise magnitude, and N-AAER under 32 noise magnitude. Even without the CO scenario, our approach can consistently outperform the baselines under all experimental conditions. Additionally, we implemented Vanilla-AAFR in Appendix A to further demonstrate the effectiveness of our method. For a fair comparison, also
worth noting that our primary
8

---

50C
1.8
prediction Confidence Variatian
Logits Distribution Variation
1.6
Training Semple
40C
0.3
AAE
A4E
Training Sample
0.2
Trining Sample
0.8
0.
0.0
10C
0.4
0
25
25
Epoch
Figure 5. The number, the variation of prediction confidence and logits distribution (from left to right) for NAEs, AAEs and training samples in RS-AAER with 16/255 noise magnitude.
attempt to use the N-FGSM augmentation on Grad Align. However, we could not find suitable hyperparameter settings for Grad Align with the original step size. After reducing the step size to = 1.0 (consistent with N-AAER), the performance of Grad Align is similar to that reported in Table (left). Besides, AAEs are not unique phenomenon SSAT, also found them in the multi-step AT, indicating the existence of non-smooth points in their classifiers. However, we could find competitive hyperparameter settings enhance the robustness of multi-step AT using AAER.
CIFAR100 Results. We also conduct experiments on the CI- FAR100 dataset, and the results summarized in Table 2 (right). It is worth noting that CIFAR 100 is more challenging the number of classes/training images per class is ten times larger/smaller than that of CIFARIO. 10. However, our proposed methods demonstrate their effectiveness in preventing CO and improving robust accuracy on CIFAR- 100 as well. These results further validate that AAER capable reliably preventing CO and effectively improving robustness across different datasets.
70%
60%
Accuracy
30%
Natural
Rebust
40%
30%
20%
4.3
Ablation Studies
In this part, investigate the impacts of RS-AAER compo- nents with 16/255 noise magnitude using PreactResNet-188 on CIFARIO under Lox threat model.
Figure The role of
validate the effectiveness our proposed method, we illustrate the variation in test accuracy and three optimization objectives during the training process. Fig- ure demonstrates that our approach can successfully prevent CO and continuously improves robustness throughout the training. Additionally, from Figure 5, ber, the variation of prediction confidence and logits distribution for AAES are effectively con- strained during the training. Specifically, when CO occurs in RS-FGSM (9th epoch), these three optimization objectives in RS-AAER are 29, 14, and 24 times smaller than it, respectively.
Optimization Objectives.
observe that the num-
Table The impact of regularization term.
The Role of a. From Figure 6, we can ob- serve that enlarging the size will lead to an increase in the robust accuracy, and corre- spondingly decrease in the natural accuracy.
(
(ii)
Natural Acc (%)
Robust Ace (%)
75.14
In light of this trade-off, we follow the base- line setting and do not adjust the size.
77.22
0.00
13.61
The impact Regularization Term. We further investigate the interaction between the three parts of our regularization term as shown in Table 3. We find that neither part (i) num- ber nor part (ii) prediction confidence can in- dependently prevent CO, and only using part
76.19
0.00
56.65
23.29
16.68
12.56
64.56
23.87

---

Table 4. CIFARIO training time single NVIDIA RTX 4090 GPU using PreactResNet--Be 18. The results are averaged over 30 epochs.
Method
PreeATl
ZeroCirad
MultiGrad
Grad Align
RS/N-FOSM
RS/N-AAER
PGD-2
PGD-10
Training Time (S)
43.8
11.0
21.7
36.1
11.0
11.2
16.4
59.1
(iii) logits distribution can partially mitigate CO. However, solely relying on part (iii) cannot ac- curately reflect the degree of classifier distortion as it lacks a comprehensive measure, resulting in poor performance. Additionally, part (i) also plays an important role in performance, without it, both natural and robust accuracy significantly drop. Meanwhile, part (ii) contributes to the stability and natural accuracy of the method. Therefore, to effectively and stably eliminate CO, all parts of the regularization term necessary and critical. Further ablation studies on other regularization methods and À selection can be found in Appendix B.
4.4 Time Complexity Study
Efficiency is key advantage of SSAT over multi-step AT, it can be readily scaled to large networks and datasets. Consequently, the computational overhead plays an important role the SSAT overall performance. In Table can be seen that AAER only imposes minor training cost of 0.2 seconds, representing a mere 1.8% increase compared to FGSM. In contrast, Grad Align and PGD-10 are 3.2 and 5.3 times slower than our method.
present time complexity comparison among various SSAT methods. It
Conclusion
In this paper, we find that the abnormal adversarial examples exhibit anomalous behaviour, i.c. they are further to the decision boundaries after adding perturbations generated by the inner maximization process. We empirically show the abnormal adversarial examples are closely related the classifier distortion and catastrophic overfitting, by analyzing their number and outputs variation during the training process. Motivated by this, we propose novel and effective method, Abnormal Adversarial Examples Regularization (AAER), through regularizer to eliminate catastrophic overfitting by suppressing the generation abnormal adversarial examples. Our approach can successfully resolve the catastrophic overfitting with different noise magnitudes and further boost adversarial robustness with negligible additional computational overhead.
Acknowledgments and Disclosure of Funding
Tongliang Liu is partially supported by the following Australian Research Council projects: FT220100318, DP220102121, LP220100527, LP220200949, and IC190100031.
10

---

References
[1] Maksym Andriushcheenko and Nicolas Flammarion. Understanding and improving fast adversarial training. Advances in Neural Information Processing Systems, 33: (0040-16059) 2020.
[2] Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses adversarial examples. International conference on machine learning, pages PMLR, 2018.
[3] Stephen Balaban. Deep learning and face recognition: the state of the art. Biometric and surveillance technology for human and activity identification XII, 9457:68- 70 2015.
[4] Francesco Croce, Sven Gowal, Thomas Brunner, Evan Shelhamer, Matthias Hein, and Taylan Cemgil. Evaluating the adversarial robustness of adaptive test-time defenses. In International Conference on Machine Learning, pages 4421-4435. PMLR, 2022.
[5] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-fiee frreeeerer-eeeeeeeeeee attacks. In International conference on machine learning, pages 2206 2216. PMLR, 2020.
[6] Pau de Jorge, Adel Bibi, Riccardo Volpi, Amartya Sanyal, Philip HS Torr, Grégory Rogez, and Punect K Dokania. Make some noise: Reliable and efficient single-step adversarial training. arXiv preprint arXiv:2202.01181, 2022.
[7] Jia Deng. Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Iece, 2009.
[8] Bradley Erickson, Panagiotis Korfiatis. Zeynettin Akkus, and Timothy Kline. Machine learning for medical imaging. Radiographics, 37(2):505-511 515,, 2017.
Francisco Erivaldo Fernandes and Gary G Yen. Automatic searching and pruning deep neural net- works for medical imaging diagnostic. IEEE Transactions on Neural Networks and Learning Systems, 32(12):5664-55 -67464 2020.
[10] Zeinab Golgooni, Mehrdad Saberi, Masih Eskandar, and Mohammad Hossein Rohban. Zerograd: Mitigat- ing and explaining catastrophic overfitting in fgsm adversarial training. arXiv preprint ar Xiv: 103. 15476, 2021.
[11] lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and hamessing adversarial examples. arXiv preprint arXiv: 1412.652,, 2014.
[12] Sorin Grigorescu, Bogdan Trasnea, Tiberiu Cocias, and Gigel Macesanu. A survey of deep learning techniques for autonomous driving. Journal of Field Robotics, 37(3):362-386 2020.
[13] Chuan Guo, Mayank Rana. Moustapha Cisse, and Laurens Van Der Maaten. Countering adversarial images using input transformations. arXiv preprint arXiv:1711.00117, 2017.
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pages 630 645. Springer, 2016.
[15] Zhichao Huang, Yanbo Fan, Chen Liu, Weizhong Zhang, Yong Zhang, Mathieu Salzmann, Sabine Süsstrunk, and Jue Wang. Fast adversarial training with adaptive step size. arXiv preprint 2rX11:2206.02417. 2022.
[16] Zhuo Huang, Miaoxi Zhu, Xiaobo Xia. Li Shen, Jun Yu. Chen Gong, Bo Han, Bo Du, and Tongliang Liu. Robust generalization against photon- -imited corruptions via worst-case sharpness minimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16175- 16185, 2023.
[17] Xiaojun Jia, Yong Zhang, Xingxing Wei, Baoyuan Wu, Ke Ma, Jue Wang, and Xiaochun Cao. Prior-guided adversarial initialization for fast adversarial training. In European Conference on Computer Vision, pages 567-584. Springer, 2022.
[18] Xiaojun Jia, Yong Zhang, Baoyuan Wu, Jue Wang, and Xiaochun Cao. Boosting fast adversarial training with learnable adversarial initialization. IEEE Transactions on Image Processing, 31:4417 4430, 2022.
[19] Harini Kannan, Alexey Kurakin, and Ian Goodfellow. Adversarial logit pairing. 06373, 2018
arXiv preprint

---

[20] Guy Katz. Clark Barrett, David L Dill. Kyle Julian, and Mykel Kochenderfer. Reluplex: An efficient smt solver for verifying deep neural networks. In International conference on computer aided verification. pages 97-117. Springer, 2017.
[21] Hoki Kim, Woojin Lee, and Jaewook Lee. Understanding catastrophic overfitting in single-step adversarial training. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 819--8127 2021.
[22] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[23] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances in neural information processing systems, 31. 2018.
[24] Tao Li, Yingwen Wu, Sizhe Chen, Kun Fang, and Xiaolin Huang. Subspace adversarial training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13409- 13418, 2022.
[25] Todd Litman. Autonomous vehicle implementation predictions. Victoria Transport Policy Institute Victoria, BC, Canada, 2017.
[26] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:/706.06600 06083, 2017.
[27] Jan Hendrik Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff. On detecting adversarial perturbations. arXiv preprint arXiv:1702.0426; 2017.
[28] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[29] Geon Yeong Park and Sang Wan Lee. Reliahly fast adversarial training via latent adversarial perturbation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 7758-7767, 2021.
[30] Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! Advances in Neural Information Processing Systems, 32, 2019.
[31] Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K Reiter. Accessorize to crime: Real and stealthy attacks on state-of-the-an face recognition. In Proceedings of the 2016 acm sigsac conference on computer and communications security, pages 1528-1540, 2016.
[32] Leslie N Smith. Cyclical learning rates for training neural networks. In 2017 IEEE winter conference on applications of computer vision (WACV), pages 464-472. IEEE, 2017.
[33] Gaurang Sriramanan, Sravanti Addepalli, Arya Baburaj, al. Guided adversarial attack for evaluating and enhancing ad adversarial defenses. Advances in Neural Information Processing Systems, 33:20297-20308. 2020.
[34] Gaurang Sriramanan, Sravanti Addepalli, Arya Baburaj, et al. Towards efficient and effective adversarial training. Advances in Neural Information Processing Systems, 34:1 1821-11833, 2021.
[35] Christian Szegedy, Wojeiech Zaremba, llya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. urXiv preprint arXiv:1312.6199 2013.
[36] BS Vivek and R Venkatesh Babu. Single-step adversarial training with dropout scheduling. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 947- 147-956. IEEE, 2020.
[37] BS Vivek, Ambareesh Revanur, Naveen Venkat, and R Venkatesh Babu. Plug-and-pipeling Efficient regularization for single-step adversarial training. 2020 JEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 136-146. IEEE, 2020.
[38] Xiaosen Wang, Chuanbiao Song, Liwei Wang, and Kun He. Multi-stage optimization based adversarial training. arXiv preprint arXiv:2 15357, 2021.
[39] Eric Wong, Leslie Rice, and J Zico Kolter: Fast is better than free: Revisiting adversarial training. arXiv preprint arXiv:2001. 03994, 2020.
[40] Dongxian Wu, Shu- Tao Xia, and Yisen Wang, Adversarial weight perturbation helps robust generalization. Advances in Neural Information Processing Systems, 33:2958 2952229889, 2020.
12

---

[41] Chaojian Yu, Bo Han, Mingming Gong, Li Shen, Shiming Ge, Bo Du, and Tongliang Liu. Robust weight perturbation for adversarial training. arXiv preprint arXiv:2205. 14826, 2022.
[42] Chaojian Yu, Bo Han, Li Shen, Jun Yu, Chen Gong, Mingming Gong, and Tongliang Liu. Understanding robust overfitting of adversarial training and beyond. In International Conference Machine Learning. PMLR, 2022.
pages
[43] Sergey Zagoruyko and Nikos Komodakis, Wide residual networks. arXiv preprint 16X1v:1605.07146, 2016.
[44] Bohang Zhang. Du Jiang, Di He, and Liwei Wang. Rethinking lipschitz neural networks and certified robustness: boolean function Aeispective.Advaneess Advances Neural Information Processing Syxtems, 35:19398- 19413, 2022.
[45] Chaoning Zhang. Kang Zhang, Axi Niu, Chenshuang Zhang. Feng, Chang D Yoo, and In Kweon. Noise augmentation is all you need for fgsm fast adversarial training: Catastrophic overfitting and robust overfitting require different augmentation. arXiv preprint arXiv:2202.054485 2022.
[46] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent Ghaoui, and Michael Jordan. Theo- retically principled trade-ofi off between robustness and accuracy. In International conference on machine learning, pages 7472 7482. PMLR, 2019.
[47] Dawei Zhou, Nannan Wang. Xinbo Gao, Bo Han, Xiaoyu Wang, Yibing Zhan, and Tongliang Liu. Improving adversarial robustness via mutual information estimation. In International Conference on Machine Learning, pages 37338-27352. PMLR, 2022.
[48] Dawei Zhou, Nannan Wang, Heng Yang. Xinbo Gao, and Tongliang Liu. thase-aware adversarial defense improving adversarial robustness. 2023.
13

---

A Vanilla-AEER
To further validate the effectiveness of our method, we implement Vanilla-AAER to prevent CO. The Vanilla-AAER method follows the settings of Vanilla-FGSM [11], which does not use random initialization and sets the step size as a = 1.0 e. The Vanilla-AAER hyperparameters setting is shown in Table 5.
Table The hyperparameters setting for different noise magnitudes. The top number A2 while the bottom number is X3. The λ is fixed as 1.0 in all settings.
Vanilla-AAER
8/255
12/255
16/255
32/255
A2
5.5
6.5
7.0
4.8
A
2.0
3.5
3.5
0.7
Table CIFARIO: Accuracy of Vanilla-FGSM and Vanilla-AAER with different noise magnitude using PreAcctReNett-Il 18 under Loc threat model. The top number is the natural accuracy (%), while the bottom number is the PGD-50-10 accuracy (%).
noise magnitude
8/255
12/255
16/255
32/255
84.16 4.68
79.86 + 2.05
72.51 + 3.79
64.29 + 3.83
Vanilla-FGSM
0.00 = 0.00
0.00 = 0.00
0.00 = 0.00
0.00 + 0.00
80.45 0.25
64.97 + 3.14
51.92 # 2.90
18.78 = 2.45
Vanilla-AAER
46.66 + 0.74
32.44 # 1.18
24.12 = 0.76
12.19 + 0.40
Based on the results presented in Table 6, we can observe that Vanilla-AAER achieves comparable or even superior robustness compared to RS-AAER. This outcome may be attributed to the fact that Vanilla-FGSM has higher expectation of adversarial perturbation, demonstrated in prior work [1]. However, the absence random initialization in Vanilla-AAER may reduce the diversity of adversarial examples, potentially impacting the natural accuracy of the model. Nonetheless, the most significant finding is that Vanilla-AAER effectively eliminates CO across various noise magnitudes, which cannot be accomplished by Vanilla-FGSM.
B Impacts of Regularization Term
To showcase the distinct effectiveness of our method in eliminating CO, conducted a comparison with other regularization methods used in multi-step AT, such as TRADES [46] and ALP [19]. In order to ensure a fair comparison, we set the iteration times for TRADES and ALP as 1, and the step size as a = 1.25 c and a = 1.0 c for superimposing RS-FGSM and N-FGSM, respectively.
From Table 7, we can observe that TRADES and ALP methods may improve adversarial robustness when CO not present in the baseline methods. However, these methods are not effective in eliminating CO. As demonstrated by the RS-TRADES and RS-ALP methods, CO still occurs with larger noise magnitudes, similar to RS-FGSM. Furthermore, these methods can even harm baseline method, as they break the N-FGSM robustness with 32/255 noise magnitude. Therefore, we conclude that the TRADES and ALP methods are not suitable for eliminating CO. Additionally, other multi-step and robust overfitting methods also prove ineffective against CO. Hence, CO has been identified as an independent phenomenon requiring distinct solutions. In contrast to these regularization methods, AAER explicitly reflects and prevents distortion of the classifier from the perspective of AAEs, which the key factor enabling AAER to effectively eliminate CO.
We further investigate the impact of hyperparameters λ1. X2, and A on the performance of AAER. Figure (left) shows the effect X on the performance. can observed that when X is small, AAER is unable to effectively suppress CO, and increasing X improves both natural and robust accuracy. However, when A increases from 1.0 to 2.0, the robust accuracy remains unchanged while the natural accuracy decreases. Therefore, choose d to balance both natural and robust
14

---

Table 7. CIFARIO: Accuracy of TRADES and ALP methods with different noise magnitude using PreActResNet-18 18 under Los threat model. The top number is the natural accuracy (%), while the bottom number is the PGD-50-10 accuracy (%)
noise magnitude
8/255
12/255
16/255
32/255
89.03
91.41
92.11
90.95
RS-TRADES (3 = 1.0)
35.56
0.86
0.00
0.00
90.72
92.19
91.50
88.69
RS-TRADES (3 = 6.0)
11.29
0.03
0.01
0.00
86.75
92.18
91.14
81.03
RS-ALP (logit pairing weight=0.5)
43.96
0.04
0.00
0.00
85.15
92.14
90.48
81.03
RS-ALP (logit pairing weight=1.0)
46.59
0.02
0.00
0.00
86.82
81.12
85.62
81.81
N-TRADES (3 1.0)
41.24
16.40
0.12
0.00
83.23
74.77
68.16
85.97
N-TRADES (.06.)
49.56
35.98
26.13
0.24
84.63
81.57
72.05
79.41
N-ALP (logit pairing weight=0.5)
46.03
27.85
0.32
0.00
82.60
76.84
69.32
82.98
N-ALP (logit pairing weight=1.0)
48.32
33.36
23.46
0.00
70%
60%
60%
50%
Accuracy
Natural Rebust
Accuracy
90%
Natural
30%
40%
30%
Rob
20%
20%
30%
10%
0%
Ag Size
Figure 7. The role of A X2 and A3 under 16/255 noise magnitude (from left to right).
accuracy. Figure 7 (middle) demonstrates the impact of Ag. It can be observed that when x is small, increasing À2 beneficial for both natural and robust accuracy. However, when A increases from to 14, the robust accuracy becomes flat while the natural accuracy decreases. Therefore, A2 = considering both natural and robust accuracy. Figure (right) shows the effect Ag- be observed that when A3 is small (2.25), the model experiences CO. Increasing X reduces the natural accuracy. From the variation of Ag between 2.25 and 4.75, we choose X3 3.25 to balance the climination of CO with natural and robustness performance.
select
C Evaluation Based on Auto Attack
Auto Attack [5] is regarded the most reliable robustness evaluation to date. It an ensemble of complementary attacks, consisting of three white-bol box attacks (APGD-CE, APGD-DLR, and FAB) and black-box attack (Square Attack). In order avoid the pseudo-robustness brought by gradient masking or gradient obfuscation, we report the Auto Attack results on 15far00/100 in Table 8 and Table 9.
In Table and Table 9, we observe that our method, AAER, consistently improves adversarial robustness under Auto Attack on both the CIFAR-10 and CIFAR-100 datasets. It demonstrates that
15

---

AAER is effective in preventing CO and enhancing robustness under various adversarial attacks. The results validate the robustness and comprehensiveness of our proposed method.
Table 8. CIFARIO: Accuracy of different methods and different noise magnitudes using PreactResNNet- 18 under Leo threat model. We only report the robust accuracy (%) under Auto Attack while the natural accuracy is same as Table 2. The results are averaged over 3 random seeds and reported with the standard deviation.
noise magnitude
8/255
12/255
16/255
32/255
FreeAT
40.23 0.33
28.04 + 0.73
0.00
0.00
0.00
0.00
ZeroGrad
43.48
MulitGrad
44.39
Grad Align
44.82 + 0.09
30.05 # 0.17
19.60 0.47
7.89 + 2.62
RS-FGSM
43.17 0.34
0.00 0.00
0.00 0.00
0.00 0.00
N-FGSM
44.43 0.24
30.32 # 0.08
19.06 + 1.81
6.78 + 0.75
RS-AAER
43.22 0.20
26.75 + 0.21
17.03 0.51
5.37 + 0.67
N-AAER
44.79 + 0.23
30.76 + 0.17
20.18 + 0.15
8.46 + 0.74
PGD-2
42.97 # 0.65
28.63 = 0.38
18.52 0.55
3.77 = 0.02
PGD-10 (20)
46.95 0.54 33.30 + 0.20 22.29 = 0.27
11.48 + 0.43
Table 9. CIFAR100:
Accuracy of different methods and different noise magnitudes using PreactResNet-18 18 under Loo threat model. We only report the robust accuracy (%) under Auto Attack while the natural accuracy is same as Table 2. The results are averaged over 3 random seeds and reported with the standard deviation.
noise magnitude
8/255
12/255
16/255
32/255
FreeAT
18.28 = 0.20
12.37 + 0.14
0.00 = 0.00
0.00 + 0.00
ZeroGrad
21.15
MulitGrad
21.62
Grad Align
21.87 # 0.13
13.78 + 0.11
9.64 # 0.12
1.76 + 0.70
RS-FGSM
7.98 + 11.91
0.00 + 0.00
0.00 + 0.00
0.00 + 0.00
N-FGSM
22.68 + 0.25
14.57 0.09
10.30 0.14
0.00 + 0.00
RS-AAER
21.41 +0.01
12.31 0.28
8.56 #0.02
2.93 + 0.17
N-AAER
22.93 # 0.10
14.73 + 0.24
10.35 + 0.11
3,46 = 0.14
PGD-2
22.52 # 0.14
13.69 + 0.02
9.56 + 0.07
1.76 + 0.22
PGD-10 (20)
23.78 # 0.08
15.61 0.09
10.93 0.05
4.13 = 0.10
D Experiment with WideResNet Architecture
We also compare the performance of our method using WideResNet-34, which more complex than PreActResNee. Since the baselines cannot adapt well to WideResNet-34 we also need to correspondingly adjust the hyperparameters. The A is fixed as 1.0 in all settings. For CIFARIO, we set RS- AAER X₂ = 4.0 and '3 = 2.0, N-AAER X2 2.5 and X 0.6. For CIFAR 100, we set RS-AAER X2 = 2.5 and A 1.0, N-AAER X₂ 1.0 and A3 = 0.2. We report the results on 10/10 in Table 10 and Table 11.
16

---

Table 10. CI Accuracy of different methods with 8/255 noise magnitude using Neee 34 under Lo threat model. The results are averaged over 3 random seeds and reported with the standard deviation.
method
RS-FGSM
N-FGSM
RS-AAER
N-AAER
PGD-2
PGD-10
natural accuracy (%)
84.41 + 0.45
84.67 + 0.32
87.39 # 0.14
84.47 # 0.23
88.68 + 0. 14
85.53 = 0.22
robust accuracy (%)
0.00 0.00
49.72 = 0.25
47.58 + 0.42
50.07 = 0.53
47.32 = 0.50
53.70 = 0.53
training time (S)
98.2
98.6
147.1
536.2
Table 11. CIFARI00: Accuracy of different methods with 8/255 noise magnitude using WideResNet- 34 under Lee threat model. The results are averaged over 3 random seeds and reported with the standard deviation.
method
RS-FGSM
N-FGSM
RS-AAER
N-AAER
PGD-2
PGD-10
natural accuracy (%)
55.04 1.24
59.02 0.63
59.81 0.38
57.76 0.36
64.64 + 0.27
60.34 + 0.34
robust accuracy (%)
0.00 0.00
28.49 = 0.54
26.88 0.30
29.09 = 0.66
26.47 = 0.10
30.02 = 0.09
In Table 10 and Table 11, we observe that when using the WideResNet-334 architecture, RS-FGSM suffers from CO with noise magnitude of 8/255, which is different from the results obtained with PreActResNNet118 architecture. However, our method, AAER, can successfully prevent CO and achieve high robustness even with complex network architectures. This demonstrates the reliability of AAER in preventing CO and improving robustness across different network architectures. is worth noting that complex networks can better reflect the efficiency of our method in terms of training time, while our method can achieve comparable robustness to multi-step AT.
E Settings and Results on SVHN, Tiny-ImageNet and Imagenet-100
SVHN Settings and Results. For experiments on SVHN, we use the cyclical learning rate schedule with 15 epochs that reaches its maximum learning rate (0.05) when 40% (6) epochs are passed. In the meantime, we uniformly increase the step size between 0 and epochs, which follow the settings of [6]. We show the hyperparameters setting SVHN in Table 12. In Table 13, we present the performance of AAER on the SVHN dataset, along with the results of the competing baseline taken from [6]. It is evident that our method can successfully prevent CO and improve robust accuracy across different noise magnitudes. This demonstrates the effectiveness of AAER in enhancing the robustness of models trained on the SVHN dataset.
Table 12. SVHN: The hyperparameters setting for different noise magnitudes. The top number is Aa while the bottom number is Ag. The A is fixed as 1.0 in all settings.
SVHN
4/255
8/255
12/255
0.5
0.6
0.45
RS-AAER
1.25
0.85
0.55
0.75
1.0
1.0
N-AAER
0.25
1.0
0.75
Settings and Results. We also scale our method to medium sized dataset Tiny- ImageNet to showcase its effectiveness. We utilized the cyclical learning rate schedule with 30 epochs, reaching the maximum learning rate of 0.2 at the midpoint of 15 epochs. For RS-AAER, we set A2 = 0.75 and λ = 0.15, while for N-AAER, we set X2 = 0.25 and λ3 = 0.05. The value of X remained fixed at 1.0 in all settings. Table 14 presents the performance of AAER on the Image-ma dataset. We can observe that our method effectively prevents CO and improves robust accuracy in this medium-scale dataset.
17

---

Table 13. SVHN: Accuracy different methods and different noise magnitudes using PreActResNett 18 under Lo threat model. The baseline results are taken from [6]. The top number is the natural accuracy (%), while the bottom number the PGD-50-16 accuracy (%). The results are averaged over 3 random seeds and reported with the standard deviation.
noise magnitude
4/255
8/255
12/255
93.66 # 0.12
91.29 # 4.07
92.36 # 1.00
FreeAT
71.61 = 0.75
0.01 = 0.00
0.00 + 0.00
94.81 # 0.16
92.42 # 1.29
88.09 = 0.40
ZeroGrad
71.59 0.22
35.93 2.73
14.14 0.32
94.71 # 0.17
94.86 = 0.97
94.48 # 0.19
MultiGrad
71.98 = 0.26
11.49 = 16.19
0.00 = 0.00
94.56 0.21
90.1 = 0.34
84.01 0.46
Grad Align
72.12 # 0.19
43.85 = 0.14
23.62 # 0.41
95.09 # 0.09
94.46 0.16 0.00
92.74 : 0.5 0.00 + 0.00
RS-FGSM
71.28 # 0.40
94.54 # 0.15
89.56 + 0.49
81.48 # 1.64
N-FGSM
72.53 # 0.19
45.63 # 0.11
26.13 0.81
94.99 0.70
90.11 0.85
83.50 4.13
RS-AAER
71.97 # 0.88
41.75 # 0.55
22.84 # 0.51
94.35 0.26
89.26 0.57
82.76 0.84 26.87 = 1.51
N-AAER
73.10 + 0.23
46.98 + 0.25
94.66 # 0.10
94.63 = 1.29
94.16 + 0.54
PGD-2
73.29 # 0.29
20.68 + 18.56
0.02 + 0.03
94.37 # 0.13
89.67 + 0.34
80.08 + 0.93
PGD-10
74.76 0.19
53.95 0.55
37.65 0.53
Table 14. Tmng-lmageNe:: Accuracy of different methods with 8/255 noise magnitude using PreActResNet-18 18 under Lxx threat model. The results are averaged over 3 random seeds and reported with the standard deviation.
method
RS-FGSM
N-FGSM
RS-AAER
N-AAER
PGD-2
natural accuracy (%)
52.28 + 2.64
48.16 + 0.61
49.86 + 0.39
47.93 + 0.27
46.43 + 0.35
robust accuracy (%)
0.00 + 0.00
20.73 # 0.40
19.66 # 0.14
20.92 + 0.01
20.72 + 0.32
Table 15. 1000: Accuracy of different methods with 8/255 noise magnitude using PreActResNet-IE 18 under Lx threat model. The results are averaged over 3 random seeds and reported with the standard deviation.
method
RS-FGSM
N-FGSM
RS-AAER
N-AAER
natural accuracy (%)
27.10 + 11.44
38.87 + 0.17
32.28 + 1.52
39.52 + 0.42
robust accuracy (%)
0.00 0.00
20.71 = 0.74
14.22 0.96
20.90 0.34
18

---

ImageNet-100 Settings and Results. We have also extended our method to a sizge-sized dataset, ImageNet- 100, to demonstrate its effectiveness. We utilized the cyclical learning rate schedule with 30 epochs, reaching the maximum learning rate of 0.2 at the midpoint of 15 epochs. For RS-AAER, we set 22 3.0 and À 2.5, while for N-AAER, we set 22 = 1.25 and X3 = 0.25. The value of X remained fixed at 1.0 in all settings. From Table 15, it is evident that our method effectively prevents CO and enhances robust accuracy in this large-scale dataset. It needs to be highlighted that the results in the above table may not be optimal, which can be further improved by increasing training epochs or adjusting hyperparameters and learning rate. However, they do establish the effectiveness of our method in trustworthyly eliminating CO on larger-scale dataset. The above outcomes underscore the scalability and effectiveness of AAER in fortifying the robustness of models trained.
F More Competing Baselines
We also compare the performance of our method with other SSAT methods, including GAT [33], NuAT [34], PGI-FGSM [17], SDI-FGSM [18] and Kim [21]. The results for GAT, NuAT and Kim are directly taken from [6], while the reported PGI-FGSM and SDI-FGSM results are based on the official code after searching the hyperparameters across different noise magnitudes.
Table 16. CIFARIO: Accuracy of different methods and different noise magnitudes using PreActResNet-13 under Loo threat model. GAT, NuAT and Kim results are taken from [6]. The top number is the natural accuracy (%), while the bottom number is the PGD-50-10 10 accuracy (%). The results are averaged over 3 random seeds and reported with the standard deviation.
noise magnitude
8/255
12/255
16/255
76.75 0.38
80.44 5.08
82.17 # 2.47 1.25 + 0.51
GAT
50.98 + 0.12
14.93 # 9.26
73.22 # 0.34
74.38 # 7.32
80.1 + 1.08
NuAT
50.10 0.33
17.54 + 8.82
3.29 + 0.87
77.94 + 0.26
83.82 + 0.86
83.42 + 0.24
PGI-FGSM
52.86 0.34
5.19 # 0.59
4.16 + 0.31
79.28 0.08
70.07 + 0.84
81.09 + 0.13
SDI-FGSM
09.26
36.56 1.34
0.05 + 0.01
88.35 0.31
90.45 + 0.08
Kim
33.01 + 0.09
13.11 = 0.63
1.88 + 0.05
83.83 + 0.27
74.40 # 0.79
64.56 + 1.45
RS-AAER
46.14 0.02
32.17 0.16
23.87 + 0.36
80.56 + 0.35
71.15 0.18
61.84 + 0.43
N-AAER
48.31 0.23
36.52 0.10
28.20 = 0.71
In Table 16, we can observe that GAT, NuAT, PGI-FGSM and SDI-FGSM demonstrate superior performance under 8/255 noise magnitude. However, it becomes apparent that these methods still suffer from CO with strong adversaries, resulting in nearly zero robustness under 16/255 noise magnitude, let alone the more challenging 32/255. In contrast, our proposed method exhibits consistent effects across different settings with negligible computational overhead, demonstrating its trustworthy effectiveness in preventing CO. We would like to emphasize that while achieving excellent performance under 8/255 noise magnitude is certainly gratifying, but can reliable defence against CO is more critical to a successful SSAT method.
G Long Training Schedule
We also compare the performance of our method using the standard robust overfitting training schedule. This training schedule consists of 200 epochs with an initial learning rate of 0.1. The learning rate is divided by 10 at the 100th and 150th epochs, respectively. During the long training
19

---

schedule, we employ warm-up up strategy in the first 20 epochs, which uniformly rises the strength of AAER from 0% 100%.
Table 17. CIFARIO: Accuracy of long training schedule with 8/255 noise magnitude using PreActResNet-18 18 under Lx threat model. The results are averaged over 3 random seeds and reported with the standard deviation.
method
RS-FGSM
N-FGSM
RS-AAER
N-AAER
natural accuracy (%)
91.21 = 0.26
83.25 0.04
85.69 + 0.20
83.23 # 0.25
robust accuracy (%)
0.00 + 0.00
36.98 + 0.34
36.05 # 0.17 37.38 + 0.16
In Table 17, we observe that our method, AAER, consistently improves adversarial robustness under the long training schedule, underscoring its consistently reliable and effective performance in preventing CO.
20


Reviews:
[{"review_id": "ZMTOfCGJiw", "reviewer": "Reviewer_GeQY", "summary": "This paper introduces the concept of abnormal adversarial examples (AAEs) and shows their strong correlation with the issue of catastrophic overfitting in single-step adversarial training. Based on this observation, the authors propose a new regularization method, AAER, which constrains the generation of abnormal adversarial examples to address this problem. The paper's insights, such as the sudden onset of catastrophic overfitting, are interesting, and the regularization-based approach is flexible enough to be combined with other methods. In addition, the experimental results confirm the effectiveness of the proposed method.", "questions": "A question of my concern is whether anomalous adversarial examples lead to catastrophic overfitting or in turn, catastrophic overfitting leads to anomalous adversarial samples or; as shown in Figure 2 in this paper, anomalous adversarial examples are also more likely to arise in a model in which the discriminant boundary is highly unsmoothed (i.e., already catastrophically overfitted); if the later, then there is no fundamental difference between this paper and other methods that regularize the smoothness of the discriminant boundary;\n\nThe ablation studies in this paper are inadequate, e.g., the baselines of this paper are N-FGSM and RS-FGSM, two advanced single-step adversarial training method, and although the regularization term proposed in this paper improves the robustness of the model even further, it is of interest to know whether the regular approach proposed in this paper is superior to other regular-based methods.\n\nThis paper presents some interesting insights in Fig.1 and Fig. 3; however, the comparisons of these experiments were performed in RS-FGSM and RS-AAER, and thus the findings may not reflect the general findings in single-step adversarial training; given that these experimental results are crucial to validate the motivation of this paper, they need to be replicated in vanilla-FGSM and vanilla-AAER for repeated validation.", "limitations": "N.A.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper provides a new perspective to explain the issue of catastrophic overfitting in single-step adversarial training.\n\n2. Some observations in this paper on anomalous adversarial examples are interesting and may provide inspiration for subsequent works.\n\n3. The proposed method is concise and intuitive, and it aligns well with the motivation of the paper.\n\n4. This paper is well writen and easy to follow.", "weaknesses": "1. Some claims in this paper are debatable.\n\n2. Insufficient ablation studies.\n\n3. The experimental comparison is insufficient.\n\nMore detailed comments please refer to the questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time and effort in reviewing our manuscript. Following thorough consideration, please find the responses to your comments below.\n\n>**Q1: Relationship between AAEs and CO**\n\n**A1:** We have observed the presence of AAEs before the occurrence of CO, indicating that CO is not the cause of AAE generation. On the other hand, directly removing AAEs can only delay but not prevent the occurrence of CO, indicating that AAEs are also not the direct cause of CO. The presence of AAEs is correlated with classifier distortion, while CO is a manifestation of a highly distorted classifier. Therefore, hindering the generation of AAEs can be considered as implicitly smoothing the loss landscape, which aligns with the common objective of AT. Notably, from the novel perspective of AAEs, our work provides two valuable findings and proposes an effective method. Firstly, we analyze the CO process and reveal that the classifier already exhibits slight distortions before CO. Secondly, we demonstrate that directly optimizing these AAEs accelerates the distortion process, leading to the rapid manifestation of CO. Finally, a lot of SSAT methods still suffer from CO when faced with strong adversaries (as shown in Tab.1 and A2) or significantly increase the computational overhead (Tab.3). In contrast, our method achieves both efficiency and robustness by explicitly hindering the generation of AAEs.\n\n>**Q2: Compare with other regular-based methods**\n\n**A2:** We compare our method with other regular-based methods, including GAT [1], NuAT [2], PGI-FGSM [3] and Grad Align [4]. The results for GAT, NuAT, and Grad Align are directly taken from [5], while the reported PGI-FGSM results are based on the official code after searching the hyperparameters across different noise magnitudes. We report both natural and **robust** accuracy (divided by a slash) of the final model on the CIFAR10 dataset.\n\n|Method / Noise Magnitude|8/255|12/255|16/255|\n|:-----|:----:|:----:|:----:|\n|GAT [1]|76.75/**50.98**|80.44/**14.93**|82.17/**1.25**|\n|NuAT [2]|73.22/**50.10**|71.15/**17.54**|80.10/**3.29**|\n|PGI-FGSM [3]|77.94/**52.86**|83.82/**5.19**|60.37/**4.16**|\n|Grad Align [4]|81.90/**48.14**|73.29/**34.51**|61.30/**26.64**|\n|RS-AAER|83.83/**46.14**|74.38/**32.17**|64.56/**23.87**|\n|N-AAER|80.56/**48.31**|71.15/**36.52**|61.84/**28.20**|\n\nBased on the above table, we can observe that GAT, NuAT and PGI-FGSM demonstrate superior performance under 8/255 noise magnitude. However, it becomes apparent that these methods still suffer from CO with strong adversaries, resulting in nearly zero robustness under 16/255 noise magnitude, let alone the more challenging 32/255. In contrast, Grad Align and our proposed method consistently exhibit effectiveness across different settings, demonstrating the reliability in preventing CO. Notably, our method performs comparably or even better than Grad Align while requiring only one-third of the training cost (Tab.3). Therefore, compared to other regularization-based methods, AAER achieves both efficiency and robustness in SSAT.\n\n>**Q3: Validate the motivation**\n\n**A3:** In Appendix A, we have demonstrated that by hindering the generation of AAEs, Vanilla-AAER can effectively prevent CO, which indirectly validates our motivation. To further validate our findings, we replicated the experiments on Vanilla-FGSM and Vanilla-AAER, observing consistent phenomena and tendencies as depicted in Fig.1 and Fig.3. We will include these additional experimental results in the revised manuscript.\n\nIf you have any further questions or concerns, please do not hesitate to contact us. We are willing and available to provide any additional information or clarification you may need.\n\n[1] Sriramanan et al. \"Guided adversarial attack for evaluating and enhancing adversarial defenses.\" NeuRIPS 2020.\n\n[2] Sriramanan et al. \"Towards efficient and effective adversarial training.\" NeuRIPS 202.\n\n[3] Jia et al. \"Prior-guided adversarial initialization for fast adversarial training.\" ECCV 2022.\n\n[4] Maksym Andriushchenko et al. \"Understanding and improving fast adversarial training.\" NeuRIPS 2020.\n\n[5] de Jorge Aranda et al. \"Make some noise: Reliable and efficient single-step adversarial training.\" NeuRIPS 2022."}, {"review_id": "OPLQ1Jkzl5", "reviewer": "Reviewer_N8c9", "summary": "The paper deals with the mitigation of catastrophic overfitting in FGSM adversarial training. The paper first presents the properties of adversarial samples before and after catastrophic overfitting. Finally, the paper proposes a regularizer (Abnormal Adversarial Examples Regularization AAER) to mitigate catastrophic overfitting during FGSM AT. The proposed regularizer prevents the generation of abnormal adversarial samples and stabilizes the training process. The effectiveness of the single-step AT with the proposed regularizer is demonstrated on CIFAR 10/100, SVHN, and TinyImageNet datasets. ", "questions": "1. Address the weakness.\n\n2. Provide experimental details for results presented in all the tables and figures\n\n3. Can the authors explain the observation considering min-max optimization AT formulation?\n\n4. Provide the plot of loss vs perturbation size of FGSM and PGD-10 attack for models trained using the proposed approach.\n\n5. Sensitivity of the algorithm to hyper-parameters: $\\lambda_1$=1, $\\lambda_2$=optimal value $\\lambda_3$=0 and $\\lambda_1$=1, $\\lambda_2$=0, $\\lambda_3$=optimal value\n\n6. Report training time for N-AAER and RS-AAER.", "limitations": "The limitation of the proposed approach is not discussed. Authors are suggested to present failure mode/cases for the proposed approach.\n", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper presents important observations on adversarial samples before and after catastrophic overfitting (CO) (i.e., presence of pseudo-adversarial samples, population and properties of pseudo-adversarial samples before and after CO). Though not novel, these observations are presented from a CO perspective. The proposed regularizer harnesses these observations to mitigate CO. ", "weaknesses": "1. The paper presents important observations on adversarial samples before and after catastrophic overfitting. However, it fails to explain the cause for these observations (why do models generate abnormal adversarial samples during single-step AT?)  \n\n2. The observations presented in this paper are not novel [1,2]. The paper presents the extreme case of gradient masking, i.e., the label leaking [1], where the accuracy of the model on adversarial samples (with large perturbation size) is greater than clean accuracy. Furthermore, not all pseudo-robust models generate abnormal adversarial samples.  \n\n3. Comparison with existing robust single step adversarial training is missing [3-6]\n\n4. The paper fails to demonstrate the effectiveness of the proposed regularizer. The proposed regularizer is plugged into existing robust single-step adversarial training methods (these methods mitigate CO) to show its effectiveness. Most of the results are shown in this setting, and a marginal improvement in robustness is observed. The robustness of the model trained using the vanilla variant is sub-par compared to models trained using existing robust single adversarial training methods (compare table-1 and 4 (supplementary)). \n\n[1] Kurakin et al. \"Adversarial machine learning at scale.\" arXiv  2016.\n\n[2] Tramer et al. “Ensemble adversarial training: attacks and defense” arXiv  2017\n\n[3] Sriramanan et al. \"Guided adversarial attack for evaluating and enhancing adversarial defenses\" NeuRIPS  2020\n\n[4] Sriramanan et al. \"Towards efficient and effective adversarial training.\" NeuRIPS  2021\n\n[5] Kim et al. \"Understanding catastrophic overfitting in single-step adversarial training\" AAAI 2021.\n\n[6] Jia et al. \"Boosting fast adversarial training with learnable adversarial initialization\" IEEE Transactions on Image Processing,  2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time and effort in reviewing our manuscript. Following thorough consideration, please find the responses to your comments below.\n\n>**Q1: Why does the model generate AAEs?**\n\n**A1:** The presence of AAEs is correlated with classifier distortion. By analyzing AAEs, we have addressed two unclear questions in SSAT: (1) the gradual distortion process of the classifier, and (2) the underlying factors of rapid classifier distortion. Our study provides valuable insights for future research to explore the underlying causes of the model's spontaneous distortion.\n\n>**Q2: The novelty of our work**\n\n**A2:** While we have observed similar phenomena as in previous studies [1, 2], it's important to emphasize that our research is grounded in a completely different research background and perspective. Notably, the concepts of PGD and CO had not been introduced at that time. Specifically, in terms of the research object, [1, 2] only focused on the final model's performance. In our work, we delved into the dynamic interaction between AAEs and CO throughout the training process, and find the presence of AAEs before the occurrence of CO. This finding reveals that before CO occurs (the model remains robust), the classifier already exhibits slight distortions. Secondly, regarding the research subject, [1, 2] treated all adversarial examples as a consistent sample. In contrast, we divide them into two groups: AAEs and NAEs, based on their distinct properties. This categorization enables us to delve into their individual impacts on the SSAT training process, and find that directly optimizing AAEs will accelerate the CO process. Lastly, for methodology, [1, 2] is based on the perspective of gradient masking, which cannot effective defence against CO. In comparison, from the perspective of AAEs, our proposed method demonstrates trustworthy effectiveness in preventing CO. The above valuable findings and effective method distinguish our work from the [1, 2].\n\n>**Q3: More competing baselines**\n\n**A3:** Please refer to **Q/A1** in general response.\n\n>**Q4: Primary objective**\n\n**A4:** Our primary objective is to eliminate CO in SSAT methods. Therefore, our method is specifically designed to address the CO, which will demonstrate significant performance improvements under CO scenarios, such as Vanilla-AAER (under 8, 12, 16, 32/255 noise magnitude), RS-AAER (under 12, 16, 32/255 noise magnitude), and N-AAER (under 32/255 noise magnitude). Besides, without the CO scenario, our approach still has complementary robustness and consistently outperforms the baselines. This clearly validates our motivation and demonstrates the effectiveness of our method.\n\n>**Q5: Experimental details**\n\n**A5:** In Section 4.1 \"Experiment Settings: Setup for Our Proposed Method,\" we have provided comprehensive details of our experimental setup, including the optimizer, learning rate schedule, and hyperparameter settings, which follow the classical setup for SSAT [3]. Additionally, for a specific table, picture, and dataset, you can find detailed setting information in their respective descriptions or appendices within the submission.\n\n>**Q6: Min-max perspective**\n\n**A6:** From a min-max perspective, the inner maximization process tries to generate the strongest adversarial examples by gradient ascent. However, due to the distortion of the classifier, the generated adversarial example might rely on the incorrect linear approximation, leading to the generation of AAEs. Subsequently, the model optimizes these AAEs through gradient descent minimization, further learning incorrect linear approximation, thereby exacerbating the distortion of the classifier. This interaction leads to a vicious circle between the AAEs and the decision boundary distortion, ultimately leading to the model rapidly manifesting as CO.\n\n>**Q7: Loss vs perturbation size**\n\n**A7:** We observed that the attack loss and successful rate (100% - test accuracy) of both the PGD and FGSM attacks on the AAER-trained method gradually increase with the perturbation size increase. We report both attack loss and successful rate of the final RS-AAER on the CIFAR10 dataset below.\n\n|Noise Magnitude|8/255|12/255|16/255|32/255|\n|:-|:-:|:-:|:-:|:-:|\n|FGSM Loss|1.21|1.54|1.77|2.14|\n|FGSM Successful Rate (%)|45.46|58.13|65.67|79.88|\n|PGD Loss|0.91|1.26|1.50|2.06|\n|PGD Successful Rate (%)|53.70|67.46|76.02|88.69|\n\n>**Q8: Sensitivity of the hyperparameters**\n\n**A8:** In Section 4.3 \"Ablation Studies: The Role of Regularization Term. Fig. 5: (i, ii)\", we observed that AAER cannot effectively mitigate CO by relying solely on $\\lambda_1$ = 1 and $\\lambda_2$ = optimal value. On the contrary, AAER that exclusively depends on $\\lambda_1$ =1 and $\\lambda_3$ =optimal value can successfully achieve robustness against adversarial attacks, as shown in Fig. 5: (i, iii). However, the absence of $\\lambda_2$ leads to an incomprehensive assessment of AAEs, resulting in lower natural accuracy. Therefore, to effectively and stably eliminate CO, all components of the regularization term are indispensable and critical.\n\n>**Q9: Computational overhead**\n\n**A9:** Please refer to **Q/A2** in general response.\n\n[1] Kurakin et al. \"Adversarial machine learning at scale.\" arXiv 2016.\n\n[2] Tramer et al. “Ensemble adversarial training: attacks and defense.” arXiv 2017.\n\n[3] de Jorge Aranda et al. \"Make some noise: Reliable and efficient single-step adversarial training.\" NeuRIPS 2022."}, {"review_id": "OUVDiTrrA1", "reviewer": "Reviewer_EF7n", "summary": "This work finds that abnormal adversarial examples(AAE) are generated during single-step adversarial training. And AAE has a deep relationship with catastrophic overfitting. According to the observation, authors propose a new regularization method AAER to regularize the numbers and output variations of AAER. Results show that their method can improve the performances of existed SSAT methods.", "questions": "1. Does the method AAER improve the performance of GradAlign?\n2. Will the extra training cost brought by AAER increase along with the dataset size? If not, what is the training cost related to?\n3. Could the AAER improve the performance of PGD-AT?\n", "limitations": "This work is about the safety in deep learning.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Figures in this work express their ideas and observations clearly, such as Figure 2.\n2. The method work for existed SOTA single-step adversarial training methods.\n3. Results of diverse networks and datasets are provided. \n4. Confidence results are provided to show the stability of methods.\n", "weaknesses": "1. Compared to N-FGSM, N-AAER does not have an obvious improvement and takes more time.\n2. AAER can improve the clean performance sometimes. For example, RS-AAER improves the clean performance a lot, 8%, compared to RS-FGSM when the $\\epsilon$ = 16/255. However, RS-AAER decreases the clean performance when the $\\epsilon$ = 32/255. And for N-FGSM, AAER always decrease the clean performance. What's the role of AAER for clean performances?\n3. In general, the high performance of AAER is build on previous successful training methods. N-AAER is the best one because N-FGSM is the best one. It's difficult to say it's the AAER make the success as the improvement brought by AAER is not that obvious.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time and effort in reviewing our manuscript. Following thorough consideration, please find the responses to your comments below.\n\n>**Q1 & 3: Primary objective**\n\n**A1 & 3:** Our primary objective is to eliminate CO in SSAT methods. Therefore, our method is specifically designed to address the CO, which will demonstrate significant performance improvements under CO scenarios, such as Vanilla-AAER (under 8, 12, 16, 32/255 noise magnitude), RS-AAER (under 12, 16, 32/255 noise magnitude), and N-AAER (under 32/255 noise magnitude). Besides, without the CO scenario, our approach still has complementary robustness and consistently outperforms the baselines. This clearly validates our motivation and demonstrates the effectiveness of our method.\n\n>**Q2: Trade-off between robustness and clean accuracy**\n\n**A2:** In AT, the model generally exhibits a trade-off between robustness and clean accuracy, as mentioned in [1]. For instance, N-AAER achieves higher robustness than N-FGSM at the expense of a slight reduction in natural accuracy. However, this trade-off does not suitable for CO models (still suitable for SSAT models without CO). Once CO occurs, the model will become fully distorted and collapse, leading to a plummeting drop in robust accuracy and a highly random and unpredictable clean accuracy (indicated by the high standard deviation reported in Tab.1). The collapse of the decision boundary significantly impacts the model's overall performance, rendering the traditional trade-off and clean accuracy irrelevant within CO phenomenon.\n\n>**Q4: AAER with Grad Align**\n\n**A4:** Both goals of AAER and Grad Align can be considered as implicitly smoothing the loss landscape. Therefore, when combining AAER and Grad Align, although robustness does not decrease, we also did not observe a significant enhancement in robustness. However, it is noteworthy that our proposed AAER method performs comparably or even better than Grad Align, while requiring only one-third of the training cost which is very crucial in SSAT.\n\n>**Q5: Training Cost**\n\n**A5:** The AAER method does not involve any extra generation, forward, or backward propagation processes, all additional processing is solely based on the original FGSM intermediate variables. Moreover, all AAER processing steps, including dividing AAEs, and calculating CE loss and L2 norm, can be efficiently executed using GPU parallel computing. As a result, the additional time overhead of AAER is directly proportional to the original FGSM training time, and this ratio remains constant regardless of changes in the dataset or model structure. **Additionally, we are delighted to report that we have observed a minimal 1.8% increase in RS/N-AAER time overhead compared to RS/N-FGSM. This observation was made using a single NVIDIA RTX 4090 GPU with PyTorch 2.0, and all experiments without half-precision float.**\n\n|Method|FreeAT|ZeroGrad|MultiGrad|Grad Align|RS\\N-FGSM|RS\\N-AAER|PGD-2|PGD-10|\n|:-----|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|\n|Training Time (S)|43.8|11.0|21.7|36.1|11.0|11.2|16.4|59.1|\n|Ratio (%)|391.07|98.21|193.75|322.32|98.21|100%|146.42|527.67|\n\n**NOTE: AAER as a baseline for training time ratio.**\n\n>**Q6: AAER with PGD-AT**\n\n**A6:** AAE is not a unique phenomenon in SSAT, but also exists in iterative-step AT methods, indicating the existence of non-smooth points in their classifiers. However, we could not find competitive hyperparameter settings to significantly enhance the robustness of multi-step AT using AAER.\n\nIf you have any further questions or concerns, please do not hesitate to contact us. We are willing and available to provide any additional information or clarification you may need.\n\n[1] Zhang et al. \"Theoretically principled trade-off between robustness and accuracy.\" ICML 2019."}, {"review_id": "tl20mpoVyG", "reviewer": "Reviewer_6vch", "summary": "The authors observe that the loss of some adversarial examples generated by the inner maximization process during training decreases, which they call abnormal adversarial examples (AAEs). Based on this, they discover a relationship between AAEs and catastrophic overfitting. Therefore, they propose an abnormal adversarial examples regularization to hinder the classifier from becoming distorted.", "questions": "It seems that this method needs to find out AAEs and NAEs for each minibatch, how much extra training time does this require?", "limitations": "See above.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The finding and definition of abnormal adversarial examples are interesting. The corresponding analysis is comprehensive.\nThe proposed method can effectively alleviate catastrophic overfitting when the perturbation radius increases.\n", "weaknesses": "The experiments were not sufficient to comprehensively evaluate the proposed method. Experiments on a larger dataset (ImageNet-100, ImageNet) should be conducted. In addition, GAT (NeurIPS 2020) and FGSM-PGI (ECCV 2022) could be added as baselines.\n30 epochs are insufficient. Experiments with more epochs should be involved to show that the proposed method can eliminate catastrophic overfitting.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time and effort in reviewing our manuscript. Following thorough consideration, please find the responses to your comments below.\n\n>**Q1: Insufficient experiments**\n\n**A1:** We acknowledge your concerns and are committed to addressing them comprehensively. In Appendix E, we have already reported the results of our method's effectiveness on the medium-sized dataset Tiny-imagenet. To further address this concern, we report the ImageNet-100 results below. For Imagenet-100, we utilized the cyclical learning rate schedule with 30 epochs, reaching the maximum learning rate of 0.2 at the midpoint of 15 epochs. For RS-AAER, we set $\\lambda_{2} = 0.75$ and $\\lambda_{3} = 0.15$, while for N-AAER, we set $\\lambda_{2} = 0.25$ and $\\lambda_{3} = 0.02$. The value of $\\lambda_{1}$ remained fixed at 1.0 in all settings. We report both natural and **robust** accuracy of the final model on the ImageNet-100 dataset under 8/255 noise magnitude.\n\n|Method |RS-FGSM|N-FGSM|RS-AAER|N-AAER|\n|:-----|:----:|:----:|:----:|:----:|\n|Natural Acc (%)|38.78|39.20|38.66|38.64|\n|Robust Acc (%)|**0.00**|**0.00**|**20.34**|**20.58**|\n\nIt needs to highlight that the results in the above table may not be optimal, which can be further improved by increasing training epochs or adjusting hyperparameters and learning rate. But, they do establish the effectiveness of our method in trustworthy eliminating CO on a larger-scale dataset.\n\n>**Q2: More competing baselines**\n\n**A2:** Please refer to **Q/A1** in general response.\n\n>**Q3: Long training schedule**\n\n**A3:** We utilize the standard robust overfitting training schedule to evaluate the effectiveness of our method. This training schedule consists of 200 epochs with an initial learning rate of 0.1. The learning rate is divided by 10 at the 100-th and 150-th epochs, respectively. During the long training schedule, we employ a warm-up strategy in the first 20 epochs, which uniformly rises the strength of AAER from 0% to 100%. We report both natural and **robust** accuracy of the final model on the CIFAR10 dataset under 8/255 noise magnitude.\n\n|Method|RS-FGSM|N-FGSM|RS-AAER|N-AAER|\n|:-----|:----:|:----:|:----:|:----:|\n|Natural Acc (%)|91.21|83.25|85.69|83.23|\n|Robust Acc (%)|**0.13**|**36.98**|**36.05**|**37.38**|\n\nBased on the above table, our proposed method demonstrates robustness with long training schedules, underscoring its consistently reliable and effective performance in preventing CO.\n\n>**Q4: Computational overhead**\n\n**A4:** Please refer to **Q/A2** in general response.\n\nIf you have any further questions or concerns, please do not hesitate to contact us. We are willing and available to provide any additional information or clarification you may need."}]

Author rebuttal:
We sincerely appreciate your time and effort in reviewing our manuscript. Following thorough consideration, please find the general responses to your comments below.

>**Q1: More competing baselines**

**A1:** We compare our method with other competing SSAT methods, including GAT [1], NuAT [2], PGI-FGSM [3], SDI-FGSM [4] and Kim [5]. The results for GAT, NuAT and Kim are directly taken from [6], while the reported PGI-FGSM and SDI-FGSM results are based on the official code after searching the hyperparameters across different noise magnitudes. We report both natural and **robust** accuracy (divided by a slash) of the final model on the CIFAR10 dataset.

|Method / Noise Magnitude|8/255|12/255|16/255|
|:-----|:----:|:----:|:----:|
|GAT [1]|76.75/**50.98**|80.44/**14.93**|82.17/**1.25**|
|NuAT [2]|73.22/**50.10**|71.15/**17.54**|80.10/**3.29**|
|PGI-FGSM [3]|77.94/**52.86**|83.82/**5.19**|83.42/**4.16**|
|SDI-FGSM [4]|79.28/**49.26**|70.07/**37.56**|81.09/**0.05**|
|Kim [5]|89.02/**33.01**|88.35/**13.11**|90.45/**1.88**|
|RS-AAER|83.83/**46.14**|74.38/**32.17**|64.56/**23.87**|
|N-AAER|80.56/**48.31**|71.15/**36.52**|61.84/**28.20**|

Based on the above table, we can observe that GAT, NuAT, PGI-FGSM and SDI-FGSM demonstrate superior performance under 8/255 noise magnitude. However, it becomes apparent that these methods still suffer from CO with strong adversaries, resulting in nearly zero robustness under 16/255 noise magnitude, let alone the more challenging 32/255. In contrast, our proposed method exhibits consistent effects across different settings with negligible computational overhead, demonstrating its trustworthy effectiveness in preventing CO. We would like to emphasize that while achieving excellent performance under 8/255 noise magnitude is certainly gratifying, but can reliable defence against CO is more critical to a successful SSAT method.

>**Q2: Computational overhead**

**A2:** The AAER method does not involve any extra generation, forward, or backward propagation processes, all additional processing is solely based on the original FGSM intermediate variables, which makes it highly efficient. The training times for N-AAER and RS-AAER are nearly identical, and for clarity, we consolidate them into AAER (30.5S), as depicted in Section 4.4 "Time Complexity Study". In terms of computational overhead, our method demonstrates a comparable cost, with only a 16% increase compared to the RS-FGSM (26.1S). In contrast, Grad Align (69.4S) and PGD-10 (140.7S) exhibit significantly higher training times, being 2.3 and 4.6 times slower than our proposed method, respectively. **Additionally, we are delighted to report that we have observed a minimal 1.8% increase in RS/N-AAER time overhead compared to RS/N-FGSM. This observation was made using a single NVIDIA RTX 4090 GPU with PyTorch 2.0, and all experiments without half-precision float.**

|Method|FreeAT|ZeroGrad|MultiGrad|Grad Align|RS\N-FGSM|RS\N-AAER|PGD-2|PGD-10|
|:-----|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|
|Training Time (S)|43.8|11.0|21.7|36.1|11.0|11.2|16.4|59.1|
|Ratio (%)|391.07|98.21|193.75|322.32|98.21|100|146.42|527.67|

**NOTE: AAER as a baseline for training time ratio.**

If you have any further questions or concerns, please do not hesitate to contact us. We are willing and available to provide any additional information or clarification you may need.

[1] Sriramanan et al. "Guided adversarial attack for evaluating and enhancing adversarial defenses." NeuRIPS 2020.

[2] Sriramanan et al. "Towards efficient and effective adversarial training." NeuRIPS 2021.

[3] Jia, X et al. "Prior-guided adversarial initialization for fast adversarial training." ECCV 2022.

[4] Jia et al. "Boosting fast adversarial training with learnable adversarial initialization." IEEE Transactions on Image Processing, 2022.

[5] Kim et al. "Understanding catastrophic overfitting in single-step adversarial training." AAAI 2021.

[6] de Jorge Aranda et al. "Make some noise: Reliable and efficient single-step adversarial training." NeuRIPS 2022.

Meta-review: This is the second time to review this paper as an AC. The current version has significant improvements and addressed most of my concerns in the last round. I have reviewed this paper again by myself and my initial evaluation on this paper is "accept" in terms of improved technique and personal emotion. 

**Research Problem**

The authors study the catastrophic overfitting problem, that classifier decision boundaries are highly distorted, and robust accuracy against iterative-step adversarial attacks suddenly drops from peak to nearly 0% in a few epochs. Specifically, the authors noticed some adversarial examples generated on the network with distorted decision boundaries exhibit anomalous behavior, which co-occurs with catastrophic overfitting. In light of this, the authors aim to prevent catastrophic overfitting by suppressing generated abnormal adversarial examples and designing a regularizer.

**Motivation**

Personally, I like that the motivation is derived from some empirical findings.

**Philosophy**

This paper starts from the conjecture that catastrophic overfitting might result from abnormal adversarial examples. Based on that, the authors design a regularizer to suppress the generation of abnormal adversarial examples.

**Presentation**

This is the second time to review this paper. The current version has significant improvement on the presentation. 

**Technique**

The novelty of this paper is the proposed Abnormal Adversarial Examples Regularization. The technical roadmap that combines the definition of NAE and AAE, empirical validation and analyses, makes sense to me.

**Experiments**

1. The experimental results are extensive.

2. To increase the readability, “S” can be removed from the table, but it can be mentioned in the table caption, such as “training time by second.”

3. It is nice to see that the authors involve more other regularizers that tackle catastrophic overfitting, according to the comments from the previous submission.

4. Figure 5 (a) is the new content that demonstrates the effectiveness of reducing the number of abnormal adversarial examples, which is good to address the comments from the previous submission.

**Clarification on My Comments** 

1. The scores from reviewers are the initial ones before the author response. Due to the no participation of reviewers during the internal discussion, these scores were unchanged.  

2. I have checked the comments of reviewers. Some of them are new and good compared to the last review I had. I also checked the author response, which was helpful to address these concerns. I **strongly** suggest the authors to modify their manuscript in the camera-ready according to the reviewers' comments, if accepted. For example, whether anomalous adversarial examples lead to catastrophic overfitting or in turn, catastrophic overfitting leads to anomalous adversarial samples. 

3. I have discussed with SAC. SAC supported my recommendation. I was very thankful for the support from SAC.
Decision: Accept (poster)