SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Runqi Lin Chaojian Yu Tongliang Liu*
Sydney AI Centre, The University of Sydney
{rlin0511, chyu8051, tongliang.liu}@sydney.edu.au
Single-step adversarial training (SSAT) has demonstrated the potential to achieve both efficiency and robustness. However, SSAT suffers from catastrophic overfitting (CO), a phenomenon that leads to a severely distorted classifier, making it vulnerable to multi-step adversarial attacks. In this work, we observe that some adversarial examples generated on the SSAT-trained network exhibit anomalous behaviour, that is, although these training samples are generated by the inner maximization process, their associated loss decreases instead, which we named abnormal adversarial examples (AAEs). Upon further analysis, we discover a close relationship between AAEs and classifier distortion, as both the number and outputs of AAEs undergo a significant variation with the onset of CO. Given this observation, we re-examine the SSAT process and uncover that before the occurrence of CO, the classifier already displayed a slight distortion, indicated by the presence of few AAEs. Furthermore, the classifier directly optimizing these AAEs will accelerate its distortion, and correspondingly, the variation of AAEs will sharply increase as a result. In such a vicious circle, the classifier rapidly becomes highly distorted and manifests as CO within a few iterations. These observations motivate us to eliminate CO by hindering the generation of AAEs. Specifically, we design a novel method, termed Abnormal Adversarial Examples Regularization (AAER), which explicitly regularizes the variation of AAEs to hinder the classifier from becoming distorted. Extensive experiments demonstrate that our method can effectively eliminate CO and further boost adversarial robustness with negligible additional computational overhead. Our implementation can be found at https://github.com/tmllab/2023_NeurIPS_AAER.
In recent years, deep neural networks (DNNs) have demonstrated impressive performance in various decision-critical domains, such as autonomous driving [25, 12], face recognition [31, 3] and medical imaging diagnosis [8]. However, DNNs were found to be vulnerable to adversarial examples [35, 9, 16]. Although these adversarial perturbations are imperceptible to human eyes, they can lead to a completely different prediction in DNNs. To this end, many adversarial defence strategies have been proposed, such as pre-processing techniques [13], detection algorithms [27], verification and provable defence [20, 44], and adversarial training (AT) [11, 26, 46]. Among them, AT is considered to be the most effective method against adversarial attacks [2, 4].
Despite the notable progress in improving model robustness, the standard multi-step AT significantly increases the computational overhead due to the iterative steps of forward and backward propagation [26, 40, 41, 42, 48, 47]. In light of this, several works have attempted to use single-step adversarial training (SSAT) as a more efficient alternative to achieve robustness. Unfortunately, a
serious problem catastrophic overfitting (CO) has been identified in SSAT [39], manifesting as a sharp decline in the model's robust accuracy against multistep adversarial attacks, plummeting from a peak to nearly  $0\%$  within a few iterations, as shown in Figure 1. This intriguing phenomenon has been widely investigated and prompted numerous efforts to resolve it. Recently, Kim [21] pointed out that the SSAT-trained classifiers are typically accompanied by highly distorted decision boundaries, which will lead to the model manifestation as CO. However, the underlying process of the classifier's gradual distortion, as well as the factor inducing rapid distortion, remains unclear.
In this study, we identify some adversarial examples generated by the distorted classifier exhibiting anomalous behaviour, wherein the loss associated with them decreases despite being generated by the inner maximization process. We refer to these anomalous training samples as abnormal adversarial examples (AAEs). Upon further investigation of the train
ing process, we observe that both the number and outputs of AAEs undergo a significant variation during CO. This observation suggests a strong correlation between the variation of AAEs and the gradually distorted classifier. By utilizing AAEs as the indicator, we re-evaluate the process of SSAT and uncover that the classifier already exhibits slight distortions even before the onset of CO, which is evidenced by the presence of few AAEs. To make matters worse, directly optimizing the model based on these AAEs will further accelerate the distortion of the decision boundaries. Furthermore, in response to this more distorted classifier, the variation in AAE will dramatically increase as a result. This interaction leads to a vicious circle between the variation of AAEs and the decision boundaries distortion, ultimately leading to the model rapidly manifesting as CO. All these atypical findings raise a question:
Figure 1. The test accuracy of RS-FGSM [39] (red line) and RS-AAER (green line) with 16/255 noise magnitude. The dashed and solid lines denote natural and robust (PGD-7-1) accuracy, respectively. The dashed black line corresponds to the 9th epoch, which is the point that RS-FGSM occurs CO.
Can CO be prevented by hindering the generation of abnormal adversarial examples?
To answer the above question, we design a novel method, called Abnormal Adversarial Examples Regularization (AAER), which prevents CO by incorporating a regularizer term designed to suppress the generation of AAEs. Specifically, to achieve this objective, AAER consists of two components: the number and the outputs variation of AAEs. The first component identifies and counts the number of AAEs in the training samples through anomalous loss decrease behaviour. The second component calculates the outputs variation of AAEs by combining the prediction confidence and logits distribution. Subsequently, AAER explicitly regularizes both the number and the outputs variation of AAEs to prevent the model from being distorted. It is worth noting that our method does not involve any extra example generation or backward propagation processes, making it highly efficient in terms of computational overhead. Our major contributions are summarized as follows:
- We identify a particular behaviour in SSAT, in which some AAEs generated by the distorted classifier have an opposite objective to the maximization process, and their number and outputs variation are highly correlated with the classifier distortion.
- We discover that the classifier exhibits initial distortion before CO, manifesting as a small number of AAEs. Besides, the model decision boundaries will be further exacerbated by directly optimizing the classifier on these AAEs, leading to a further increase in their number, which ultimately manifests as CO within a few iterations.
- Based on the observed effect, we propose a novel method - Abnormal Adversarial Examples Regularization (AAER), which explicitly regularizes the number and outputs variation of AAEs to hinder the classifier from becoming distorted. We evaluate the effectiveness of our method across different adversarial budgets, adversarial attacks, datasets and network architectures, showing that our proposed method can consistently prevent CO even with extreme adversaries and boost robustness with negligible additional computational overhead.
2.1 Adversarial Training
DNNs are known to be vulnerable to adversarial attacks [35], and AT has been demonstrated to be the most effective defence method [2]. AT is generally formulated as a min-max optimization problem [26, 4]. The inner maximization problem tries to generate the strongest adversarial examples to maximize the loss, and the outer minimization problem tries to optimize the network to minimize the loss on adversarial examples, which can be formalized as follows:
$$
\min  _ {\theta} \mathbb {E} _ {(x, y) \sim \mathcal {D}} \left[ \max  _ {\delta \in \Delta} \ell (x + \delta , y; \theta) \right], \tag {1}
$$
where  $(x,y)$  is the training dataset from the distribution  $D$ ,  $\ell(x,y;\theta)$  is the loss function parameterized by  $\theta$ ,  $\delta$  is the perturbation confined within the boundary  $\epsilon$  shown as:  $\Delta = \{\delta : \| \delta \|_p \leq \epsilon\}$ .
2.2 Catastrophic Overfitting
Since the intriguing phenomenon of CO was identified [39], there has been a line of work trying to explore and mitigate this problem. [39] first suggested using a random initialization and early stopping to avoid CO. Furthermore, [36] empirically showed that using a dynamic dropout schedule can avoid early overfitting to adversarial examples, and [6, 45] found that incorporating a stronger data augmentation is effective in avoiding CO. Another alternative approach imports partial multi-step AT, for example, [38] periodically trained the model on natural, single-step and multi-step adversarial examples, and [37] built a regularization term by comparing with the multi-step adversarial examples.
However, the above methods have not provided a deeper insight into the essence of CO. Separate works found that CO is closely related to anomalous gradient updates. [24] constrained the training samples to a carefully extracted subspace to avoid abrupt gradient growth. [10] ignored the small gradient adversarial perturbations to mitigate substantial weight updates in the network. [15] proposed an instance-adaptive SSAT approach where the perturbation size is inversely proportional to the gradient. [29] leveraged the latent representation of gradients as the adversarial perturbation to compensate for local linearity. [34] introduced a relaxation term to find more suitable gradient directions by smoothing the loss surface. [1] proposed a regularization term to avoid the nonlinear surfaces around the samples. More recently, [21] introduced a new perspective that CO is a manifestation of highly distorted decision boundaries. Accordingly, they proposed to reduce the perturbation size for the already misclassified adversarial examples.
Unfortunately, the aforementioned methods tend to either suffer from CO with strong adversaries or significantly increase the computational overhead. In this work, we delve into the interaction between AAEs and distorted decision boundaries, revealing a close relationship between them. Based on this insight, we propose a novel approach, AAER, that eliminates CO by explicitly hindering the generation of AAEs, thereby achieving both efficiency and robustness.
In this section, we first define the abnormal adversarial example (AAE) and show how their numbers change throughout the training process (Section 3.1). We further compare the outputs variation of normal adversarial examples (NAEs) and AAEs and find that their outputs exhibit significantly different behaviour after CO (Section 3.2). Building upon our observations, we propose a novel regularization term, Abnormal Adversarial Examples Regularization (AAER), that uses the number and outputs variation of AAEs to explicitly suppress the generation of them to eliminate catastrophic overfitting (CO) (Section 3.3).
3.1 Definition and Counting of AAE
Adversarial training employs the most adversarial data to reduce the sensitivity of the network's outputs w.r.t. adversarial perturbation of the natural data. Consequently, the inner maximization process is expected to generate effective adversarial examples that maximize the classification loss. As empirically demonstrated by [21], the decision boundaries of the classifier become highly distorted
Figure 2. A conceptual diagram of the classifier's decision boundary and training samples. The training samples belonging to NAE (blue) can effectively mislead the classifier, while AAE (red) cannot. The left panel shows the decision boundary before optimizing AAEs, which only has a slight distortion. The middle panel shows the decision boundary after optimizing AAEs, which exacerbates the distortion and generates more AAEs.
after the occurrence of CO. In this study, we find that after adding the adversarial perturbation generated by the distorted classifier, the loss of certain training samples unexpectedly decreases. This particular behaviour is illustrated in Figure 2, we can observe that the NAEs (blue) can either lead to the model misclassifications or position themselves closer to the decision boundary after the inner maximization process. In contrast, the AAEs (red) will be located further away from the decision boundary and fail to mislead the classifier after adding the perturbation generated by the distorted classifier. Therefore, we introduce the following formula to define AAEs:
$$
\delta = \operatorname {s i g n} \left(\nabla_ {x + \eta} \ell (x + \eta , y; \theta)\right),
$$
$$
x ^ {A A E} \stackrel {\text {d e f}} {=} \ell (x + \eta , y; \theta) > \ell (x + \eta + \delta , y; \theta), \tag {2}
$$
where  $\eta$  is the random initialization.
Next, we observe the variation in the number of AAEs throughout the model training process, and the corresponding statistical results are presented in Figure 3 (left). It can be observed that before the occurrence of CO, a small number of AAEs already existed, indicating the presence of slight initial distortion in the classifier. To further validate this point, we visualize the loss surface of both AAEs and NAEs using the method proposed by [23] as shown in Figure 4 (left and middle). It's evident that before CO, the classifier showcases a more nonlinear loss surface around AAEs in comparison to NAEs. This empirical observation strongly suggests that the generation of AAEs is directly influenced by the distorted classifier.
Besides, the number of AAEs experiences a dramatic surge during CO occurrences. For example, the number of AAEs (red line) exploded 19 times at the onset of CO (9th epoch), as shown in Figure 3 (left). Importantly, this rapid increase in the AAEs number implies a continuous deterioration in the classifier's boundaries, which in turn leads to a further increase in their number. The number of AAEs reaches its peak at the 10th epoch, surging to approximately 66 times than that before CO. To meticulously analyze the interaction between AAEs and CO, we delve into their relationship at the iteration level, as shown in Figure 4 (right). We can observe that the robustness accuracy from peak sharply drops to nearly  $0\%$  within 18 iterations, simultaneously, the number of AAEs rises from 0 to 70. Remarkably, the trends in robustness accuracy and the number of AAEs display a completely opposite relationship, suggesting a vicious cycle between optimizing AAEs and CO. Lastly, the number of AAEs consistently maintains a high level until the end of the training. Given this empirical and statistical observation, we can infer that there is a close correlation between the number of AAEs and the CO phenomenon, which also prompts us to wonder (Q1): whether CO can be mitigated by reducing the number of abnormal adversarial examples.
3.2 Outputs Variation of NAE and AAE
The above observations indicate the close relationship between CO and AAEs. In this part, we further analyze the outputs variation of AAEs during CO. Specifically, we discover that CO has a
Figure 3. The number, the variation of prediction confidence and logits distribution (from left to right) for NAEs, AAEs and training samples in RS-FGSM with 16/255 noise magnitude. The dashed black line corresponds to the 9th epoch, which is the point that the model occurs CO.
significant impact on both the prediction confidence and the logits distribution of AAEs. To quantify the variation in prediction confidence, we utilize the cross-entropy (CE) to calculate the change in loss during the inner maximization process, which is formulated as follows:
$$
\ell (x + \eta + \delta , y; \theta) - \ell (x + \eta , y; \theta). \tag {3}
$$
We investigate the prediction confidence of NAEs and AAEs variation during the model training. From Figure 3 (middle), we can observe that the change in prediction confidence of NAEs is consistently greater than 0, indicating that their lead to a worse prediction in the classifier. On the contrary, this variation in AAEs is atypical negative implying that the associated adversarial perturbation has an unexpected opposite effect. Moreover, we delve into the impact of CO on the variation in prediction confidence. Before CO, we note a slight negative variation in the AAEs' prediction confidence, which has an insignificant impact on all training samples (blue line). However, during CO, the prediction confidence of AAEs undergoes a rapid and substantial drop, reaching a decline of 17 times at the 9th epoch. After CO, the prediction confidence of AAEs is 43 times (10th epoch) smaller than before and significantly impacts all training samples.
In addition to the inability to mislead the classifier, the logits distribution of AAE is also disturbed during the CO process. To analyze the variation in logits distribution, we employ the Euclidean (L2) distance to quantify the impact of adversarial perturbation, which is formulated as follows:
$$
\left\| f _ {\theta} (x + \eta + \delta) - f _ {\theta} (x + \eta) \right\| _ {2} ^ {2}, \tag {4}
$$
where  $f_{\theta}$  is the DNN classifier parameterized by  $\theta$  and  $\| \cdot \|_2^2$  is the L2 distance.
The logits distribution variation of both NAEs and AAEs are illustrated in Figure 3 (right). Comparing the logits distribution variation between NAEs and AAEs, we can find that their magnitudes are similar before CO. However, it becomes evident that the logits distribution variation of AAEs increases dramatically during CO, being 13 times larger than before. After further optimization on AAEs, the variation in logits distribution reaches the peak, approximately 62 times larger than before. This observation highlights that even a small adversarial perturbation can cause a substantial variation in the logits distribution, this phenomenon typically happens on the highly distorted decision boundaries. Additionally, it's worth noting that the increase in logits distribution variation for NAEs (green line) occurs one epoch later than that of AAEs, indicating that the primary cause of decision boundary distortion lies within the AAEs. In other words, directly optimizing the network using these AAEs exacerbates the distortion of decision boundaries, resulting in a significant change in the logits distribution for NAEs. Even after CO, the logits distribution variance of AAEs remains twice as large as NAEs. The significant difference between NAEs and AAEs in the variation of prediction confidence and logits distribution inspires us to wonder (Q2): whether CO can be mitigated by constraining the outputs variation of abnormal adversarial examples.
3.3 Abnormal Adversarial Examples Regularization
Recognizing the strong correlation between CO and AAEs, we first attempt a passive approach by removing AAEs and training solely on NAEs. This simple approach demonstrates the capability
Figure 4. Left/Middle panel: The visualization of AAEs/NAEs loss surface before CO (8th epoch). Right panel: The number of AAEs and the test robustness within each iteration at CO (9th epoch). The green and red lines represent the robust accuracy and number of AAEs, respectively.
to delay the onset of CO, thereby confirming that direct optimization of AAEs will accelerate the classifier's distortion. However, it is important to note that the generation of AAEs is caused by the distorted classifier. Passively removing AAEs cannot provide the necessary constraints to promote smoother classifiers, thereby only delaying the onset of CO but not preventing it.
To truly relieve this problem, we design a novel regularization term, Abnormal Adversarial Examples Regularization (AAER), which aims to hinder the classifier from becoming distorted by explicitly reducing the number and constraining the outputs variation of AAEs. Specifically, part  $(i)$  categorizes the training samples into NAEs and AAEs according to the definition in Eq. 2, and then penalizes the number of AAEs. The AAEs' outputs variation is simultaneously constrained by part  $(ii)$  prediction confidence and part  $(iii)$  logits distribution. In terms of prediction confidence, we penalize the anomalous variation in AAEs that should not be negative during the inner maximization process, which is formalized as follows:
$$
A A E \_ C E = \frac {1}{n} \sum_ {i = 1} ^ {n} \left(\ell \left(x _ {i} ^ {A A E} + \eta , y _ {i}; \theta\right) - \ell \left(x _ {i} ^ {A A E} + \eta + \delta , y _ {i}; \theta\right)\right), \tag {5}
$$
where  $n$  is the number of abnormal adversarial examples.
For logits distribution, we first calculate the logits distribution variation of AAEs and NAEs separately, as shown in Eq. (6) and Eq. (7):
$$
A A E _ {-} L 2 = \frac {1}{n} \sum_ {i = 1} ^ {n} \left(\left\| f _ {\theta} \left(x _ {i} ^ {A A E} + \eta + \delta\right) - f _ {\theta} \left(x _ {i} ^ {A A E} + \eta\right) \right\| _ {2} ^ {2}\right); \tag {6}
$$
$$
N A E \_ L 2 = \frac {1}{m - n} \sum_ {j = 1} ^ {m - n} \left(\left\| f _ {\theta} \left(x _ {j} ^ {N A E} + \eta + \delta\right) - f _ {\theta} \left(x _ {j} ^ {N A E} + \eta\right) \right\| _ {2} ^ {2}\right), \tag {7}
$$
where  $m$  is the number of training samples.
Then, we use the logits distribution variation of NAEs as a reference to constrain the variation in AAEs. It's essential to emphasize that our optimization objective is to make the logits distribution variation of AAEs closer to that of NAEs, rather than less. To achieve this, we use the max function to limit the minimum value, which is formalized as follows:
$$
\text {C o n s t r a i n e d} _ {\text {V a r i a t i o n}} = \max  \left(A A E _ {-} L 2 (6) - N A E _ {-} L 2 (7), 0\right), \tag {8}
$$
where  $\max(, )$  is the max function.
Although Figure 3 (right) illustrates that the logits distribution variation of NAEs will significantly increase and instability after CO. However, that is a natural consequence of the highly distorted classifier which disrupted the logits distribution of NAEs. In contrast, after using the AAER to hinder the classifier from becoming distorted, the NAEs can be used as a stable standard throughout the training, as shown in Figure 5 (a:right).
Based on the optimization objectives described above, we can build a novel regularization term - AAER, which aims to suppress the AAEs by the number, the variation of prediction confidence
Algorithm 1 Abnormal Adversarial Examples Regularization (AAER)
Input: Network  $f_{\theta}$ , epochs T, mini-batch M, perturbation radius  $\epsilon$ , step size  $\alpha$ , initialization term  $\eta$ .
Output: Adversarily robust model  $f_{\theta}$
1: for  $t = 1 \dots T$  do
2: for  $k = 1 \ldots M$  do
3:  $\delta = \alpha \cdot \mathrm{sign}\left(\nabla_{x + \eta}\ell (x_k + \eta ,y_k;\theta)\right)$
4:  $CE = \frac{1}{m}\sum_{k = 1}^{m}\ell (x_k + \eta +\delta ,y_k;\theta)$
5:  $AAER = \mathrm{Eq}$  (9)
6:  $\theta = \theta -\nabla_{\theta}$  (CE + AAER)
7: end for
8: end for
and logits distribution, ultimately achieving the purpose of preventing CO, which is shown in the following formula:
$$
A A E R = \left(\lambda_ {1} \cdot \frac {n}{m}\right) \cdot \left(\lambda_ {2} \cdot A A E _ {-} C E (5) + \lambda_ {3} \cdot C o n s t r a i n e d _ {-} V a r i a t i o n (8)\right), \tag {9}
$$
where  $\lambda_{1},\lambda_{2}$  and  $\lambda_{3}$  are the hyperparameters to control the strength of the regularization term.
AAER can effectively hinder the generation of AAEs that are highly correlated with the distorted classifier and CO, thereby encouraging training for a smoother classifier that can effectively defend against adversarial attacks. By considering both the number and output variation of AAEs, we establish a more adaptable and comprehensive measure of classifier distortion. Importantly, our method does not require any additional generation or backward propagation processes, making it highly convenient in terms of computational overhead. The proposed algorithm AAER realization is summarized in Algorithm 1.