SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Despite these fruitful developments from this view, a concurrent work [24] challenges this common understanding by showing that non-robust features are not really useful. In particular, when transferred, non-robust features yield significantly worse performance than robust features under self-supervised learning, contrary to their behaviors under supervised learning. Meanwhile, they also show that robust features alone might not be enough to attain robustness (in practice), since the robust dataset constructed in Ilyas et al. [21] has a false sense of robustness under AutoAttack [9]. Based on these observations, they suggest that non-robust features should be considered as natural backdoor patterns of the supervised learning task alone, instead of universally useful across different learning paradigms. We note that this understanding is actually not contrary to our explanation of robust overfitting through the non-robust features, since we only need the usefulness of non-robust features for classification in our discussion. In Section 3.2, we have also mentioned that the falsely memorized non-robust features behave as learning shortcuts, which agrees with the analysis in Li et al. [24].
A.3 Dynamic Strategies for Adjusting the Trainer and the Attacker in Adversarial Training
In Secion 4, we propose three new strategies to adjust the trainer and the attacker in order to restore the balance of the minimax game. Prior to our work, there existed many proposed strategies to adaptively adjust the trainer or the attacker during adversarial training. For example, Rice et al. [38] show that different schedules of learning rate has a large effect on robust overfitting and the best-epoch robustness, and they propose early stopping to prevent robust overfitting. Wang and Wang [45] instead explore a combination of weight decay and learning and learning rate schedule and obtain better performance. On self-supervised learning, Luo et al. [29] show that a dynamic change
of data augmentation strength can improve robustness significantly. On the other side, Wang et al. [49] propose to dynamically increase the attack steps to attain better convergence, while Mo et al. [31] further apply the dynamic strategy in ViTs. Concurrent to our work, Yu et al. [55] and Wei et al. [52] also empirically find that using a large perturbation budget can mitigate robust overfitting to some extent. In our work, this strategy naturally rises from our minimax game perspective of adversarial training. Compared to sophisticated adaptive designs, our method is simpler to use and delivers better robustness: we only improve the attack strength by a constant after the learning decay.
B Additional Experiment: Adversarial Examples (in Adversarial Training)  
Contain Adversarial Non-robust Features
In Section 3, we collect misclassified adversarial examples on an early checkpoint  $A$  and evaluate them on a later checkpoint  $B$  with their correct labels to examine the memorization of those adversarial non-robust features. This is because adversarial examples used for AT consist of robust features from the original label  $y$  and non-robust features from the misclassified label  $\hat{y}$ , so  $B$  must have memorized those features (non-robust on  $A$ ) to correctly classify them. In this section, we further provide a rigorous discussion on the non-robust features contained in the adversarial examples through an additional experiment.
Experiment Design. Recall that Ilyas et al. [21] leverage targeted PGD attack [30] to craft adversarial examples on a standard (non-robust) classifier and relabel them with their target labels to build a non-robust dataset. Finding standard training on it yields good accuracy on clean test data, they prove that those adversarial examples contain non-robust features corresponding to the target labels (Section 3.2 of their paper).
Figure 8: Mining non-robust features from adversarial examples in adversarial training. 1) Craft adversarial examples  $\hat{x}_i$  using untargeted PGD attack on a AT checkpoint. 2) Relabel them (only misclassified ones) with misclassified label  $\hat{y}_i$  to build a non-robust dataset  $\{(\hat{x}_i, \hat{y}_i)\}$ . 3) Perform standard training from the checkpoint. 4) Achieve good natural accuracy ( $>55\%$ ).
However, different from their settings where we know the target labels of the adversarial examples and they are generated on a non-robust classifier, we lay emphasis on mining non-robust features from adversarial examples generated on-the-fly in AT. To this end, we first craft adversarial examples on the AT checkpoint at some epoch using untargeted PGD attack [30] following the real setting of AT, then relabel the misclassified ones  $\hat{x}_i$  (e.g., a dog) with their misclassified labels  $\hat{y}_i \neq y_i$  (e.g., cat) to build a non-robust dataset  $\{(\hat{x}_i, \hat{y}_i)\}$ . In order to capture non-robust features at exactly the training epoch these adversarial examples are used, we continue to perform standard training directly from the AT checkpoint on the non-robust dataset, and finally evaluate natural accuracy. See Figure 8 for an illustration of our experiment.
In details, we select the AT models at epoch 60 (before LR decay) and epoch 1,000 (after LR decay) to conduct the above experiment. We use untargeted PGD-20 with perturbation norm  $\varepsilon_{\infty} = 16 / 255$  to craft adversarial examples on those checkpoints from the training data of CIFAR-10 [23]. The attack we adopt is a little bit stronger than the attack of PGD-10 with  $\varepsilon_{\infty} = 8 / 255$  that is commonly used to generate adversarial examples in baseline adversarial training, because the training robust accuracy rises to as high as  $94.67\%$  at epoch 1,000 (Figure 1a) and gives less than 2,700 misclassified examples, which is significantly insufficient for further training. Using the stronger attack, we obtain a success rate of  $78.38\%$  on the checkpoint at epoch 60 and a success rate of  $34.86\%$  on the checkpoint at epoch 1,000. Since the attack success rates are different, we randomly select a same number of misclassified adversarial examples for standard training for a fair comparison. The learning rate is
initially set to 0.1 and decays to 0.01 after 20 epochs for another 10 epochs of fine-tuning. Given that the original checkpoints already have non-trivial natural accuracy, we also add two control groups that train with random labels instead of  $\hat{y}$  to exclude the influence that may brought by the original accuracy.
Figure 9: Natural accuracy during standard training. Standard training on the non-robust dataset built from the checkpoint at either epoch 60 or epoch 1,000 converges to fairly good natural accuracy  $(>55\%)$ . The failure of training with random labels proves that the good accuracy has nothing to do with the original natural accuracy of the AT checkpoint.
Results. As shown in Figure 9, we find that standard training on the non-robust dataset  $\{(\hat{x}_i,\hat{y}_i)\}$  successfully converges to fairly good accuracy  $(>55\%)$  on natural test images, i.e., predicting cats as cats, no matter from which AT checkpoints (either epoch 60 or epoch 1,000) the adversarial examples are crafted. Also, we can see that the non-trivial natural accuracy has nothing to do with the original accuracy of the AT checkpoints, as the accuracy plummets to around  $10\%$  (random guessing) as soon as the standard training starts with random labels. This proves that adversarial examples in adversarial training do contain non-robust features w.r.t. the classes to which they are misclassified, which strongly corroborates the validity of the experiments in Section 3.
Discussion on the Influence Brought by Robust Features During Standard Training. Since untargeted PGD attack cannot be assigned with a target label, we cannot guarantee that the misclassified labels  $\hat{y}$  to be uniformly distributed regardless of the original labels (especially for real-world datasets). This implies that the non-robust features are not completely decoupled from robust features, i.e., training on  $\{(\hat{x}_i,\hat{y}_i)\}$  may take advantage of  $\hat{x}_i$  's robust features from  $y_{i}$  through the correlation between  $y_{i}$  and  $\hat{y}_i$ . However, we argue that robust features from  $y_{i}$  mingling with non-robust features from  $\hat{y}_i$  only increases the difficulty of obtaining a good natural accuracy. This is because learning through the shortcut will only wrongly map the robust features from  $y_{i}$  to  $\hat{y}_i$  that never equals to  $y_{i}$ , but during the evaluation, each clean test example  $x_{i}$  will always have robust and non-robust features from  $y_{i}$ , and such wrong mapping will induce  $x_{i}$  to be misclassified to some  $\hat{y}_i$  due to the label of its robust features. As a result, despite the negative influence brought by robust features, we still achieve good natural accuracy at last, which further solidifies our conclusion.
C Experiment Details
In this section, we provide more experiment details that are omitted before due to the page limit.
C.1 Baseline Adversarial Training
In this paper, we mainly consider classification task on CIFAR-10 [23]. The dataset contains 60,000  $32 \times 32$  RGB images from 10 classes. For each class, there are 5,000 images for training and 1,000 images for evaluation. Since we mainly aim to track the training dynamic to verify our understandings in RO instead of formally evaluating an algorithm's performance, we do not hold a validation set in most of our verification experiments and directly train on the full training set.
Figure 10: More test-time confusion matrices during the first 200 epochs of the training. After LR decays (the second row), the confusion matrix  $A$  immediately becomes symmetric, as the spectral norm  $\| A - A^T\| _2$  w.r.t. the matrix decreases from  $\geq 350$  before epoch 100 to  $\leq 150$  after epoch 200.
(a) train
Figure 11: Confusion matrices of the training and the test data at an epoch before robust overfitting starts. They show nearly a same pattern of attacking preference among classes.
(b) test
For baseline adversarial training, We use PreActResNet-18 [17] model as the classifier. We use PGD-10 attack [30] with step size  $\alpha = 2 / 255$  and perturbation norm  $\varepsilon_{\infty} = 8 / 255$  to craft adversarial examples on-the-fly. Following the settings in Madry et al. [30], we use SGD optimizer with momentum 0.9, weight decay  $5\times 10^{-4}$  and batch size 128 to train the model for as many as 1,000 epochs. The learning rate (LR) is initially set to be 0.1 and decays to 0.01 at epoch 100 and further decays to 0.001 at epoch 150. For the version without LR decay used for comparison in our paper, we simply keep the LR to be 0.1 during the whole training process.
Each model included in this paper is trained on a single NVIDIA GeForce RTX 3090 GPU. For PGD-AT, it takes about 3d 14h to finish 1,000 epochs of training.
C.2 Verification Experiments for Our Minimax Game Perspective on Robust Overfitting
Memorization of Adversarial Non-robust Features After LR Decay. We craft adversarial examples on a checkpoint before LR decay (60th epoch) and evaluate the misclassified ones on a checkpoint after LR decay ( $\geq 150$ th epoch) with their correct labels to evaluate the memorization of non-robust features in the training adversarial examples (see Appendix B for detailed discussions) in Section 3.1 and 3.2. Following Appendix B, we adopt PGD-20 attack with perturbation norm  $\varepsilon_{\infty} = 16 / 255$  to craft adversarial examples which is stronger than the common attack setting we use in PGD-AT. We note that test adversarial examples crafted by a stronger attack indicates stronger extraction of the non-robust features, so they are more indicative of non-robust feature memorization when still correctly classified.
Verification I: More Non-robust Features, Worse Robustness. At the beginning of Section 3.2, we create synthetic datasets to demonstrate that memorizing the non-robust training features indeed harms test-time model robustness. To instill non-robust features into the training dataset, we minimize the adversarial loss w.r.t. the training data in a way that just like PGD attack, with the only difference that we minimize the adversarial loss instead of maximizing it. Since we only use a very small perturbation norm  $\varepsilon_{\infty} \leq 4/255$ , the added features are bound to be non-robust. For a fair comparison, we also perturb the training set with random uniform noise of the same perturbation norm to exclude the influence brought by (slight) data distribution shifts. We continue training from the 100-th baseline AT checkpoint (before LR decay) on each synthetic dataset for 10 epochs, and then evaluate model robustness with clean test data.
Verification II: Vanishing Target-class Features in Test Adversarial Examples. This is to say that when RO happens, we expect that test adversarial examples become less informative of the classes to which they are misclassified according to our theory. To verify this, we first craft adversarial examples on a checkpoint  $T$  after RO begins, then evaluate the misclassified ones with their misclassified labels  $\hat{y}$  on the checkpoint saved at epoch 60. As a result, the accuracy reflects how much information (non-robust features) from  $\hat{y}$  the adversarial examples have to contain to be misclassified to  $\hat{y}$  on  $T$ . All adversarial examples evaluated in the experiments in Section 3.2.2 are crafted using PGD-10 attack with perturbation norm  $\varepsilon_{\infty} = 8 / 255$ .
Verification III: Bilateral Class Correlation. To quantitatively analyze the correlation strength of bilateral misclassification described in Section 3.2.2, we first summarize all  $y_{i} \to y_{j}$  misclassification rates into two confusion matrices  $P^{\mathrm{train}}$ ,  $P^{\mathrm{test}} \in \mathbb{R}^{C \times C}$  for the training and test data, respectively. Because we are mainly interested in the effect of the LR decay, we focus on the relative change on the test confusion matrix before and after LR decay, i.e.,  $\Delta P^{\mathrm{test}} = P_{\mathrm{after}}^{\mathrm{test}} - P_{\mathrm{before}}^{\mathrm{test}}$ . According to our theory, for each class pair  $(i,j)$ , there should be a strong correlation between the training misclassification of  $i \to j$  before LR decay, i.e.,  $(P_{\mathrm{before}}^{\mathrm{train}})_{ij}$ , and the increase in test misclassification of  $j \to i$ , i.e.,  $\Delta P_{ji}^{\mathrm{test}}$ , as  $i \to j$  training misclassification (may due to intrinsic class bias, as will be further discussed below in details) induces  $i \to j$  false mappings and creates  $j \to i$  shortcuts. To examine their relationship, we plot the two variables  $((P_{\mathrm{before}}^{\mathrm{train}})_{ij}, \Delta P_{ji}^{\mathrm{test}})$  and compute their Pearson correlation coefficient  $\rho$ .
Verification IV: Symmetrized Confusion Matrix. In Section 3.2.2, we mention the growing symmetry of the test-time confusion matrix after LR decay as an evidence of the strengthening  $y \to y'$  and  $y' \to y$  correlation. Here we present more confusion matrices during the 200 epochs of the training in Figure 10, and it is very clear that the confusion matrices soon become symmetric after LR decay and RO starts. For a deeper comprehension of this phenomenon, we first visualize the confusion matrices of the training and the test data at an epoch before RO starts in Figure 11. They exhibit nearly a same pattern of attacking preference among classes (e.g.,  $y \to y'$ ) due to the bias rooted in the dataset, e.g., class 6 is intrinsically vulnerable in this case. For the test data, this intrinsic bias wouldn't be wiped out through learning due to the non-generalizability of the memorization of non-robust features in the training data, as discussed at the beginning of Section 3.2 (i.e.,  $y \to y'$  bias still holds); and for the training data, this biased feature memorization will open shortcuts for test-time adversarial attack as discussed in Section 3.2 (i.e.,  $y' \to y$  begins). Combining both  $y \to y'$  and  $y' \to y$ , we arrive at the symmetry of test-time confusion matrix.
Additional Results: Changing LR Decay Schedule. Our discussion above is based on the piecewise LR decay schedule, in which the sudden decay of LR most obviously reflects our understandings. Besides, we also explore other LR decay schedules, including Cosine/Linear LR decay, to check whether different LR decay schedules will affect the observations and claims we made in this paper. For each schedule, we train the model for 200 epochs following the settings in Rice et al. [38]. As demonstrated in Figure 12a, we arrive at the same finding as Rice et al. [38] that with Cosine/Linear LR decay schedule, the training still suffers from severe RO after epoch 130. Then, we rerun the empirical verification experiments in Section 3.2.2 and find that under both the two LR decay schedules 1) the test adversarial examples indeed contain less and less target-class non-robust features as the training goes and RO becomes severer and severer (Figure 12b), 2) the bilateral class correlation becomes increasingly strong (Figure 12c) and 3) the confusion matrix indeed becomes symmetric (Figure 13). The results are almost the same as the results achieved when we adopt piecewise LR decay schedule because even though these LR decay schedules are mild, the LR eventually becomes small and makes the trainer  $\mathcal{T}$  overly strong to memorize the harmful non-robust features, indicating
that our understandings in the cause of RO is fundamental and regardless of whatever LR decay schedule is used.
(a) test robust accuracy during training
(b) target-class non-robust features
(c) bilateral class correlation
Figure 12: Empirical verification of our explanation for robust overfitting when Cosine/Linear LR decay schedule is applied. (a) With Cosine/Linear LR decay schedule, the training still suffers from severe RO. (b) After RO begins, non-robust features in the test data become less and less informative of the classes to which they are misclassified. (c) Increasingly strong correlation between training-time  $y \to y'$  misclassification and test-time  $y' \to y$  misclassification increase.
Figure 13: Test-time confusion matrix also becomes symmetric and implies that the bilateral correlation also exists when Cosine/Linear LR decay schedule is applied.
Table 3: Training with stronger attack and evaluating model robustness on CIFAR-10 under the perturbation norm  $\varepsilon_{\infty} = 8 / 255$  based on the PreActResNet-18 architecture.
Attack Strength	Natural	PGD-20	AutoAttack
best	final	diff	best	final	diff	best	final	diff
ε = 8/255, PGD-10 (baseline)	83.50	84.94	-1.44	55.05	47.60	7.45	49.89	43.83	6.06
ε = 10/255, PGD-12	80.66	82.48	-1.82	56.63	50.63	6.00	50.89	46.13	4.76
ε = 12/255, PGD-15	78.17	80.25	-2.08	57.09	53.13	3.96	50.99	47.66	3.33
ε = 14/255, PGD-17	73.92	76.70	-2.78	56.42	54.04	2.38	50.28	48.53	1.75
ε = 16/255, PGD-20	69.51	73.11	-3.60	55.09	54.27	0.82	49.58	48.53	1.05
C.3 Experiments on the Effect of Stronger Training Attacker
In Section 4.2, we point out that using a stronger attacker in AT is able to mitigate RO to some extent by neutralizing the trainer  $\mathcal{T}$  's fitting power when it is overly strong. To achieve the results reported in Figure 6d, we craft adversarial examples on-the-fly with more PGD iteration steps when  $\varepsilon$  is larger (see Table 3), and further evaluate the best and last robustness of the WA models against PGD-20 and AA. Although RO is only partially mitigated and natural accuracy decreases when a stronger attacker is applied as summarized in Addepalli et al. [1], it may be surprising to find from Table 3 that an attacker of appropriate strength may significantly boost the best WA robustness. This suggests using a stronger attack could potentially be an interesting new path to stronger adversarial defense, and we leave it for future work
C.4 Detailed Experiment Setup of ReBAT for Mitigating Robust Overfitting
Datasets. Beside CIFAR-10, we also include CIFAR-100 [23] and Tiny-/ImageNet [10] for evaluation of the effectiveness of ReBAT. CIFAR-100 shares the same training and test images with CIFAR-10, but it classifies them into 100 categories, i.e., 500 training images and 100 test images for each class. Tiny-/ImageNet is a subset of ImageNet [10] which contains labeled  $64 \times 64$  RGB images from 200 classes. For each class, it includes 500 and 50 images for training and evaluation respectively. Following Rice et al. [38], we hold out 1,000 images from the original CIFAR-10/100 training set, and similarly 2,000 images from the original Tiny-/ImageNet training set as validation sets.
Training Strategy. For CIFAR-10 and CIFAR-100, we follow exactly the same training strategy as introduced in Appendix C.1, except that for  $\mathrm{ReBAT}++$  we adopt PGD-12 with perturbation norm  $\varepsilon_{\infty} = 10 / 255$  for training after LR decay. For Tiny-ImageNet, we follow the learning schedule of Chen et al. [8], in which the model is trained for a total of 100 epochs and the LR decays twice (by 0.1) at epoch 50 and 80.
Choices of Hyperparameters. For  $\mathrm{KD + SWA}$  [8], PGD-AT+TE [12], AWP [53] and WA+CutMix [36], we strictly follow their original settings of hyperparameters. For MLCAT $_{\mathrm{WP}}$  [54], we simply report the test results reported in their paper. Following Chen et al. [8], SWA/WA as well as the ReBAT regularization pursued in Section 4.1 start at epoch 105 (5 epochs later than the first LR decay where robust overfitting often begins), and following Rebuffi et al. [36] we choose the EMA decay rate of WA to be  $\gamma = 0.999$ . Please refer to Table 4 for our choices of the decay factor  $d$  and regularization strength  $\lambda$ . We notice that since CutMix improves the difficulty of learning, the model demands a relatively larger decay factor to better fit the augmented data. For Tiny-ImageNet, we also apply a larger  $\lambda$  after the second LR decay to better maintain the flatness of adversarial loss landscape and control robust overfitting. We provide more discussions on the choice of hyperparameters in Appendix D.1.
Table 4: Choices of hyperparameters when training models on different datasets using different network architectures with ReBAT.
Network Architecture	Method	CIFAR-10/CIFAR-100	Tiny-ImageNet
PreActResNet-18	ReBAT	d=1.5,λ=1.0	d=4.0,λ1=2.0,λ2=10.0
ReBAT++	d=1.7,λ=1.0	d=4.0,λ1=2.0,λ2=10.0
ReBAT+CutMix	d=4.0,λ=2.0	d=6.0,λ=1.5
WideResNet-34-10	ReBAT	d=1.3,λ=0.5	-
ReBAT++	d=1.3,λ=0.5	-
ReBAT+CutMix	d=4.0,λ=2.0	-
C.5 Training Robust Accuracy
Figure 14a shows the robust accuracy change on the training data during the training. Compared with vanilla PGD-AT that yields training robust accuracy over  $80\%$  at epoch 200, ReBAT manages to suppress it to only  $65\%$ . It successfully prevents the trainer  $\mathcal{T}$  from learning the non-robust features w.r.t. the training data too fast and too well, and therefore significantly reduces the robust generalization gap (from  $\sim 35\%$  to  $\sim 9\%$ ) and mitigates RO.
C.6 Training Efficiency
We also test and report the training time (per epoch) of several methods evaluated in this paper. For a fair comparison, all the compared methods are integrated into a universal training framework and each test runs on a single NVIDIA GeForce RTX 3090 GPU.
From Table 5, we can see that ReBAT requires nearly no extra computational cost compared with vanilla PGD-AT in each epoch (136.2s v.s. 131.6s), implying that it is an efficient training method in practical. As we regularize the model trainer to restore the balance of the minimax game, we can see from Figure 14b that the training takes slightly longer to attain optimal performance, e.g., 110 epochs (PGD-AT) and 170 epochs (ours). However, in practice, people usually train longer in AT (typically 200 epochs) to attain better accuracy, and use early stopping to select the best checkpoint. As our
(a) training