SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In summary, the conditions for RO are two folds: 1) the attacker  $\mathcal{A}$  should be strong enough to generate enough non-robust features (which explains why AT overfits while ST does not); and 2) the trainer  $\mathcal{T}$  should be stronger than  $\mathcal{A}$  to memorize the non-robust features (which explains how LR decay induces RO). This understanding also suggests that we can alleviate RO by restoring the balance between the minimax players, as we explore in the next section.
(a) robustness comparison
(b) flatness comparison
(c) LR decay factor
Figure 6: Mitigating RO from three different techniques. (a, b) The purported BoAT loss successfully boosts model robustness and mitigates RO by bootstrapping loss landscape flatness between online and WA model. (c) Small LR decay factor leads to better best-epoch robustness of WA model with only slight RO. (d) Stronger training attacker also helps mitigate RO and boost robustness.
(d) attack strength
4 Mitigating Robust Overfitting by Rebalacing the Minimax Game
Based on our understanding of RO from the imbalanced minimax game perspective, we further investigate how to alleviate RO by restoring balance. In particular, we can either weaken the trainer's fitting ability (Sections 4.1), or strengthen the attacker's attacking ability (Section 4.2). We observe that strategies from both directions can prevent the model trainer  $\mathcal{T}$  from fitting non-robust features too quickly and too adequately, thus can mitigate RO.
4.1 Trainer Regularization
From the trainer's side, one approach to rebalance the game is to regularize the trainer's local fitting ability after LR decay, e.g., by enforcing better landscape flatness. In this way, we could prevent the learner from drawing complex decision boundaries and memorizing non-robust features. Among existing approaches targeting at this, weight averaging (WA) [22] taking the (moving) average of history model weights is effective and efficient that it brings neglectable training overhead, so we focus on WA in this work. Along the online model  $f_{\theta}$  that is adversarially trained, we also maintain a WA model  $f_{\varphi}$  that is an exponential moving average (EMA) of the online parameters  $\varphi \gets \gamma \cdot \varphi + (1 - \gamma) \cdot \theta$ , where  $\gamma \in [0, 1]$  is the decay rate.
Bootstrapping. We notice that when the online model  $f_{\theta}$  deteriorates, the WA model, although flatter [22], will eventually deteriorate as a moving average of  $f_{\theta}$  (Figure 6a). Inspired from the latent bootstrap mechanism in self-supervised learning [16], we align the predictions of  $f_{\varphi}$  and  $f_{\theta}$  to improve the flatness of the online model  $f_{\theta}$  simultaneously. Specifically, given an adversarial example  $x$  generated by PGD attack using vanilla AT loss (Eq. 1), we have
$$
\ell_ {\mathrm {B o A T}} (x, y; \theta) = \ell_ {\mathrm {C E}} \left(f _ {\theta} (x), y\right) + \lambda \cdot \operatorname {K L} \left(f _ {\theta} (x) \| f _ {\varphi} (x)\right), \tag {4}
$$
where  $\lambda$  is a coefficient balancing the CE loss (for better training robustness) and the KL regularization term (for better landscape flatness). In one direction, the term provides more fine-grained supervision from the flatter WA model that helps avoid overfitting to the training labels [33, 8], leading to better loss landscape flatness and robustness of the online model; in return, a better online model will further improve the flatness of the WA model. Since this regularization encourages loss landscape flatness of both  $f_{\varphi}$  and  $f_{\theta}$  in a bootstrapped fashion, we name it Bootstrapped Adversarial Training (BoAT) loss.
We further verify the effectiveness of the proposed BoAT loss in Figure 6a, in which both the online model and the WA model suffer significantly less from RO and achieve higher best-epoch robustness compared with using the vanilla AT loss (see Appendix D.1 for details). Moreover, we compare their 1-D loss landscapes [25] at the last checkpoints, and Figure 6b demonstrates that both the online model and the WA model trained with our BoAT loss enjoy much better flatness, which implies weak memorization of non-robust features under the rebalanced game that explains the alleviation in RO.
Smaller LR Decay Factor. As studied in Section 3, LR decay has a dramatic influence on RO and reducing the LR decay factor can largely ease RO (Figure 5a). Therefore, we also adopt a smaller LR decay factor (together with the BoAT loss) to further regularize the trainer's fitting ability. To demonstrate its effectiveness, even when we go back to the vanilla AT loss, Figure 6c shows the WA model obtained with a very small decay rate  $d = 2$  not only has very slight RO but also attains better best test robustness than  $d = 10$ . We also remark how the WA model benefits from continuous supply of good individuals, which couldn't be guaranteed under strong RO, also throws new meaningful light on our effort in mitigating RO.
Table 1: Comparing our method with several training methods on CIFAR-10 under the perturbation norm  $\varepsilon_{\infty} = 8 / 255$  based on the PreActResNet-18 and WideResNet-34-10 architectures.
Method	PreActResNet-18	WideResNet-34-10
Natural	PGD-20	AutoAttack	Natural	PGD-20	AutoAttack
best	final	best	final	best	final	best	final	best	final	best	final
PGD-AT	81.61	84.67	52.51	46.20	47.51	42.31	86.38	87.04	56.62	48.54	51.94	45.87
TRADES	80.45	82.89	52.32	49.39	48.09	46.40	84.23	84.89	56.02	47.38	52.76	45.62
WA	83.50	84.94	55.05	47.60	49.89	43.83	87.66	87.12	56.62	49.12	52.65	46.34
KD+SWA	84.06	84.48	53.70	53.68	49.82	49.37	87.45	88.21	56.29	55.76	53.59	53.55
PGD-AT+TE	82.04	82.59	54.98	53.54	50.12	49.09	85.97	85.77	56.42	53.40	52.88	50.56
AWP	81.11	80.62	55.60	55.03	50.09	49.85	85.63	85.61	58.95	59.05	53.32	53.38
MLCATWP	-	-	58.48	57.65	50.70	50.32	-	-	64.73	63.94	54.65	54.56
ReBAT	81.86	81.91	56.36	56.12	51.13	51.22	85.25	85.52	59.59	59.29	54.78	54.80
ReBAT++	78.71	78.85	56.68	56.70	51.49	51.39	81.65	81.59	59.81	59.88	54.80	54.91
WA+CutMix	80.24	80.30	56.02	55.95	49.67	49.59	85.08	87.95	60.41	59.36	55.11	53.60
ReBAT+CutMix	79.02	78.95	56.15	56.16	50.18	50.22	86.28	87.01	61.33	61.24	55.75	55.72
Table 2: Comparing our method with several training methods on CIFAR-100 and Tiny-ImageNet under the perturbation norm  $\varepsilon_{\infty} = 8 / 255$  based on the PreActResNet-18 architecture.
Method	CIFAR-100	Tiny-ImageNet
Natural	PGD-20	AutoAttack	Natural	PGD-20	AutoAttack
best	final	best	final	best	final	best	final	best	final
PGD-AT	55.94	56.39	29.74	22.47	24.84	19.80	45.29	48.70	21.86	17.54
TRADES	55.04	57.09	29.17	26.36	24.21	23.06	48.32	47.59	22.25	21.14
WA	57.26	58.40	30.35	23.52	25.83	20.51	49.56	49.35	24.24	19.30
KD+SWA	57.17	58.23	29.50	29.33	25.66	25.65	50.28	50.67	24.37	24.30
PGD-AT+TE	56.41	57.26	30.90	29.07	25.84	24.99	46.71	50.50	22.76	19.26
AWP	54.10	54.78	30.47	30.15	25.16	25.01	43.54	43.33	23.75	23.55
MLCATWP	-	-	31.27	30.57	25.66	25.28	-	-	-	-
ReBAT	56.13	56.79	32.52	32.02	27.60	27.29	48.28	47.92	24.93	24.14
ReBAT++	52.44	54.65	32.49	32.11	27.49	27.33	45.10	44.12	25.23	24.52
WA+CutMix	56.73	57.17	31.70	31.71	26.43	26.08	46.48	46.48	24.80	24.72
ReBAT+CutMix	56.05	56.02	32.71	32.57	27.08	27.20	46.47	46.62	25.28	25.28
4.2 Stronger Training-time Attacker
From our minimax game perspective, beside directly restricting the fitting ability of the trainer  $\mathcal{T}$ , another natural approach is to strengthen the attacker  $\mathcal{A}$ 's ability to counter it after LR decay to help strike a new balance between the two players. Therefore, we apply four different training perturbation budgets after decay,  $\varepsilon = 8/255, 10/255, 12/255, 16/255$  (with fixed step size  $\alpha = 2/255$  and increasing PGD steps  $k \approx 10 \cdot \varepsilon/(8/255)$ ), where  $\varepsilon = 8/255$  is the default budget used before LR decay and in test-time attack. As can be seen from Figure 6d, the adopted stronger attacker indeed suppress RO for both the online and WA models, thanks to the hard-to-memorize adversarial non-robust features they attached to the clean training data; and again we can see how WA benefits from the alleviation of RO, especially for  $\varepsilon = 12/255$  that even hits  $>57\%$  WA model robustness against PGD-20 attack (see also Appendix C.3). Besides, a too strong training attacker may prevent the  $\mathcal{T}$  from learning useful robust features (e.g.,  $\varepsilon = 16/255$ ) that harms robustness, implying that we should carefully choose the attacker strength while rebalancing the minimax game via this approach.
4.3 Overall Approach
Above, we have introduced three effective techniques to mitigate RO from different perspectives: model regularization, learning rate decay, and attacker strength. Since stronger attackers often induce a large decrease in natural accuracy [1] (we additionally study this trade-off in Appendix C.3) and needs more careful balancing in order not to harm robustness as discussed above, we design two
(a) training longer with CutMix
Figure 7: Empirical understandings on ReBAT. (a) ReBAT can benefit from training longer (with CutMix). (b) Compared to PGD-AT, test adversarial examples still have to contain fairly many target-class non-robust features to successfully attack a model trained with ReBAT. (c) ReBAT does not result in strong bilateral class correlation as in PGD-AT.
(b) target-class non-robust features
(c) bilateral class correlation
versions of our final approach: ReBalanced Adversarial Training (ReBAT), which combines the BoAT loss with smaller LR decay factor; and  $\mathbf{ReBAT} + +$ , which combines all the three techniques.
In this section, we evaluate the effectiveness of the proposed ReBAT and  $\mathrm{ReBAT + + }$  on several benchmark datasets using different network architectures. We consider the classification tasks on CIFAR-10, CIFAR-100 [23], and Tiny-ImageNet [10] with the PreActResNet-18 [17] and WideResNet-34-10 [57] architectures. Besides the vanilla PGD-AT [30], we include the following baseline methods for alleviating robust overfitting: WA [22], KD+SWA [8], PGD-AT+TE [12], AWP [53], WA+CutMix [36] and MLCATwP [54]. During evaluation, we use PGD-20 [30] and AutoAttack (AA) [9] for the adversarial attack methods. We train each model for 200 epochs and record the checkpoint on which the validation set achieves the highest PGD-20 robustness and the final checkpoint for evaluation on the test set. Please refer to Appendix C.4 and C.5 for details and Appendix D for additional results.
5.1 Results on Benchmark Datasets
Performance across Different Networks. In Table 1, we evaluate ReBAT against previous AT variants on two popular backbone networks: PreActResNet-18 and WideResNet-34-10. We can see that without strong data augmentations like CutMix, ReBAT attains the best robustness on both models and outperforms all baseline methods against AutoAttack (AA), arguably the strongest white-box attacker. Meanwhile, ReBAT almost shows no RO on both PreActResNet-18 (51.13% best and 51.22% final AA robustness), and WideResNet-34-10 (54.78% best and 54.80% final AA robustness). For ReBAT++, it indeed results in even better best and final robustness than ReBAT on both models also with nearly perfect ability in suppressing RO, though at the cost of degradation in natural accuracy as discussed in Section 4.2. The results verify the effectiveness of our proposed method as well as the importance of rebalancing the minimax game.
Strong Augmentations. WideResNet-34-10 is more prone to overfit as it has larger capacity, and existing WA+CutMix strategy [36] still overfits by more than  $1.5\%$  (55.11% best and 53.60% final AA robustness). In comparison, ReBAT+CutMix achieves 55.75% best robustness (chosen according to PGD) and 55.72% final robustness, showing that this combination indeed alleviates RO very effectively on the relatively large WideResNet-34-10 while maintaining very high robustness. In comparison, CutMix brings little improvement on PreActResNet-18, mainly because this model has lower capacity that still underfits under strong augmentations, and it requires longer training to reach higher performance. Even though, ReBAT+CutMix still outperforms WA+CutMix by 0.63% AA robustness at the conclusion of 200 epochs of training.
Performance across Different Datasets. We also conduct experiments on two additional benchmark datasets, CIFAR-100 and Tiny-ImageNet, that are more complex than CIFAR-10. As shown in Table 2, our ReBAT, ReBAT++ and ReBAT+CutMix still achieve the highest best and final robustness, demonstrating its scalability to larger scale datasets. Similarly, CutMix does not help much as it often leads to underfitting with stronger regularization effects within relatively few training epochs.
5.2 Empirical Understandings
ReBAT Can Benefit from Longer Training. Table 1 suggests that when CutMix is applied to a PreActResNet-18 model, it demands longer training to fully unleash the beast. This motivates us to
examine the effect of longer training, e.g., 500 epochs. As shown in Figure 7a, longer training with WA+CutMix brings little improvement on final robustness (49.59% (200 epochs) v.s. 49.70% (500 epochs) under AA). Instead, ReBAT+CutMix achieves higher best robustness and the robustness keeps improving along training. In particular, the additional 300 epochs improve the final AA robustness of ReBAT+CutMix from 50.22% to 51.32%. This provides strong evidence on the effectiveness of ReBAT against RO and also shows that combining ReBAT with strong data augmentations can benefit from long training as in standard training.
Verification on Robust Overfitting Experiments. Beyond its success, we empirically study how ReBAT manages to suppress RO through our minimax game perspective. Following the experiments in Section 3.2.2, we compare PGD-AT and ReBAT in terms of 1) target-class non-robust features in test adversarial examples and 2) the strength of bilateral class correlation. We can see that the test adversarial examples still have to contain much target-class non-robust features to be misclassified (Figure 7b) and there is no longer a strong bilateral class correlation (Figure 7c). This confirms that ReBAT indeed successfully prevents the model from learning false mappings of non-robust training features which creates shortcuts for test-time attack.
6 Revisting Previous Works from the Minimax Game Perspective
From a dynamic minimax game perspective of AT, we build a generic understanding of RO as a result of memorizing non-robust features under an imbalanced game, and introduce three strategies to rebalance the minimax game to mitigate RO. In fact, we notice that this perspective can also enable a unified understanding that existing approaches to alleviate RO can be perceived as rebalancing techniques developed from the following three dimensions. Additional related works, including discussions on the game-theoretical views of adversarial training, are deferred to Appendix A.
Data Regularization. Since RO is induced by non-robust features in the training data, one way is to corrupt those features with various transformations. Since non-robust features often correspond to local and high-frequency patterns in images [21], stronger augmentations like CutMix [56] and IDBH [26] will help mitigate RO [38, 26]. A complementary approach is to correct the supervision of the adversarial examples accordingly to avoid learning false label mapping (Section 3.2), e.g., the soft labels adopted in Dong et al. [11], and the distillation-alike methods in KD+SWA [8] and TE [12].
Training Regularization. As discussed in Section 3.3, a necessary condition of RO is that the model trainer is stronger than the attacker, so restricting the trainer's ability in fitting non-robust features helps suppress RO. One approach is to directly promote model flatness, e.g., AWP [53], MLCAT $_{\mathrm{WP}}$  [54], EMA [36], SWA [8], AdvLC [27] and our BoAT loss (Section 4.1). Another approach is to adjust the learning rate schedule to avoid dramatic increase in  $\mathcal{T}$ 's ability, which is previously studied by Rice et al. [38] and Wang et al. [45]. While RO is indeed delayed by some mild LR decay schedules attempted, it still happens as  $\mathcal{T}$  eventually becomes strong. Particularly, we show that simply piecewise decay with a small LR decay factor can be significantly helpful for both better robustness and less severe RO.
Stronger Training Attacker. Previous attempts in changing attacker strength, including Cai et al. [7], Qin et al. [35] and Wang et al. [49], mainly start training from a weaker attacker (e.g.,  $\varepsilon = 0$ ), and gradually lift its strength to the standard level ( $\varepsilon = 8/255$ ), which is later pointed out by Pang et al. [33] that these methods are hardly useful when measured by AA robustness. OA-AT [1] introduces a stronger attacker from the beginning but aiming to achieve robustness against stronger attacks without much concern in RO. Instead, our method is designed from a minimax game perspective that adopts a stronger training attacker ( $\varepsilon > 8/255$ ) after LR decay in order to counter RO. Our method shows clear benefits on improving best AA robustness as well as mitigating RO.
In this paper, we investigate the underlying cause of robust overfitting through a dynamic minimax game perspective of adversarial training. Based on this perspective, we extend the robust feature framework to analyze the training dynamics of AT, and analyze how the change of LR decay induces an imbalance between the two players of AT. Specifically, we have developed a new understanding of robust overfitting through the false memorization of non-robust features, and explored three new approaches to restore the balance between the model trainer and the attacker. Remarkably, the proposed ReBalanced Adversarial Training (ReBAT) have shown that adversarial training can also benefit from long training as long as the robust overfitting issue is resolved.
Yisen Wang was supported by National Key R&D Program of China (2022ZD0160304), National Natural Science Foundation of China (62006153, 62376010, 92370129), Open Research Projects of Zhejiang Lab (No. 2022RC0AB05), Beijing Nova Program (20230484344), and CCF-Baichuan-EB Fund. Zhouchen Lin was supported by the major key project of PCL, China (No. PCL2021A12), the NSF China (No. 62276004), and Qualcomm.
[1] Sravanti Addepalli, Samyak Jain, Gaurang Sriramanan, and R Venkatesh Babu. Scaling adversarial training to large perturbation bounds. In ECCV, 2022.
[2] Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In ICML, 2018.
[3] Yang Bai, Yuyuan Zeng, Yong Jiang, Shu-Tao Xia, Xingjun Ma, and Yisen Wang. Improving adversarial robustness via channel-wise activation suppressing. In ICLR, 2021.
[4] Maria-Florina Balcan, Rattana Pukdee, Pradeep Ravikumar, and Hongyang Zhang. Nash equilibria and pitfalls of adversarial training in adversarial robustness games. In AISTATS, 2023.
[5] Joey Bose, Gauthier Gidel, Hugo Berard, Andre Cianflone, Pascal Vincent, Simon Lacoste-Julien, and Will Hamilton. Adversarial example games. In NeurIPS, 2020.
[6] Samuel Rota Bulò, Battista Biggio, Ignazio Pillai, Marcello Pelillo, and Fabio Roli. Randomized prediction games for adversarial machine learning. IEEE transactions on neural networks and learning systems, 28(11):2466-2478, 2016.
[7] Qi-Zhi Cai, Min Du, Chang Liu, and Dawn Song. Curriculum adversarial training. In *IJCAI*, 2018.
[8] Tianlong Chen, Zhenyu Zhang, Sijia Liu, Shiyu Chang, and Zhangyang Wang. Robust overfitting may be mitigated by properly learned smoothening. In ICLR, 2021.
[9] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In ICML, 2020.
[10] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
[11] Chengyu Dong, Liyuan Liu, and Jingbo Shang. Double descent in adversarial training: An implicit label noise perspective. arXiv preprint arXiv:2110.03135, 2021.
[12] Yinpeng Dong, Ke Xu, Xiao Yang, Tianyu Pang, Zhijie Deng, Hang Su, and Jun Zhu. Exploring memorization in adversarial training. In ICLR, 2022.
[13] Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Brandon Tran, and Aleksander Madry. Adversarial robustness as a prior for learned representations. arXiv preprint arXiv:1906.00945, 2019.
[14] Liam H Fowl, Micah Goldblum, Ping-yeh Chiang, Jonas Geiping, Wojciech Czaja, and Tom Goldstein. Adversarial examples make strong poisons. In NeurIPS, 2021.
[15] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015.
[16] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. In NeurIPS, 2020.
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In ECCV, 2016.
A.1 Game-Theoretical View of Adversarial Training
There is a body of literature that analyzes AT from a game theory perspective, see [6, 32, 5, 34, 37, 4]. However, existing theory papers only analyze AT's Nash equilibrium under toy models (e.g., Gaussian features and linear models), and static minimax players. None of the existing literature considers a dynamic game and could explain robust overfitting from a game perspective, particularly in practical AT algorithms, as done in our work. Nevertheless, we find some interesting connections and new insights between them that worth noting.
Under these simplified assumptions, prior works show that although the Nash equilibrium of the AT objective exists and is robust, the current alternating optimization of AT may fail to converge to the Nash equilibrium, see, e.g., a recent work [4]. The key reason is that the trainer can falsely fit the non-robust features [4], which is in a similar spirit to our analysis that robust overfitting is caused by the falsely memorized non-robust features after LR decay. This shows that our explanation is not only in line with the cutting-edge theory of AT, but also further explains robust overfitting from a game perspective for the first time. Besides, we also notice that existing AT game theory papers cannot explain robust overfitting, and from the perspective of our theory, this is possible because they overlook the dynamic change of the AT players. Thus, our understanding of AT from a dynamic game can inspire more in-depth theoretical characterization of AT from the game theory perspective.
As our work mainly focuses on the understanding robust overfitting of practical AT algorithms, and it is generally hard to theoretically analyze the training dynamics of practical DNNs, we leave more theoretical investigations under strong assumptions to future work.
A.2 Robust and Non-robust Feature View of Adversarial Training
Ilyas et al. [21] firstly propose to understand the existence of adversarial examples from a data feature perspective (introduced in Section 2). Viewing from this framework, adversarial training discards the useful yet non-robust features contained in natural images and only utilizes the robust features, which also explains the well-known trade-off between accuracy and robustness [59, 44, 46]. In particular, Ilyas et al. [21] show that robust features are generally aligned with human perception while non-robust features seem random noise. Built upon this observation, researchers explore utilizing adversarial training for representation learning [13, 47] and even image synthesis [40, 48]. This robust feature perspective of adversarial training also inspires various "adversarial-for-good" applications, such as, privacy protection [20, 51].