Balance, Imbalance, and Rebalance: Understanding Robust Overfitting from a Minimax Game Perspective
Authors: Yifei Wang
Liangchen Li
Jiansheng Yang
Zhouchen Lin
Yisen Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: Adversarial Training

Abstract: Adversarial Training (AT) has become arguably the state-of-the-art algorithm for extracting robust features. However, researchers recently notice that AT suffers from severe robust overfitting problems, particularly after learning rate (LR) decay. In this paper, we explain this phenomenon by viewing adversarial training as a dynamic minimax game between the model trainer and the attacker. Specifically, we analyze how LR decay breaks the balance between the minimax game by empowering the trainer with a stronger memorization ability, and show such imbalance induces robust overfitting as a result of memorizing non-robust features. We validate this understanding with extensive experiments, and provide a holistic view of robust overfitting from the dynamics of both the two game players. This understanding further inspires us to alleviate robust overfitting by rebalancing the two players by either regularizing the trainer's capacity or improving the attack strength. Experiments show that the proposed ReBalanced Adversarial Training (ReBAT) can attain good robustness and does not suffer from robust overfitting even after very long training. Code is available at https://github.com/PKU-ML/ReBAT.

Balance, Imbalance, and Rebalance: Understanding Robust Overfitting from a Minimax Game Perspective
Yifei Wang
Liangchen Li 116730cam ac uk
yifei_wang@ple edu
Jiansheng Yang!
Zhouchen 3.
Yisen Wang
yjs@math. pku. edu.cm
zlin@pku. edu. edu. cn yisen. wangēpku edu, cn
1 School of Mathematical Sciences, Peking University Department of Engineering. University Cambridge
National Key Lab of General Artificial Intelligence.
School of Intelligence Science and Technology, Peking University
Institute for Artificial Intelligence, Peking University Peng Cheng Laboratory
Abstract
Adversarial Training (AT) has become arguably the state-of-the-art algorithm for extracting robust features. However, researchers recently notice that AT suffers from severe robust overfitting problems, particularly after learning rate (LR) decay. this paper, we explain this phenomenon by viewing adversarial training as a dynamic minimax game between the model trainer and the attacker. Specifically, we analyze how LR decay breaks the balance between the minimax game by empowering the trainer with stronger memorization ability, and show such imbalance induces robust overfitting as a result of memorizing non-robust features. We validate this understanding with extensive experiments, and provide holistic view of robust overfitting from the dynamics of both the two game players. This understanding further inspires us alleviate robust overfitting by rebalancing the two players by either regularizing the trainer's 's capacity or improving the attack strength. Experiments show that the proposed ReBalanced Adversarial Training (ReBAT) can attain good robustness and does not suffer from robust overfitting even after very long training. Code is available https: /ttpps///ithub.. Con/PKU-ML/ReBBA
Introduction
Overfitting seems to have become history in the deep learning era. Contrary to the traditional belief statistical learning theory that large hypothesis class will lead to overfitting, Zhang et al. [58] note that DNNs have good generalization ability test data even if they are capable of memorizing random training labels. Nowadays, large-scale training often does not require early stopping, and is observed that longer training simply brings better generalization [19]. However, researchers recently notice that in Adversarial Training (AT), overfitting is still a severe issuc on both small and large scale data and models [38]. As the most effective defence against adversarial perturbation [2], AT solves a minimax optimization problem with training data Deruin and model fo [15, 30, 49]:
min E2 max (cB(fo(z).)),
(I)
(EE,(
where fce denotes the cross entropy (CE) loss function, E,(T) fxr || FI, < $) denotes the l no-m ball with radius E. However, contrary to standard training (ST), AT suffers severely from
Eqqual Contribution.
Corresponding Authors: Yisen Wang (yisen.wang @ @kt.gguuuuddu.c) and Zhouchen Lin (zlin @ pku.edu.en)
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

51-
Shin decap!
97-
0.44
0.22
0.4
4200 600
1000
0.93
2000
600
10:0
150
Souch
Spach
Spacch
(a) training robustness during ad- versarial training
(b) test robustness during adver- sarial training
(c) test robustness when decay
different time
Figure 1: (a, b) Training/test robust accuracy on CIFAR-10 using vanilla PGD-AT [30] w/ and w/o LR decay. (c) RO happens whenever LR decays.
robust overfitting (RO): after particular point (e.g., learning rate decay), its training robustness will keep increasing (Figure la, red line) while its test robustness decreases dramatically (Figure 1b, red line). This abnormal behavior of AT has attracted much interest. Previous works find that the AT loss landscape becomes much sharper during RO [42, 8, 53], but we are still unclear about what causes the sharp landscape. Some researchers try to explain RO through known ST phenomena, such as, random label memorization [12] and double descent [11]. However, these theories cannot explain why only AT overfits while ST hardly does.
Instead, we argue that the reason of RO should lie exactly in the difference between AT and ST. Sharing similar training recipes and loss functions, their key difference is that AT adopts a bilevel minimax objective, compared to the min-only ST objective. We can think of AT as a minimax game between two players (Eq. I): the model trainer T (minimizing loss via updating 0) and the attacker A (maximizing loss via updating x), both adopting first-order algorithms practice. With proper configuration of both players, AT can strike balance during which model robustness hinges constant level. In comparison, we also notice that when one player (typically the trainer) changes its original strategy (e.g.. decaying learning rate), the original balance of the game is broken and RO also occurs subsequently. We further verify this observation through a controlled experiment. As shown in Figure Ic, if we do not perform LR decay, RO will not occur till the end; but whenever perform LR decay, RO will immediately happen. This provides strong evidence showing that breaking the balance between minimax layers can cause robust overfitting.
To gain further insights into this lalaane--ibbllnee process, in Section 3, we provide a generic explanation for RO from the perspective of feature learning. Specifically, when the balance breaks after the LR decay, the trainer with a smaller LR is endowed with a stronger local fitting ability capable of memorizing adversarial examples such that the attacker can no longer succeed in attack. However, during this process, the model learns false mapping of non-robust features contained the training adversarial examples, which opens shorteuts for the test-time attacker and induces large test robust error. Accordingly, we design a series of experiments that align well with our analysis.
The minimax game perspective also indicates a simple fix to RO, that is to adjust the ability of the two minimax players to new balance by modifying their strategies, namely, rebalancing. In this work, we explore three new strategies for rebalancing the game after LR decay: bootstrapping objective, smaller LR decay factor, and stronger attacker. The former two restrict the local fitting ability of the trainer while the third enhances the ability of the attacker. We show that all the three strategies help mitigate robust overfitting to a large extent. Based on these insights, we propose ReBalanced Adversarial Training (ReBAT) that achieve superior robustness with a neglectable (if any) degree of robust overfitting, even if we train it further to 500 epochs. Meanwhile, ReBAT attains superior robustness (both best-epoch and final-epoch) This suggests that with appropriately balanced training strategies, AT could also enjoy similar "train longer, generalize better" property like ST. To summarize, main contributions are:
benchmark datasets including CIFAR- 10,
CIFAR-IOG 100, and
We propose to understand adversarial training (AT) as a dynamic minimax game between two actual players: the model trainer T and the attacker A. We characterize the balance status of the minimax game from the perspective of non-robust features.
We show robust overfitting is caused by the imbalance between two minimax players induced by the LR decay, and explain this process the false memorization of non-robust

---

features. We design series of experiments to verify this explanation and provide holistic characterization of robust overfitting from the views of both players.
From our dynamic perspective, we propose three effective approaches that mitigate robust overfitting by re-striking a balance between the model trainer and the attacker. Experiments show that our proposed ReBAT method largely alleviates robust overfitting while attaining good robustness on different network architectures and benchmark datasets.
2 Preliminaries
Adversarial Attack. Given image classifier f. adversarial attack [43] is proposed to perturb the image by an imperceptible noise such that the image is misclassified by f. A well-known white-box attacker A is PGD (Projected Gradient Descent) [30], which uses multi-step gradient ascent steps to maximize the cross entropy loss while projecting intermediate steps to the feasible region:
Tk+1 = Tk + a Vay as (co(J)()))) y), + = (e) (ox+1), ( k = 0....., K - 1.
(2)
where xo E,E ~ U10, s), and E,( {-| E [0, 1] -1-,11x-111 =11 S $} is the feasible region.
Adversarial Training. To counter misclassification caused by the attacker, adversarial training (AT) [43, 15, 30, 38, 50, 3] aims build robust classifier through the minimax objective in Equation 1. In practice, AT iterates between two steps: 1) solve the inner-loop maximization w.e.. input x using few (e.g,, 10) PGD steps (Eq. 2) generate minibatch of adversarial examples {.). and then 2) solve the outer-loop minimization w.r.t. NN parameters with SGD as the trainer T. Therefore, AT can be seen minimax game between two players (specifically, two first-order optimization algorithms), the attacker A and the trainer T, with the objective function
y).
Robust and Non-robust Features. The arguably most prevailing understanding of adversarial train- ing is the framework proposed by Ilyas et al. [21], where they regard training data as a composition of robust and non-robust features that are both useful for classification. Consider binary classification task, a feature f X - R is categorized as 1) a useful feature Eg,y f(x)] 2 pt 2) a robust feature if it is useful for p 0 and Eg,y inf
f(()] 2 x: 3) non-robust feature if it is useful for p > 0 and not robust for any 7 2 O. We can also choose a threshold y to determine its belonging class since there is clear dividing line between robust and non-robust features in practice. According to Ilyas et al. [21], standard training (ST) adopts non-robust features for better accuracy, which also leads adversarial vulnerability. Instead, AT only uses robust features and becomes resistant to adversarial attack. Our work also adopts the notion of non-robust features. But compared with their framework which mainly considers the ideal, static solution the minimax problem (a learned robust classifier), our work zooms into the dynamic interplay of the two actual players (how a classifier learns the robust & non-robust features in AT) to understand RO.
3 A Minimax Game Perspective on Robust Overfitting
In this section, we propose a minimax game perspective to understand the cause of robust overfitting. We first show the minimax game in Equation achieves balance when the trainer T is not capable of memorizing the rob-st features, and then demonstrate through several verification experiments that the break of this balance, which leads to memorization of the non-robust features, actually induces shortcuts for test-time adversarial attack that results in RO.
3.1 AT Strikes a Balance when Non-robust Features Cannot be Memorized
First, we look at the balanced status of adversarial training. From Figure Ia (blue line), we observe that without LR decay, the two players can maintain a constant robustness level throughout training. Inside the equilibrium, the attacker AA injects non-robust features from a different class y/ to x in order to misclassify an input (z, 3) [21] while the trainer can only correctly classify part of them using robust features. In other words, the game can strike balance when there is limit of the trainer's s fitting ability that after a certain robustness level, it can no longer improve robustness by fitting the non-robust features that the attacker can modify (i.e., non-robust features are not memorizable).
To verify this point, we collect misclassified adversarial examples (containing non-robust features w.r.l. of if misclassified to y on an early checkpoint (60th epoch) and evaluate their accuracy on
We refer the reader Appendix for simple verification and to Fowl et al. [14] for further discussion.
3

---

AOOLIN with mixeete Woel
friss Mol.
+10
dociy tran)
decey
-20
(a) non-robust feature memorization (b) influence stronger non-robust (c) target-class non-robust features
features (larger E) on test robustness
Figure 2: (a) Almost non-gen non-genrralizable non-robust features training data is memorized only after LR decay. (b) With LR decay, injecting stronger non-robust features (larger E) induces severer degradation in robustness, while the degradation is less obvious w/o LR decay. (c) After LR decay, test adversarial examples contains less and less target-class non-robust features.
Robust overfitting happens!
failed test attack
successful test allack
train
before LR decay / good robustness
after LR decay
X poor robustness
A
test
test
Xtest
Xtest,
class y
class y'
Xtrain
Xtraín
Xtrain
attack
decision
(a) before LR decay, trainer fails to well adversarial examples due to limited fit- ting ability
(b) after LR decay, stronger trainer fits misclassified ad- versarial examples with str- onger local fitting ability
(c) with decision bound- ary, the attacker finds short- cuts misclassify test ples easily
boundary
Figure 3:
An illustration example of the rise of robust overfitting. Here, the attack (from the clean example to the adversarial example) follows the direction to the nearest decision boundary.
later checkpoint (2 150th epoch) w.t. their true labels y (detailed settings in Appendix C.2). If the adversarial non-robust features can be memorized, we would expect high accuracy on these adversarial examples the later checkpoints. However, in practice, the blue solid line in Figure 2a shows the accuracy is still relatively low (< 25%), suggesting that these non-robust features are almost not memorized under the same training configuration without LR decay.
3.2 Balance -> Imbalance: Robust Overfitting as a Result of Imbalanced Minimax Game
Next, look how LR decay breaks the balance of the minimax game. As pointed out by the theoretical work [28], small LR favors -oggeerllee and hard-to-fit patterns, which correspond non-robust features AT [21]. Thus, with a smaller LR, the trainer becomes more capable of memorizing these features by drawing more complex decision boundaries in a small local region. Indeed, we observe significant increase training robust accuracy Figure la. Moreover, we can see from the red line in Figure 2a that most of the pre-LR-decay adversarial examples (with adversarial non-robust features) are correctly classified after LR decay, indicating stronger memorization of non-robust features. Therefore, we claim that LRR decay induces imbalance between the allacker and the trainer T: the boosted local fitting ability of T overpowers the attack ability of A that capable of memorizing the adversarial non-robust features that A can leverage.
Meanwhile, the red dashed line in Figure 2a implies, it turns out that these memorized non- robust features in training data do not generalize to improve test robustness, resulting in a large robust generalization gap (see Figure l). Next, we further reveal how such memorization harms test robustness by inducing robust overfitting.
3.2.1 How Imbalance Leads to Robust Overfitting
Under the imbalanced minimax game, the stronger trainer pushes up training robustness by memo- rizing the non-robust features in the training adversarial examples. However, the trainer's (overly) strong fitting ability actually learns false mappings of non-robust features which induce shorteuts for

---

test-time adversarial attack that lead to robustness degradation. Below, we explain how this happens during adversarial training, and an simpliiipl illustration is shown in Figure
1) Model Learns False Mapping of Non-robust Training Features after Decay. Consider a training adversarial example x' from class y that is misclassified to elass y' # y before LR decay. According to [21]. the non-robust feature g of x' belongs to class y', i.e., 9 E Fy- After LR decay, the trainer T becomes more capable of memorizing with smaller LR and it successfully learns to map this adversarial example x' to its correct label y. Accordingly, as we argued above, the non-robust feature g E Fy in z also (falsely) mapped to class y, L.e., (gg) = y.
2) False Non-robust Mapping Opens Shortcuts for Test-time Adversarial Attack. Now we consider clean test sample è whose true label is y'. Before LR decay, in order to misclassify it, the attacker A has to add non-robust features from a different class, say g' E FFg D' * y'. However, after LR decay, because of the existence of false non-robust mapping yc, the attacker A can simply add the non-robust feature from the same class, i.e., E Fy such that p(g) y, # y' to misclassify & to y. In other words, the injected non-robust features do not even have to come from a different class to misclassify 2, which are much easier for the attacker to find when starting from & (compared with non-robust features g' from other classes). Therefore, the attacker will have a larger attack success rate with the help of thesefalsely memorized non-robust features, as they create easy-to-find shortcuts for test-time adversarial attack.
Remark. To be rigorous, we give a formal definition for these non-robust features. Denote the hypothesis class of features that the model can robustly fit training data under configuration as H4 fo : X + | EezggrDerace
inf (y fo())) > 0,
(
(3)
where fo is robust training data, and 0 E θ is attainable under the configuration
We say a non-robust feature / is falsely memorized if f & Hheforee J E Hafter, where Hixefore: Hafter refer the hypothesis classes before and after LRR decay, respectively. other words, the non-robust is falsely memorized by the model (as robust feature) under the mismatch could happen because the trainer only sees training examples and it can overfit them under an imbalanced minimax game. Since this feature is essentially non-robust feature (in the population sense), this feature still behaves non-robust on test data and introduce shortcuts to test-time attack, as we elaborate above.
trainer. This
3.2.2 Verification
To further validate our explanation above, we design a series of experiments regarding the influence of memorization of the adversarial non-robust features after LR decay (Verification I & II). Guided by our explanation, we also observe some new intriguing phenomena of RO, which also help verify minimax perspective (Verification III & IV). Please refer to Appendix C.2 for more details.
Verification I: More Non-robust Features, Worse Robustness. To study the effect of non-robust features, we deliberately add non-robust features of different strengths to the clean data (a larger 8 indicates stronger non-robust features) and perform AT on these synthetic datasets. As shown in Figure 2b, we can see that 1) with I.R decay, stronger non-robust features induce severer test robustness degradation, and 2) it is much less severe without LR decay. This directly shows that the memorization of non-robust features induced by LR decay indeed hurts test robustness lot.
Verification II: Vanishing Target-class Features in Test Adversarial Examples. In our example above, the added non-robus, feature comes from the same class y' as ê, so the resulting adversarial example a' actually contains little features of class y. even misclassified to y. Figure 2c shows that the test adversarial examples generated after LR decay indeed have less and less target-class features as RO becomes more and more severe, which proves the shortcuts indeed exist to make test-time adversarial attack easier.
Verification III: Bilateral Class Correlation. Our theory suggests that if RO happens, the training misclassification from class + y/ (before decay) will induce an increase in test misclassification from class 1/ - y (after decay), as the y + y/ false non-robust mappings create y' + y shortcuts. We verify this in Figure 4a, in which we observe that this bilateral correlation indeed consistently increases after LR decay, while remaining very low (nearly no correlation) without decay.
'By target-class mean the class which adversarial example misclassified.
5

---

epoch 60
epoch 200
hefore docay
without decay
with LR decay
Truc Jabe
Trme Jobe)
04-
BE307
200
-00
200
500
1020
A [A-AF
Predicted label
Predicted label
456.00
A- 188.76
Predicted label
(a) bilateral class correlation
(b) confusion matrix symmetry
Figure 4: (a) Increasingly strong correlation between training-time y y misclassification and
test-time y + misclassification changes (due to RO). (b) Test-time confusion matrix of adversarial examples becomes symmetric with LR decay.
Eochh
Eooch
A080 Swoee 055
(a) diffferent LR decay factor
(b) different attack strength
Figure 5: (a) RO becomes increasingly severe as larger LR decay factor is adopted. (b) Varying the attack strength from extremely weak to extremely strong, the degree of RO first increases and then decreases after a turning point around 10/255.
Verification IV: Symmetrized Confusion Matrix. As a result of the bilateral correlation, we can deduce very interesting phenomenon that the + y' and y' - misclassification will have more similar error rates, which means that the confusion matrix of test robustness will become more symmetric. Indeed, as shown in Figure 4b, the confusion matrix (denoted as A) becomes much more symmetric (smaller distance between and A T) after LR decay than without LR decay.
3.3
A Holistic View of Robust Overfitting from the Minimax Game Perspective
Based on our understanding of RO through the lens of the minimax game, we give a holistic view of RO, revealing the influence of both the trainer and the attacker.
Trainer. We first fix the attacker strength to be PGD-10 attack with 8/255 and change the LR decay factor d from 1 (w/o LR decay) to 20 at epoch 100, and we observe that stronger LR decay induces monotonically increasing degree of RO (Figure 5a). This agrees well with explanation that smaller LR induces a severer imbalance and can memorize more non-robust features that eventually harm test robustness.
Attacker. We then fix the training configuration and vary the attack strength from PGD-0 with
0/255 (equivalent to standard training (ST)) to PGD-25 with 20/255.. Different from previous beliefs that RO will become severer with stronger attack [54], we find an intriguing inverted U curve in Figure 5b: the degree of RO firstly rises with stronger attackers, peaks at around 10/255, and decreases under even stronger attackers (e > 10/255). Our theory can naturally explain this phenomenon: 1) when using weak attacker (small e), training robustness is very high and there only a few non-robust features. Thus memorizing them only leads to slight or no RO: 2) a very strong attacker (e.g., 8 > 10/ (0/255) can also counter the strong fitting ability of the post-decay trainer, which also prevents the memorization of non-robust features and alleviates RO.
In summary, the conditions for RO are two folds: 1) the attacker A should be strong enough to generate enough non-robust features (which explains why AT overfits while ST does not); and 2) the trainer T should be stronger than A to memorize the non-robust features (which explains how LR decay induces RO). This understanding also suggests that we can alleviate RO by restoring the balance between the minimax players, explore in the next section.

---

Liel ball collne
(a) robustness comparison
(b) flatness comparison
(c) LR decay factor
(d) attack strength
Figure 6: Mitigating RO from three different techniques. (a. b) The purposed BoAT loss successfully
boosts model robustness and mitigates RO by bootstrapping loss landscape flatness between online and WA model. (c) Small LR decay factor leads to better best-epoch robustness of WA model with only slight RO. (d) Stronger training attacker also helps mitigate RO and boost robustness.
4 Mitigating Robust Overfitting by Rebalacing the Minimax Game
Based on our understanding of RO from the imbalanced minimax game perspective, we further investigate how to alleviate RO by restoring balance. In particular, we can either weaken the trainer's fitting ability (Sections 4.1), or strengthen the attacker's 's attacking ability (Section 4.2). We observe that strategies from both directions can prevent the model trainer T from fitting non-robust features too quickly and too adequately, thus can mitigate RO.
4.1 Trainer Regularization
From the trainer's side, one approach to rebalance the game is to regularize the trainer's local fitting ability after LR decay, e.g., by enforeing better landscape flatness. In this way, could prevent the learner from drawing complex decision boundaries and memorizing non-robust features. Among existing approaches targeting at this, weight averaging (WA) [22] taking the (moving) average of history model weights is effective and efficient that it brings neglectable training overhead, so we focus on WA in this work. Along the online model So that is adversarially trained, we also maintain a WA model fo that is an exponential moving average (EMA) of the online parameters X 5 7 40 (1 - ?) 6, where y E [o, 1] is the decay rate.
Bootstrapping. We notice that when the online model fo deteriorates, the WA model, although flatter [22], will eventually deteriorate
moving average of fo (Figure 6a). Inspired from the latent bootstrap mechanism in self-super learning [16], we align the predictions of f& and fo to improve the flatness of the online model fo simutaneously. Specifically, given an adversarial example x generated by PGD attack using vanilla AT loss (Eq. 1), we have
BAATTE y; 0) = (c=(Fo(z).))) + X KL. (Jo(2))(
where A is a coefficient balancing the CE loss (for better training robustness) and the KL regularization term (for better landscape flatness). In one direction, the term provides more fine- ane-grained supervision from the flatter WA model that helps avoid overfitting to the training labels [33, 8]. leading to better loss landscape flatness and robustness of the online model; in return, better online model will further improve the flatness of the WA model. Since this regularization encourages loss landscape flatness of both fo and fe in a bootstrapped fashion, we name Bootstrapped Adversarial Training (BoAT) loss.
We further verify the effectiveness of the proposed BoAT loss in Figure 6a, in which both the online model and the WA model suffer significantly less from RO and achieve higher best-epoch robustness compared with using the vanilla AT loss (see Appendix D.1 for details). Moreover, we compare their 1-D loss landscapes [25] at the last checkpoints, and Figure 6b demonstrates that both the online model and the WA model trained with our BoAT loss enjoy much better flatness, which implies weak memorization of non-robust features under the rebalanced game that explains the alleviation in RO.
Smaller LR Decay Factor. As studied in Section 3, L.R decay has a dramatic influence on RO and reducing the LR decay factor can largely ease RO (Figure 5a). Therefore, we also adopt smaller LR decay factor (together with the BoAT loss) to further regularize the trainer's fitting ability. To demonstrate its effecctiveness, even when we go back to the vanilla AT loss, Figure 6c shows the WA model obtained with a very small decay rate = 2 not only has very slight RO but also attains better best test robustness than d 10. We also remark how the WA model benefits from continuous supply of good individuals, which couldn't 't be guaranteed under strong RO, also throws new meaningful light on our effort in mitigating RO.
7

---

Table 1: Comparing our method with several training methods on CIFAR-10 under the perturbation norm cx 8/255 based on the PreActResNet--18 and WideResNet-34 10 architectures.
Pre
SedeResNNe-33-10
Method
Natural
PGD-20
AutoAttack final
Natural
PGD-20
AutoAttack final
best
final
best
final
best
best
final
best
final
best
PGD-AT
81.61 84.67
52.51 46.20
47.51
42.31
86.38
87.04
56.62 48.54 51.94 45.87
TRADES
80,45 82.89
52.32 49.39
48.09
46.40
84.23
84.89
56.02 47.38 52.76 45.62
WA
83,50 84.94 55,05 47.60
49.89
43.83
87.66 87.12
56.62 49.12 52.65 46.34
KD+SWA
84.06 84.48 53.70 53.68
49.82
49.37
87.45
88.21
56.29
55.76 53.59 53,55
PGD-AT+TE
82.04 82.59
54.98 53.54
50.12 49.09
85.77 85.97
56.42 53.40 52.88 50.56
AWP
81.11 80.62
55.60 55.03
50.09 49.85
85.63 85.61
58.95
59.05 53.32 53.38
MLCATwP WP
58.48
57.65
50.70
50.32
64.73 63.94 54.65 54.56
ReBAT
81.86 81.91
56.36
56.12
51.13
51.22
85.52 85.25
59.59
59.29 54.78 54.80
ReBAT++
78.71 78.85
56.68
56.70
51.49
51.39
81.65
81.59
59.81
59.88 54.80 54.91
WA+CutMix
80.24 80.30
56.02
55.95
49.67
49.59
85.08
87.95
60.41
59.36
55.11 53.60
ReBAT+CutMix 79.02 78.95 56.15 56.16
50.18 50.22
86.28
87.01
61.33
61.24
55.75 55.72
Table 2: Comparing our method with several training methods on CIFAR- 100 and Tiny- ImageNet under the perturbation norm Eo = 8/255 based on the PreAActResNet-IB 18 architecture.
CIFAR-10 PGD-20 final
Tiny-ImageNer PGD-20 best
Method
Natural
AutoAttack best final
Natural best
AutoAttack final
best
final
best
final
final
best
PGD-AT
55.94 56.39 29.74 22.47 24.84 19.80 45.29 48.70 21.86
17.54
17.29 14.19
TRADES
55.04 57.09
29.17 26.36 24.21 23.06
48.32
47.59
22.25 21.14
16.55 15.99
WA
57.26 58.40 30.35 23.52 25.83 20.51
49.56
49.35
24.24
19.30
19.47 15.56
KD+SWA
57.17 58.23 29.50 29.33 25.66 25.65
50.28
50.67
24.37
24.30
19.46 19.51
PGD-AT+TE
56.41 57.26
30.90 29.07 25.84 24.99
46.71 50.50
22.76
19.26
18.02 16.13
AWP
54.10 54.78
30.47 30.15 25.16 25.01
43.54
23.75
23.55
18.12 18.07
MLCAT
31.27
30.57 25.66 25.28
ReBAT
56.13 56.79
32.52
32.02 27.60 27.29
48.28
47.92
24.93
24.14
20.54 19.79
ReBAT++
52.44 54.65
32.49 32.11 27.49
27.33
45.10
44.12
25.23
24.52
20.67 20.51
WA+CutMix
56.73 57.17 31.70
31.71
26.43
26.08
46.48
46.48
24.80
24.72
18.87 18.87
ReBAT+
56.05 56.02 32.71 32.57
27.08
27.20
46.47
46.62
25.28
25.28
19.40 19.35
4.2
Stronger Training-time Attacker
From our minimax game perspective, beside directly restricting the fitting ability of the trainer T. another natural approach is to strengthen the attacker A's ability to counter it after LR decay to help strike a new balance between the two players. Therefore, we apply four different training perturbation 12/255, 16/255 (with fixed step size
budgets after decay, & 8/255, increasing PGD steps l ~ 10 (8/255)) where = 8/255 is the default budget used before LR decay and in test-time attack. As be seen from Figure 6d, the adopted stronger attacker indeed suppress RO for both the online and WA models, thanks to the hard-to-memorte non-robust features they attached the clean training data; and again we can see how WA benefits from the alleviation of RO, especially for ε = /2/ 122555 that even hits > 57% WA model robustness against PGD-20 attack (see also Appendix C.3). Besides, too strong training attacker may prevent the 'T from learning useful robust features (e.g., E = 16/255) that harms robustness, implying that we should carefully choose the attacker strength while rebalancing the minimax game via this approach.
2/255 and
adversarial
4.3 Overall Approach
Above, we have introduced three effective techniques to mitigate RO from different perspectives: model regularization, learning rate decay, and attacker strength. Since stronger attacker often induces a large decrease in natural accuracy [1] (we additionally study this trade-off in Appendix C.3) and needs more careful balancing in order not to harm robustness as discussed above. we design two
8

---

60-
54.5
33.0
PSD-IT
54.5
Fperh
Fpoth
Sperh
(a) training longer with CutMix
(b) target-class non-robust features
(c) bilateral class correlation
Figure 7: Empirical understandings on ReBAT. (a) ReBAT can benefit from training longer (with CutMix). (b) Compared PGD-AT, test adversarial examples still have to contain fairly many target-class non-robust features to successfully attack model trained with ReBAT. (c) ReBAT does not result strong bilateral class correlation as in PGD-AT.
versions of our final approach: ReBalanced Adversarial Training (ReBAT), which combines the BoAT loss with smaller LR decay factor; and ReBAT++, which combines all the three techniques.
5 Experiments
In this section, we evaluate the effectiveness of the proposed ReBAT and ReBAT++ on several bench- mark datasets using different network architectures. We consider the classification tasks CIFAR-IO, [10] with the PreActResNet- 18 [17] and WideResNet-344
CIFAR- 100 [23], and
[57] architectures. Besides the vamilla PGD-AT [30], we include the following baseline methods for alleviating robust overfitting: WA [22], KD+SWA [8], PGD-AT+TE [12], AWP [53]. WA+CutMix
[36] and MILCATwr wr [54]. During evaluation, we use PGD-20 [30] and AutoAttack (AA) [9] for the adversarial attack methods. We train each model for 200 epochs and record the checkpoint on which
the validation set achieves the highest PGD-20 robustness and the final checkpoint for evaluation on
the test set. Please refer to Appendix C.4 and C.5 for details and Appendix D for additional results.
5.1 Results on Benchmark Datasets
Performance across Different Networks. In Table 1, we evaluate ReBAT against previous AT variants on two popular backbone networks: PreActResNet-18 and WideResNet-34-10. We can see that without strong data augmentations like CutMix, ReBAT attains the best robustness on both models and outperforms all baseline methods against Auto Attack (AA), arguably the strongest white-box attacker. Mcanwhile, ReBAT almost shows no RO on both PreActResNet-IB 18 (51.13% best and 51.22% final AA robustness), and WidcResNeet34-110 (54.78% 78% best and 54.80% final AA robustness). For ReBAT++, it indeed results in even better best and final robustness than ReBAT on both models also with nearly perfect ability suppressing RO, though the cost of degradation in natural accuracy as discussed in Section 4.2. The results verify the effectiveness of our proposed method as well as the importance of rebalancing the minimax game.
Strong Augmentations. WideReeNee-344100 is more prone to overfit as it has larger capacity, and existing WA+CutMix strategy [36] still overfits by more than 1.5% (55.11% best and 53.60% final AA robustness). In comparison, ReBAT+CuuMi achieves 55.75% best robustness (chosen according to PGD) and 55.72% final robustness, showing that this combination indeed alleviates RO very effectively on the relatively large WideResNet-34-11 while maintaining very high robustness. In comparison, CutMix brings little improvement on PreActResNet-I5 18, mainly because this model has lower capacity that still underfits under strong augmentations, and it requires longer training to reach higher performance. Even though, ReBAT+CutMix still outperforms WA+CutMix by 0.63% AA robustness at the conclusion of 200 epochs of training.
Performance across Different Datasets. We also conduct experiments on two additional benchmark datasets, CIFAR-100 and Tiny-lmageNet, that are more complex than CIFAR-10. As shown in Table 2, our ReBAT, ReBAT++ and ReBAT+CutMix still achieve the highest best and final robustness, demonstrating its scalability to larger scale datasets. Similarly, CutMix does not help much as it often leads to underfitting with stronger regularization effects within relatively few training epochs.
5.2 Empirical Understandings
ReBAT Can Benefit from Longer Training. Table suggests that when CutMix is applied to a PreActResNet-IB 18 model, it demands longer training to fully unleash the beast. This motivates to

---

examine the effect of longer training, e.g. 500 epochs. As shown in Figure 7a, longer training with WA+CutMix brings little improvement on final robustness (49.59% (200 epochs) v.s. 49.70% (500 epochs) under AA). Instead, ReBAT+CutMix achieves higher best robustness and the robustness keeps improving along training. In particular, the additional 300 epochs improve the final AA robustness of ReBAA+CCtMix from 50.22% to 51 51.32%. This provides strong evidence on the effectiveness of ReBAT against RO and also shows that combining ReBAT with strong data augmentations can benefit from long training as in standard training.
Verification on Robust Overfitting Experiments. Beyond its success, we empirically study how ReBAT manages to suppress RO through our minimax game perspective. Following the experiments in Section 3.2.2, we compare PGD-AT and ReBAT in terms of 1) target-class non-robust features in test adversarial examples and 2) the strength of bilateral class correlation. We can see that the test adversarial examples still have to contain much target-class non-robust features to be misclassified (Figure 7b) and there is no longer strong bilateral class correlation (Figure 7c). This confirms that ReBAT indeed successfully prevents the model from learning false mappings of non-robust training features which creates shortcuts for test-time attack.
6 Revisting Previous Works from the Minimax Game Perspective
From dynamic minimax game perspective of AT, we build generic understanding of RO result of memorizing non-robust features under an imbalanced game, and introduce three strategies to rebalance the minimax game mitigate RO. In fact, we notice that this perspective can also enable a unified understanding that existing approaches to alleviate RO can be perceived as rebalancing techniques developed from the following three dimensions. Additional related works, including discussions on the game-theoretical views of adversarial training, are deferred to Appendix A.
Data Regularization. Since RO is induced by non-robust features in the training data, one way is to corrupt those features with various transformations. Since non-robust features often correspond to local and high-frequency patterns in images [21], stronger augmentations like CutMix [56] and IDBH [26] will help mitigate RO [38, 26). A complementary approach is to correct the supervision of the adversarial examples accordingly to avoid learning false label mapping (Section 3.2), e.g., the soft labels adopted in Dong et al. [11], and the distillation-alik methods in KD+SWA [8] and TE [12].
Training Regularization. As discussed in Section 3.3, necessary condition of RO is that the model trainer is stronger than the attacker, so restricting the trainer's ability in fitting non-robust features helps suppress RO. One approach is to directly promote model flatness, e.g., AWP [53]. MLCATwP [54], EMA [36]. SWA [8], AdvLC [27] and our BoAT loss (Section 4.... Another approach is to adjust the learning rate schedule to avoid dramatic increase in T's ability, which is previously studied by Rice et al. [38] and Wang et al. [45]. While RO is indeed delayed by some mild LR decay schedules attempted, it still happens as T eventually becomes strong. Particularly, we show that
simply piecewise decay with a small LR decay factor can be significantly helpful for both better robustness and less severe RO.
Stronger Training Attacker. Previous attempts in changing attacker strength, including Cai et al. [7], Qin et al. [35] and Wang et al. [49], mainly start training from a weaker attacker (e.g., 0), and gradually lift its strength to the standard level (= = 8/255), which is later pointed out by Pang et al. [33] that these methods are hardly useful when measured by AA robustness. OA-AT [1] introduces stronger attacker from the beginning but aiming to achieve robustness against stronger attacks without much concern in RO. Instead, our method is designed from minimax game perspective that adopts stronger training attacker (8 > 8/255/ after LR decay in order to counter RO. Our method shows clear benefits on improving best AA robustness well mitigating RO.
7 Conclusion
In this paper, we investigate the underlying cause of robust overfitting through dynamic minimax game perspective of adversarial training. Based on this perspective, we extend the robust feature framework to analyze the training dynamics of AT, and analyze how the change of LR decay induces an imbalance between the two players of AT. Specifically, we have developed new understanding of robust overfitting through the false memorization of non-robust features, and explored three new approaches to restore the balance between the model trainer and the attacker. Remarkably, the proposed ReBalanced Adversarial Training (ReBAT) have shown that adversarial training can also benefit from long training as long as the robust overfitting issue is resolved.
10

---

Acknowledgements
Yisen Wang was supported by National Key R&D Program of China (20222D0160304) National Natural Science Foundation of China (62006153, 62376010, 92370129), Open Research Projects of Zhejiang Lab (No. 2022RC0AB05), Beijing Nova Program (20230483344), and CCF-Baichuan-EB Fund. Zhouchen Lin was supported by the major key project of PCL, China (No. PCL2021A12). the NSF China (No. 62276004), and Qualcomm.
References
[1] Sravanti Addepalli, Samyak Jain, Gaurang Sriramanan, and R Venkatesh Babu. Scaling adversarial training to large perturbation bounds. In ECCV, 2022.
[2] Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. In ICML, 2018.
[3] Yang Bai, Yuyuan Zeng. Yong Jiang, Shu-Tao Xia, Xingjun Ma, and Yisen Wang. Improving adversarial robustmess via channel-wise activation suppressing. In ICLR, 2021.
[4] Maria-Florina Balcan, Rattana Pukdee, Pradeep Ravikumar, and Hongyang Zhang. Nash equilibria and pitfalls of adversarial training in adversarial robustness games. In AISTATS, 2023.
[5] Joey Bose, Gauthier Gidel, Hugo Berard, Andre Cianflone, Pascal Vincent, Simon Lacoste- Julien, and Will Hamilton. Adversarial example games. In NeurIPS, 2020.
[6] Samuel Rota Bulò, Battista Biggio, Ignazio Pillai, Marcello Pelillo, and Fabio Roli. Randomized prediction games for adversarial machine learning. IEEE transactions on neural networks and learning systems, 28(11):2466-24 2478, 2016.
[7] Qi-Zhi Cai, Min Du, Chang Liu, and Dawn Song. Curriculum adversarial training. In IJCAI, 2018.
[8] Tianlong Chen, Zhenyu Zhang, Sijia Liu, Shiyu Chang, and Zhangyang Wang. Robust overfitting may be mitigated by properly learned smoothening. ICLR, 2021.
[9] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In ICML, 2020.
[10] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
[11] Chengyu Dong, Liyuan Liu, and Jingbo Shang. Double descent in adversarial training: An implicit label noise perspective. arXiv preprint arXiv:21 10.03135, 2021.
[12] Yinpeng Dong, Ke Xu, Xiao Yang. Tianyu Pang, Zhijie Deng, Hang Su, and Jun Zhu. Exploring memorization adversarial training. ICLR, 2022.
[13] Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Brandon Tran, and Aleksander Madry. Adversarial robustness prior for learned representations. arXiv preprint arXiv:1906.0094/ 00945. 2019.
[14] Liam H Fowl, Micah Goldblum, Ping-yeh Chiang, Jonas Geiping, Wojeiech Czaja, and Tom Goldstein. Adversarial examples make strong poisons. In NeurIPS, 2021.
[15] Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar- ial examples. In ICLR, 2015.
[16] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent-a new approach to self-supervised learning. In NeurIPS, 2020.
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings deep residual networks. In ECCV, 2016.

---

[18] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint 1XXiv:5503.02533 2015.
[19] Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. In NeurIPS, 2017.
[20] Hanxun Huang, Xingjun Ma, Sarah Monazam Erfani, James Bailey, and Yisen Wang. Unlearn- able examples: Making personal data unexploitable. In ICLR, 2021.
[21] Andrew llyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In NeurIPS, 2019.
[22] Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv.1803.05407, 2018.
[23] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[24] Ang Li, Yifei Wang, and Yisen Wang. Adversarial examples are not real features. In NeurIPS, 2023.
[25] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. NeurlPS, 2018.
[26] Lin Li and Michael Spratling. Data augmentation alone can improve adversarial training. In ICLR, 2023.
[27] Lin Li and Michael Spratling. Understanding and combating robust overfitting via input loss landscape analysis and regularization. Pattern Recognition, 2023.
[28] Yuanzhi Li, Colin Wei, and Tengyu Ma. Towards explaining the regularization effect of initial large learning rate in training neural networks. In NeurlPS, 2019.
[29] Rundong Luo, Yifei Wang. and Yisen Wang. Rethinking the effect of data augmentation in adversarial contrastive learning. In JCLR, 2023.
[30] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018.
[31] Yichuan Mo, Dongxian Wu, Yifei Wang, Yiwen Guo, and Yisen Wang. When adversarial training meets vision transformers: Recipes from training to architecture. In NewrIPS, 2022.
[32] Ambar Pal and René Vidal. A game theoretic analysis of additive adversarial attacks and defenses. In NeurlPS, 2020.
[33] Tianyu Pang, Xiao Yang, Yinpeng Dong, Hang Su, and Jun Zhu. Bag of tricks for adversarial training. In ICLR, 2020.
[34] Rafael Pinot, Raphael Ettedgui, Geovani Rizk, Yann Chevaleyre, and Jamal Atif. Randomization matters how to defend against strong adversarial attacks. In JCML, 2020.
[35] Chongli Qin, James Martens, Sven Gowal, Dilip Krishnan, Krishnamurthy Dvijotham, Alhus- sein Fawzi, Soham De, Robert Stanforth, and Pushmeet Kohli. Adversarial robustness through local linearization. In NeurlPS, 2019.
[36] Sylvestre-Alvise Rebuffi, Sven Gowal, Dan Andrei Calian, Florian Stimberg, Olivia Wiles, and Timothy A Mann. Data augmentation can improve robustness. In NeurIPS, 2021.
[37] Jie Ren, Die Zhang, Yisen Wang, Lu Chen, Zhanpeng Zhou, Yiting Chen, Xu Cheng, Xin Wang. Meng Zhou, Jie Shi, et al. A unified game-theoretic interpretation of adversarial robustness. In NeurlPS, 2021.
[38] Leslie Rice, Eric Wong, and Zico Kolter. Overfitting adversarially robust deep learning. In ICML, 2020.
12

---

[39] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018.
[40] Shibani Santurkar, Andrew Ilyas, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry, Image synthesis with a single (robust) classifier. In NeurIPS, 2019.
[41] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
[42] David Stutz, Matthias Hein, and Bernt Schiele. Relating adversarially robust generalization to flat minima. In ICCV, 2021.
[43] Christian Szegedy, Wojeiech Zaremba, llya Sutskever, Joan Bruna, Dumitru Erhan, lan Good- fellow, and Rob Fergus. Intriguing properties of neural networks. In ICLR, 2014.
[44] Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. In ICLR, 2018.
[45] Hongjun Wang and Yisen Wang. Self-ensemble adversarial training for improved robustness. In ICLR, 2022.
[46] Hongjun Wang and Yisen Wang. Generalist: Decoupling natural and robust generalization. In CVPR, 2023.
[47] Yifei Wang, Dan Peng, Furui Liu, Zhenguo Li, Zhitang Chen, and Jiansheng Yang. Decoder-free robustness disentanglement without (additional) supervision. arXiv preprint arXiv:2007.013: 200.00356... 2020.
[48] Yifei Wang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. A unified contrastive energy- based model for understanding the generative ability of adversarial training. ICLR, 2022.
[49] Yisen Wang, Xingjun Ma, James Bailey, Jinfeng Yi, Bowen Zhou, and Quanquan Gu. On the convergence and robustmess of adversarial training. In ICML, 2019.
[50] Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving adversarial robustness requires revisiting misclassified examples. In ICLR, 2020.
[51] Zhirui Wang. Yifei Wang. and Yisen Wang. Fooling adversarial training with inducing noise. 2021.
arXiv preprint arXiv:2
[52] Zeming Wei, Yifei Wang, Yiwen Guo, and Yisen Wang. Cfa: Class-wise calibrated fair adversarial training. In CVPR, 2023.
[53] Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust generalization. In NeurlPS, 2020.
[54] Chaojian Yu, Bo Han, Li Shen, Jun Yu, Chen Gong, Mingming Gong, and Tongliang Liu. Understanding robust overfitting of adversarial training and beyond. In ICML, 2022.
[55] Chaojian Yu, Dawei Zhou, Li Shen, Jun Yu, Bo Han, Mingming Gong, Nannan Wang, and Tongliang Liu. Strength-adaptive adversarial training. arXiv preprint arXiv:2210.01288. 2022.
[56] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV,20 2019.
[57] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.
[58] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. In ICLR, 2017.
[59] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In ICMI., 2019.
13

---

A Additional Related Work
A.1
Game-Theoretical View of Adversarial Training
There is a body of literature that analyzes AT' from a game theory perspective, see [6, 5, 37, 4]. However, existing theory papers only analyze AT's Nash equilibrium under toy models (e.g., Gaussian features and linear models), and static minimax players. None of the existing literature considers a dynamic game and could explain robust overfitting from game perspective, particularly in practical AT algorithms, as done in our work. Nevertheless, we find some interesting connections and new insights between them that worth noting.
Under these simplified assumptions, prior works show that although the Nash equilibrium of the AT objective exists and is robust, the current alternating optimization of AT may fail to converge to the Nash equilibrium, see, c.g., a recent work [4]. The key reason is that the trainer can falsely fit the non-robust features [4], which is in similar spirit our analysis that robust overfitting caused by the falsely memorized non-robust features after LR decay. This shows that our explanation is not only in line with the cutting-edge theory of AT, but also further explains robust overfitting from a game perspective for the first time. Besides, we also notice that existing AT game theory papers cannot explain robust overfitting, and from the perspective of our theory, this is possible because they overlook the dynamic change of the AT players. Thus, our understanding of AT from a dynamic game can inspire more in-depth theoretical characterization of AT from the game theory perspective.
As our work mainly focuses on the understanding robust overfitting of practical AT algorithms, and is generally hard to theoretically analyze the training dynamics of practical DNNs, we leave more theoretical investigations under strong assumptions to future work.
A.2 Robust and Non-robust Feature View of Adversarial Training
Ilyas et al. [21] firstly propose to understand the existence of adversarial examples from a data feature perspective (introduced in Section 2). Viewing from this framework, adversarial training discards the useful yet non-robust features contained in natural images and only utilizes the robust features, which also explains the well-known trade-off between accuracy and robustness [59, 44, 46). In particular, Ilyas et al. [21] show that robust features are generally aligned with human perception while non-robust features seem random noise. Built upon this observation, researchers explore utilizing adversarial training for representation learning [13, 47] and even image synthesis [40, 48]. This robust feature perspective of adversarial training also inspires various "adversariall-or-gooo""" applications, such as, privacy protection [20, 51].
Despite these fruitful developments from this view, concurrent work [24] challenges this com- mon understanding by showing that non-robust features are not really useful. In particular, when transferred, non-robust features yield significantly worse performance than robust features under self-supervised learning, contrary to their behaviors under supervised learning. Meanwhile, they also show that robust features alone might not be enough to attain robustness (in practice), since the robust dataset constructed in Ilyas et al. [21] has a false sense of robustness under AutoAttack [9]. Based on these observations, they suggest that non-robe non-robust features should be considered as natural backdoor patterns of the supervised learning task alone, instead of universally useful across different learning paradigms. We note that this understanding is actually not contrary to our explanation of robust overfitting through the non-robust features, since only need the usefulness of non-robust features for classification in our discussion. In Section 3.2, we have also mentioned that the falsely memorized non-robust features behave as learning shortcuts, which agrees with the analysis in Li et al. [24].
A.3 Dynamic Strategies for Adjusting the Trainer and the Attacker in Adversarial Training
In Secion 4, we propose three new strategies to adjust the trainer and the attacker in order to restore the balance of the minimax game, Prior to our work, there existed many proposed strategies to adaptively adjust the trainer or the attacker during adversarial training. For example, Rice et al. [38] show that different schedules of learning rate has large effect on robust overfitting and the best-epoch robustness, and they propose early stopping to prevent robust overfitting. Wang and Wang [45] instead explore a combination of weight decay and learning and learning rate schedule and obtain better performance. On sulf-supervised learning, Luo et al. [29] show that a dynamic change
14

---

of data augmentation strength can improve robustness significantly. On the other side. Wang et al. [49] propose to dynamically increase the attack steps to attain better convergence, while Mo al. [31] further apply the dynamic strategy in ViTs. Concurrent to our work, Yu et al. [55] and Wei et al. [52] also empirically find that using a large perturbation budget can mitigate robust overfitting to some extent. In our work, this strategy naturally rises from our minimax game perspective of adversarial training. Compared to sophisticated adaptive designs, our method is simpler to use and delivers better robustness: we only improve the attack strength by constant after the learning decay.
B Additional Experiment: Adversarial Examples (in Adversarial Training) Contain Adversarial Non-robust Features
In Section 3, we collect misclassified adversarial examples on an early checkpoint A and evaluate them on a later checkpoint B with their correct labels to examine the memorization of those adversarial non-robust features. This is because adversarial examples used for AT consist of robust features from the original label y and non-robust features from the misclassified label ÿ. so B must have memorized those features (non-robust A) to correctly classify them. In this section, we further provide rigorous discussion on the non-robust features contained in the adversarial examples through an additional experiment.
Experiment Design. Recall that Ilyas et al. [21] leverage targeted PGD attack [30] to craft adversarial examples on standard (non-robust) classifier and relabel them with their target labels to build non-robust dataset. Finding standard training on it yields good accuracy on clean test data, they prove that those adversarial examples contain non-robust features corresponding to the target labels (Section 3.2 of their paper).
clean training data
relabel as cat
evaluate natural accuracy
(untargeted) PGD attack
standard training
good accuracy (> 55%)
dog
cat
cat
robust features: dog non-robust fentures: dog
robust features: dog non-robust features: cat
robust features: call non-robust features: cat
Figure 8: Mining non-robust features from adversarial examples in adversarial training. 1) Craft adversarial examples á; using untargeted PGD attack on a AT checkpoint. 2) Relabel them (only misclassified ones) with misclassified label iA to build a non-robust dataset { 9.)). 3) Perform standard training from the checkpoint. 4) Achieve good natural accuracy (> 55%).
However, different from their settings where we know the target labels of the adversarial examples and they are generated on a non-robust classifier, we lay emphasis on mining non-robust features from adversarial examples generated on-the-tly fly in AT. To this end, we first craft adversarial examples on the AT checkpoint at some epoch using untargeted PGD attack [30] following the real setting of AT. then relabel the misclassified ones 2; (e.g.. g., dog) with their misclassified labels ÿ: * yi (e.g., cat) build non-robust dataset {( ú.)}. In order to capture non-robust features at exactly the training epoch these adversarial examples are used, we continue to perform standard training directly from the AT checkpoint on the non-robust dataset, and finally evaluate natural accuracy. See Figure 8 for an illustration of our experiment.
In details, we select the AT models at epoch 60 (before LR decay) and epoch 1,000 (after LR decay)
to conduct the above experiment. We use untargeted PGD-20 with perturbation norm E oo 16/255 to craft adversarial examples on those checkpoints from the training data of CIFAR- -10 [23]. The attack we adopt is a little bit stronger than the attack of PGD-10 with & 8/255 that is commonly used generate adversarial examples in baseline adversarial training, because the training robust accuracy rises to as high as 94.67% at epoch 1,000 (Figure la) and gives less than 2,700 misclassified examples, which is significantly insufficient for further training. Using the stronger attack, we obtain a success rate of 78.38% on the checkpoint at epoch 60 and a success rate of 34.86% on the checkpoint at epoch 1,000. Since the attack success rates are different, we randomly select a same number of misclassified adversarial examples for standard training for fair comparison. The learning rate is
15

---

initially set to 0.1 and decays to 0.01 after 20 epochs for another 10 epochs of fine-tuning. Given that the original checkpoints already have non-trivial natural accuracy, we also add two control groups that train with random labels instead of y to exclude the influence that may brought by the original accuracy.
80
70
Natural accuracy (%)
60
80
epoch 50 (misclassified labels)
30-
1,000 (misclassrfied labels)
epoch 60 (random labels)
(random
Epoch (ST)
Figure 9: Natural accuracy during standard training. Standard training the non-robust dataset built from the checkpoint at either epoch 60 or epoch 1,000 converges to fairly good natural accuracy (> 55%). The failure of training with random labels proves that the good accuracy has nothing to do with the original natural accuracy of the AT checkpoint.
Results. As shown Figure 9, find that standard training on the non-robust dataset {( ût) successfully converges to fairly good accuracy (> 55%) natural test images, ie., predicting cats as cats, no matter from which AT checkpoints (either epoch 60 or epoch 1,000) the adversarial examples are crafted. Also, we can see that the non-trivial natural accuracy has nothing to do with the original accuracy of the AT checkpoints, soon as the standard training starts with random labels. This proves that adversarial examples in adversarial truining do contain non-re non-robust features w.r.f. the classes to which they are misclassifedd, which strongly corroborates the validity of the experiments in Section 3.
the accuracy plummets around 10% (random guessing) as
Discussion on the Influence Brought by Robust Features During Standard Training. Since untargeted PGD attack cannot be assigned with a target label, we camnot guarantee that the misclassi- fied labels be uniformly distributed regardless of the original labels (especially for real-world datasets). This implies that the non-robust features are not completely decoupled from robust features, i.e., training on {( út)} may take advantage ft; robust features from y/ through the correlation between y, and ú.. However, we argue that robust features from Y; mingling with non-robust features from ii only increases the difficulty of obtaining a good natural accuracy. This is because learning through the shorteut will only wrongly map the robust features from to ÿJ that never equals to yi- during the evaluation, each clean test example x; will always have robust and non-robust features from yı, and such wrong mapping will induce x, to be misclassified to some due to the label of its robust features. As a result, despite the negative influence brought by robust features, we still achieve good natural accuracy at last, which further solidifies our conclusion.
C Experiment Details
In this section, we provide more experiment details that are omitted before due to the page limit.
C.1 Baseline Adversarial Training
In this paper, we mainly consider classification task on CIFAR-10 [23]. The dataset contains 60,000 32 32 RGB images from 10 classes. For each class, there are 5,000 images for training and 1,000 images for evaluation. Since we mainly aim track the training dynamic to verify our understandings RO instead of formally evaluating an algorithm's 's performance, we do not hold validation set in most of our verification experiments and directly train the full training set.
16

---

Protton BONE
Postood Miel
The kabel
Tree label
The lahel qpack
Trae label
Toue lubel spech
epoch
spach 40
epech 50
Fedd lake
Predsned LOXE
Prodeced
1pocs
Truc line
spech 120
epech 140
spaca 160
epech 150
spect. 200
Figure 10: More test-time confusion matrices during the first 200 epochs of the training. After LR decays (the second row), the confusion matrix A immediately becomes symmetrie, as the spectral norm || A AT w.r.t. the matrix decreases from 2 350 before epoch 100 to S 150 after epoch 200.
4000
X00
32C
2C.0
24
XPC
257
TAYS
3000
31
26
300
2000
22C
800
2TL
67C
SBC
2000
LSA
15000
S7C
220
1000
300
D1B
116
A7
56
3DC
344
+2
24
P0
P3
Producted labe
Predicted
(a) train
(b) test
Figure 11: Confusion matrices of the training and the test data at an epoch before robust overfitting
starts. They show nearly same pattern of attacking preference among classes.
For baseline adversarial training, We use PreActResNet-188 [17] model as the classifier. We use PGD-10 attack [30] with step size = 2/255 and perturbation norm &oo 8/255 to craft adversarial examples on-the-fly. Following the settings in Madry et al. [30], we use SGD optimizer with momentum 0.9, weight decay 5 10-4 and batch size 128 to train the model for many as 1,000 epochs. The learning rate (LR) is initially set to be 0.1 and decays to 0.01 at epoch 100 and further decays 0.001 at epoch 150. For the version without LR decay used for comparison in our paper, we simply keep the LR to be 0.1 during the whole training process.
Each model included in this paper is trained on single NVIDIA GeForce RTX 3090 GPU. For PGD-AT, it takes about 3d 14h to finish 1,000 epochs of training.
C.2 Verification Experiments for Our Minimax Game Perspective on Robust Overfitting
Memorization of Adversarial Non-robust Features After LR Decay. We craft adversarial examples on a checkpoint before LR decay (60th epoch) and evaluate the misclassified ones on checkpoint after LR decay (2 150th epoch) with their correct labels to evaluate the memorization of non-robust features in the training adversarial examples (see Appendix for detailed discussions) in Section 3.1 and 3.2. Following Appendix B, we adopt PGD-20 attack with perturbation norm & 16/255 to craft adversarial examples which is stronger than the common attack setting we use in PGD-AT. We note that test adversarial examples crafted by a stronger attack indicates stronger extraction of the non-robust features, so they are more indicative of non-robust feature memorization when still correctly classified.
17

---

Verification I: More Non-robust Features, Worse Robustness. At the beginning of Section 3.2, we create synthetic datasets to demonstrate that memorizing the non-robust training features indeed harms test-time model robustness. To instill non-robust features into the training dataset, we minimize the adversarial loss w.e.. the training data in a way that just like PGD attack, with the only difference that we minimize the adversarial loss instead of maximizing it. Since we only use a very small perturbation norm Eso I 4/255, the added features are bound to be non-robust For a fair comparison, we also perturb the training set with random uniform noise of the same perturbation norm exclude the influence brought by (slight) data distribution shifts. We continue training from the 100-th baseline AT checkpoint (before LR decay) each synthetic dataset for 10 epochs, and then evaluate model robustness with clean test data.
Verification I: Vanishing Target-class Features in Test Adversarial Examples. This is to say that when RO happens, we expect that test adversarial examples become less informative of the classes to which they are misclassified according to our theory. To verify this, we first craft adversarial examples on a checkpoint T after RO begins, then evaluate the misclassified ones with their misclassified labels y on the checkpoint saved at epoch 60. As result, the accuracy reflects how much information (non-robust features) from ý the adversarial examples have to contain to be misclassified to 9 on T. All adversarial examples evaluated in the experiments in Section 3.2.2 are crafted using PGD-10 attack with perturbation norm coo = 8/255.
Verification III: Bilateral Class Correlation. To quantitatively analyze the correlation strength of bilateral misclassification described in Section 3.2.2, we first summarize all y; + yj misclassification rates into two confusion matrices ptrain ptest E RCCC for the training and test data, respectively. Because we are mainly interested in the effect of the LR decay, we focus on the relative change on the test confusion matrix before and after LR decay, i.e., A ptestt to our theory, for each class pair (i.j), there should be a strong correlation between the training misclassification of i + j before LR decay, ie., (Pbelore)ss and the increase in test misclassification of - i, i.e., P as i + j training misclassification (may due to intrinsic class bias, as will be further discussed below in details) induces i + j false mappings and creates j - shortcuts. To (tppumppping examine their relationship, we plot the two variables ((Phrain before) iij, A Pfeat) and compute their Pearson correlation coefficient p.
Ptest. Ptest According
Verification IV: Symmetrized Confusion Matrix. In Section 3.2.2, we mention the growing symmetry of the test-time confusion matrix after LR decay as an evidence of the strengthening + 3' and y' + correlation. Here we present more confusion matrices during the 200 epochs of the training in Figure 10, and it is very clear that the confusion matrices soon become symmetric after LR decay and RO starts. For a deeper comprehension of this phenomenon, we first visualize the confusion matrices of the training and the test data at an epoch before RO starts Figure 11. They exhibit nearly same pattern of attacking preference among classes (e.g., - y²) due to the bias rooted in the dataset, e.g., class 6 is intrinsically vulnerable this case. For the test data, this intrinsic bias wouldn't 't be wiped out through learning due to the -onrrennllliiabbiityy of the memorization of non-robust features in the training data, as discussed at the beginning of Section 3.2 (i.e., y + y bias still holds); and for the training data, this biased feature memorization will open shortcuts for test- time adversarial attack as discussed in Section 3.2 (i.e,, y' - y begins). Combining both y - y and g' - y, we arrive at the symmetry of test-time confusion matrix.
Additional Results: Changing LR Decay Schedule. Our discussion above is based on the piecewise LR decay schedule, in which the sudden decay of LR most obviously reflects our understandings. Besides, we also explore other LR decay schedules, including Cosine/L.inea LR decay, to check whether different IR decay schedules will affect the observations and claims we made in this paper. For each schedule, we train the model for 200 epochs following the settings in Rice et al. [38]. As demonstrated in Figure 12a, we arrive at the same finding as Rice et al. [38] that with Cosine/Linear LR decay schedule, the training still suffers from severe RO after epoch 130. Then, we rerun the empirical verification experiments in Section 3.2.2 and find that under both the two LR decay schedules 1) the test adversarial examples indeed contain less and less target-class non-robust features as the training goes and RO becomes severer and severer (Figure 12b), 2) the bilateral class correlation becomes increasingly strong (Figure 12c) and 3) the confusion matrix indeed becomes symmetric (Figure 13). The results are almost the same as the results achieved when we adopt piecewise L.R decay schedule because even though these LR decay schedules are mild, the LR eventually becomes small and makes the trainer T overly strong memorize the harmful non-robust features, indicating
18

---

that our understandings in the cause of RO is fundamental and regardless of whatever LR decay schedule used.
Coeree Linuar
Coure Linsar
Codfe Linear
Rabust sccintt 6%
-0-
Seeroon connection Conf
2.3
8
0
0.2
100
150
200
130
140
150
160
170
160
150
200
140
155
160
170
160
150
200
1p301
(a) test robust accuracy during train- (b) target-class non-robust features
(c) bilateral class corretation
ing
Figure 12: Empirical verification of our explanation for robust overfitting when Cosine/Linear LR decay schedule is applied. (a) With Cosine/Linear LR decay schedule, the training still suffers from severe RO. (b) After RO begins, non-robust features in the test data become less and less (c) Increasingly strong correlation between
informative of the classes to which they are misclassifidd. training-time time y y misclassification and test-time y' + y misclassification increase.
epoch
epoch 200
before LR decay
Cosine
Linear
Trie labe!
True label
Predicted label a| AT la2 477.42 A 11- 132.12
Predicted label
Predicted label 164.18
||A
Figure 13: Test-time confusion matrix also becomes symmetric and implies that the bilateral correla- tion also exists when Cosine/Linear LR decay schedule applied.
Table 3: Training with stronger attack and evaluating model robustness on CIFAR-10 under the perturbation norm = 8/255 based on the Pretesssee 18 architecture.
Natural
PGD-20
AutoAttack final diff
Attack Strength
best
final
diff
best
final
diff
best
8/255, PGID-10 (baseline)
83.50 84.94 -1.44
55.05
47.60 7.45 49.89 43.83 6.06
E 10/255, PGD-12
80.66 82.48 -1.82
56.63
50.63 6.00
50.89 46.13 4.76
12/255, PGD-15
78.17 80.25 -2.08
57.09
53.13 3.96
50.99
47.66 3.33
E 14/255, PGD-17
73.92 76.70 -2.78
56.42
54.04 2.38
50.28
48.53 1.75
E 16/255, PGD-20
69.51 73.11 -3.60
55.09
54.27 0.82
49.58
48.53 1.05
C.3 Experiments the Effect of Stronger Training Attacker
In Section 4.2, we point out that using a stronger attacker in AT is able to mitigate RO to some extent by neutralizing the trainer T's fitting power when it overly strong. To achieve the results reported in Figure 6d, we craft adversarial examples on-the-fly with more PGD iteration steps when is larger (see Table 3), and further evaluate the best and last robustness of the WA models against PGD-20 and AA. Although RO is only partially mitigated and natural accuracy decreases when a stronger attacker is applied as summarized in Addepalli et al. [1], it may be surprising to find from Table that an attacker of appropriate strength may significantly boost the best WA robustness. This suggests using a stronger attack could potentially be an interesting new path to stronger adversarial defense, and we leave it for future work
19

---

C.4 Detailed Experiment Setup of ReBAT for Mitigating Robust Overfitting
Datasets. Beside CIFAR- 10, we also include CIFAR-100 [23] and Tiny- Inay-eet [10] for evaluation of the effectiveness of ReBAT. CIFAR-100 shares the same training and test images with CIFAR- 10, but it classifies them into 100 categories, Lee, 500 training images and 100 test images for each class. Tiny-ImageNet is subset of ImageNet [10] which contains labeled 64 x 64 RGB images from 200 classes. For each class, it includes 500 and 50 images for training and evaluation respectively. training set,
Following Rice et al. [38], we hold out 1,000 images from the original CIFAR-10/
and similarly 2,000 images from the original Tiny-ImageNet training set as validation sets.
Training Strategy. For CIFAR-10 and CIFAR-100, we follow exactly the same training strategy as introduced in Appendix C.I, except that for ReBAT++ we adopt PGD- 12 with perturbation norm Exc 10/255 for training after LR decay. For Tiny-ImageNet, we follow the learning schedule of Chen al. [8]. in which the model is trained for total of 100 epochs and the LR decays twice (by 0.1) at epoch 50 and 80.
Choices of Hyperparameters. For KD+SWA [8], PGID-AT+TE [12], AWP [53] and WA+CutMix [36], we strictly follow their original settings of hyperparameters. For MLCATwP [54], we simply report the test results reported their paper. Following Chen al. [8], SWA/WA as well as the ReBAT regularization purposed in Section 4.1 start at epoch 105 (5 epochs later than the first LR decay where robust overfitting often begins), and following Rebuffi et al. [36] we choose the EMA decay rate of WA to be y = 0.999. Please refer to Table 4 for our choices of the decay factor d and regularization strength A. We notice that since CutMix improves the difficulty of learning, the model demands a relatively larger decay factor to better fit the augmented data. For Imn-IImaeNNet, we also apply a larger after the second LR decay to better maintain the flatness of adversarial loss landscape and control robust overfitting. We provide more discussions on the choice of hyperparameters in Appendix D.I.
Table 4: Choices of hyperparameters when training models on different datasets using different network architectures with ReBAT.
Network Architecture Method
CIFAR-IO/CIFAR 100
TingeImaeeNet
ReBAT
d 1.5,A 1.0
d= 4.0, 2.0, A2 10.0
18
ReBAT+
d= 1.7, 1.0
d 4.0, \ 2.0, = 10.0
ReBAT+CutMix
d= 4.0, 2.0
ReBAT
d = 1.3, = 0.5
NideResNNet33-110
ReBAT+
d = 1.3, À -055
ReBAT+CutMix
d 4.0. A -2.0
C.5 Training Robust Accuracy
Figure 14a shows the robust accuracy change on the training data during the training. Compared with vanilla PGD-AT that yields training robust accuracy over 80% at epoch 200, ReBAT manages to suppress to only 65%. It successfully prevents the trainer TT from learning the non-robust features w.r.t. the training data too fast and too well, and therefore significantly reduces the robust generalization gap (from 35% to 9%) and mitigates RO.
C.6 Training Efficiency
We also test and report the training time (per epoch) of several methods evaluated in this paper. For fair comparison, all the compared methods are integrated into a universal training framework and each test runs single NVIDIA GeForce RTX 3090 GPU.
From Table 5, we can see that ReBAT requires nearly no extra computational cost compared with vanilla PGD-AT in each epoch (136.2s v.s. 131.6s), implying that it is an efficient training method in practical. As we regularize the model trainer to restore the balance of the minimax game, we can see from Figure 14b that the training takes slightly longer to attain optimal performance, e.g., 110 epochs (PGD-AT) and 170 epochs (ours). However, in practice, people usually train longer AT (typically 200 epochs) to attain better accuracy, and use early stopping to select the best checkpoint. As our
20

---

PGD-AT
60
PGD-AT
ReBAT
55
RoBAT
Robust Accuracy (%)
Robust Accuracy
70
50
45
60
40
30
40
160
Epocn
Epoch
(a) training
(b) test
Figure Training and test robust accuracy of PGD-AT and ReBAT on CIFAR-10 10 under the perturbation norm Sxo 8/255 based on the PreActResNet-18 18 architecture. The gray dashed lines in (a) indicate LR decays, and the colored dashed lines (b) indicate the epochs where each method attains its best robustness, respectively.
Table 5: Combining training time per epoch on CIFAR-10 under the perturbation norm Egc = 8/255 based on the PreActResNet-118 architecture.
Method
Training Time per Epoch (s)
PGD-AT
131.6
WA
132.1
KD+SWA
131.6+16.5+1411
AWP
142.8
MLCAT wp
353.3
ReBAT
136.2
WA+CutMix
168.6
ReBAT+CutMix
173.1
method does not need early stopping techniques (almost no robust overfitting), and have neglectable computation overhead per epoch, the total training time is comparable with vanilla AT. We also remark that KD+SWA, one of the most competitive methods that aims to address the RO issue, not really computationally efficient it requires to pretrain robust classifier and non-robust as AT teacher and ST teacher respectively.
D More Experiments on ReBAT
In this section, conduct extensional experiments the proposed ReBAT method to further demonstrate its effectiveness, efficiency and flexibility.
D.1 Additional Results on BoAT Loss
In Section C.4, we discuss the detailed configurations for the experiments in Figure 6a, where show that BoAT can largely mitigate robust overfitting. Here, we further summarize the performance of best and final checkpoints of the original AT+WA method and our BoAT. As shown in Table 6, BoAT not only boosts the best robustness by large margin (0.67% higher against AA) but also significantly suppresses RO (1. (1.58% v.s. 6.06% against AA). To achieve the reported robustness, first use Å₁ = 10.0 after the first LR decay and then apply X = 60.0 after the second LR decay to better maintain the flatness of adversarial loss landscape and control robust overfitting.
D.2 Additional Results the Effeet of LR decay
Below we show that when a relatively large decay factor dl applied, i.e., the model has overly strong fitting ability that results in robust overfitting, large regularization coefficient À should be choosen
21

---

Table 6: Comparing model robustness w/ and w/o BoAT loss on CIFAR-10 under the perturbation
norm Ex 8/255 based on the PreActResNet-18 architecture.
Natural
PGD-20
AutoAttack final
Method
best
final
diff
best
final
diff
best
diff
AT+
83.50 84.94
-1.44
55.05
47.60
7.45
49.89
43.83 6.06
ReBAT(d = 10) 81.54
82.42
-0.88
55.29
53.43 1.86
50.56
48.98 1.58
for better performance. Table 7 reveals this relationship between d and A. When d = 1.3, even a X as small as 1.0 will harm both the best and last robustness as well as natural accuracy, as d 1.3 is already too small decay factor that makes the model suffering from underfitting and naturally requiring no more flatness regularization. On the other side, when is relatively large, even a strong regularization of X = 4.0 is not adequate to fully suppress RO. Besides, comparing the situation of and = for a fixed d(2 1.5), we emphasize that the purposed BoAT loss again exhibits its apparent effectiveness in simultaneously boosting the best robustness and mitigating RO.
Table 7: Changing decay factor d and regularization strength À and evaluating model robustness on CIFAR-10 under the perturbation norm Eoo = 8/255 based on the PreActResNet-I1 18 architecture.
Natural
PGD-20
AutoAttack
Method
best
final
diff
best
final
diff
best
final
diff
ReBAT(d = 1.3, À = 0.0) 81.17 81.27
-0.10
56.43
56.23 0.20
50.80
50.75 0.05
ReBAT(d 1.3, À 1.0)
80.67 80.63
0.04
56.27
56.15 0.12
50.74
50.65 0.09
ReBAT(d = 1.3, À = 4.0) 78.50 78.36
0.14
55.10
55.04
0.06
50.31
50.30
0.01
ReBAT(d = 1.5, A 0.0) 81.90 82.39
-0.49
56.21
55.95
0.26
50.81
50.58
0.23
ReBAT(d = 1.5, À = 1.0) 81.86 81.91
-0.05
56.36
56.12 0.24
51.13
51.22
-0.09
ReBAT(d 1.5, À 4.0) 79.68 79.88
-0.20
55.72
55.65
0.07
50.52
50.50
0.02
ReBAT(d 4.0, 0.0) 83.05 85.38 -2.33
55.98
50.87
5.11
50.38
46.95
3.43
ReBAT(d = 4.0, À = 1.0) 82.46 84.84 -2.38
55.86
52.02 3.84
50.87
47.88
2.99
ReBAT(d 4.0, À 4.0) 80.99 84.07 -3.08
56.06
53.58 2.48
51.00
49.02
1.98
D.3 Additional Results Adopting Stronger Training Time Attacker
In Section 4.3, we design two versions of ReBAT, and we name the stronger one that also uses a stronger training attacker as ReBAT++. Here we fix the hyperparameter used in ReBAT above (note that we use a larger LR decay factor d = 1.7 in ReBAT++ for CIFAR- 10, and now we still use d 1.5 as in ReBAT) and adjust the attacker strength to study its influence when combined with ReBAT. According to Figure 15, though a slightly stronger attack (e.g., & = 9/255) may marginally improves the best and last robust accuracy, it heavily degrades natural accuracy, particularly when much stronger attack is used. We deem that this is because it breaks the balance from the other side that the overly strong attacker A dominates the adversarial game and results in an underfitting state that harms both robust and natural accuracy.
D.4 Further Improving Natural Accuracy with Knowledge Distillation
Chen al. [8] propose to adopt knowledge distillation (KD) [18] to mitigate RO and it is worth mentioning that their method achieves relatively good natural accuracy according to Table and 2. Since our ReBAT method is orthogonal to KD, we propose to combine our techniques with KD to further improve natural accuracy. Specifically, we simplify their method by using only a non-robust standard classifier because i) a large sum computational cost for training the AT teacher will be saved, ii) our main goal is to improve natural accuracy the ST teacher matters more and iii) ReBAT already use the WA model as a very good teacher. This gives the final loss function as
teacher (ST teacher) instead of using both ST teacher and a AT teacher,
(ReBAT+KK(I, y; 0) =(1 - AST) BOATT(Z (I, y; 0) + ÄST KL (fe (f(())).fsr())
(5)
where fsT indicates the ST teacher and ASt is trade-off parameter.
22

---

80
75
Accuracy (%)
70
best natural
last natural
65
best robust
last robust
60
55
50
10
13 14
(x 255)
Figure 15: Using different adversarial attack strength in ReBAT.
Table 8: Combining our methods with knowledge distillation and evaluating model robustness on CIFAR-10 under the perturbation norm Eoo = 8/ 255 based on the PreActResNet-Ii 18 architecture.
Natural
PGD-20
AutoAttack
Method
best
final
diff
best
final
diff
best
final diff
ReBAT (AST 0.0) 81.86 81.91 -0.05
56.36
56.12 0.24
51.13 51.22 -0.09
ReBAT+KD (AST = 0.4) 83.59 83.64
-0.05
54.91
54.77 -0.14
50.70
50.99 -0.29
ReBAT+KD (AST 0.5) 84.12 84.20
-0.08
55.28
55.39
-0.11
50.47
50.72 -0.25
ReBAT+KD (AST = 0.6) 84.34 84.72 -0.38
53.83
54.30
-0.47
50.15 50.37 -0.22
Table 8 compares the performance of ReBAT+KD when different ÀST is applied, and clearly large ÀST results in improvement in natural accuracy and decreases robustness (may due to the theoretically principled trade-off' between natural accuracy and robustness [59]). However, it is still noteworthy that when Ägr = 0.4, a notable increase in natural accuracy (~ 1.7%) is achieved at the cost of only small slide of ~ 0.2% in final robustness against AA. Also when ÄST 0.5, it achieves comparable natural accuracy with KD+SWA [8] but has much higher AA robustness. Moreover, we emphasize that RO is almost completely eliminated regardless of the trade-off, which is the main concern of this paper and demonstrates the superiority of our method against previous ones. An intriguing phenomenon is that nearly all the final results better than the results on the best checkpoints selected by the validation set, which implies that in this training scheme AT enjoys the same property of "training longer, generalize better" as ST without any need of early stopping.
D.5 The Effeet of Different Learning Rate Schedules
In the previous experiments we only investigate the piecewise LR decay schedule. However, a natural idea would be using mild LR decay schedules, e.g., Cosine and Linear decay schedule, instead of suddenly decaying it by a factor of d at some epoch in the piecewise decay schedule. As mentioned in Section 6. previous works have shown that changing LR decay schedule fails to effectively suppress RO whether with [45] or without WA [38] because the LR finally becomes small and endows the trainer with overly strong fitting ability. Therefore, here we continue to experiment with modified Cosine and Linear decay schedules that follow similar LR scale of the piecewise LR decay schedule, and summarise the results in Table 9. To be specific, the LR still decays to 0.01 at epoch 150 and 0.001 epoch 200, though the two decay stages (from epoch 100 to 150 and 150 to 200) designed to be gradual following the Cosine/Liness schedule. We also gradually increase the strength of ReBAT regularization from zero as the LR gradually decreases, following the "larger decay factor goes with stronger ReBAT regularization" principle that we introduced in Appendix D.2.
It can be concluded from Table 9 that simply changing the LR decay schedule indeed improves the best robust accuracy from 49.89% to 50.59% and 50.67% % against AA respectively, but it provides no help at all in mitigating RO as the final robust accuracy is still below 45% against AA. We also note that in this situation, the application of BoAT loss not only significantly mitigates RO but also further improves the best model robustness, which also proves its effectiveness.
23

---

Table Comparing our method with WA on CIFAR-10 under the perturbation norm Ecc 8/255 based on PreActResNet-Be 18 architecture, when Cosine/Linear LR decay schedule are applied.
Cosine
Linear
Method
Natural
PGD-20
AutoAttack final
Natural
PGD-20
AutoAttack
best
final
best
final
best
best
final
best
final
best
final
WA(d=10)
82.32 84.99 55.97 47.84
50.59 44.08
82.21 85.01 56.21 48.10
50.67 44.63
ReBAT(d=10))
82.06 82.22
56.12 54.44
50.98 49.81
81.98 82.13
56.33 54.50
51.08 49.63
D.6 Results on Different Network Architectures
In previous experiments we compare methods based PreActResNet-18 and WideResNet-34- 10 architecture, and here we also adopt VGG-16 [41] and MobileNetV2 [39] architecture. The significant improvement against baseline PGD-AT in both best and final robust accuracy and in mitigating RO reported in Table 10 further demonstrates that our method works on a wide range of network architectures.
Table 10: Comparing method with PGD-AT on CIFAR-10 under the perturbation norm Ex. 8/ 8/255 based on VGG- 16 and MobileNetV2 architecture.
VGG-16
MobileNetV2 PGD-20 best
Method
Natural
PGD-20
AutoAttack final
Natural best
AutoAttack best
best
final
best
final
best
final
final
final
PGD-AT 78.43 81.64
50.56 44.25
44.19 39.66 79.85 80.67 51.56 50.67
46.01 45.27 47.66 47.35
ReBAT 78.17 78.37
53.13 53.01
47.24 47.13
78.98 80.81
53.18 52.57
24


Reviews:
[{"review_id": "w3BnsC5TWS", "reviewer": "Reviewer_Znmp", "summary": "This paper investigates the phenomenon of robust overfitting in adversarial training and explains it from a minimax game perspective. The authors analyze how the decay of the learning rate disrupts the balance between the model trainer and the attacker, leading to robust overfitting. They propose a method called ReBalanced Adversarial Training (ReBAT) to mitigate robust overfitting and achieve good robustness even after long training.", "questions": "Please see the above weaknesses.", "limitations": "Missing...\n\nPlease stately explicitly the limitations and potential negative impact. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1 The paper is well organized and easy to follow.\n\n2  The paper provides a holistic understanding of robust overfitting in adversarial training by analyzing the imbalance between the model trainer and the attacker from a minimax game perspective. This perspective helps explain why robust overfitting occurs in adversarial training and why it does not occur in other training methods.\n\n3 The experiments cover different network architectures and benchmark datasets, providing strong empirical evidence.", "weaknesses": "1 it is better to clearly state the diffference between the defined robust/non-robust features and the previous one in [15]\n\n2 ReBAT[strong] seems like a strange notation, please consider to change one. \n\n3 The caption of Table 1 seems not correct?\n\n4 Does the minimax game view provide any insights on the accuracy and robustness tradeoff?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer Znmp for your careful reading and for appreciating the proposed understanding and method for robust overfitting. Below, we address your main concerns of this work.\n\n---\n\n**Q1.** Diffference between the defined robust/non-robust features and the previous one in Ilyas et al.\n\n**A1.** In this paper, we **follow Ilyas et al’s definitions of robust and non-robust features**, and extract the features following the same procedure. The main difference is that, instead of focusing only on the last trained model (as in Ilyas et al), we study **the dynamic behaviors of AT models on utilizing these robust/non-robust features during the training process.** Specifically, we extract non-robust features from models obtained before and after LR decay, and use these features to verify our understanding of robust overfitting from a dynamic minimax game perspective (Sec 3.2).\n\n---\n\n**Q2.** ReBAT[strong] seems like a strange notation, please consider to change one.\n\n**A2.** Here, ReBAT[strong] refers to the combination of ReBAT with stronger attack. **ReBAT++** could be a good alternative name since this version further enhances adversarial robustness over ReBAT. Please let us know if you have better options.\n\n---\n\n**Q3.** The caption of Table 1 seems not correct?\n\n**A3.** Thanks for pointing it out. There is a typo here. Table 1 shows CIFAR-10 results with different backbones. We will change the caption to “Comparing our method with several training methods on CIFAR-10 under the perturbation norm $ε_\\infty = 8/255$ based on the PreActResNet-18 and WideResNet-34-10 architectures.” \n\n---\n\n**Q4.** Insights on the accuracy and robustness trade-off from our view of minimax game. \n\n**A4.** The minimax view can also provide new insights into understanding the accuracy-robustness trade-off. As we discussed in the paper, during AT, $\\mathcal{A}$ attacks by injecting non-robust features into adversarial examples. \n- When the minimax game is **balanced**, these non-robust features will not be fitted (i.e., discarded). Since these non-robust features also contribute to classification accuracy (see Ilyas et al.), discarding more non-robust features (i.e., more robust) will lead to worse accuracy. \n- When the minimax is **imbalanced** and the model trainer is stronger than the attacker (as studied in our paper), the model will capture more and more non-robust features and get better accuracy, while at the cost of introducing shortcuts to the attacker and leading to worse and worse robustness (revealed in Section 3.2). \n\nTherefore, in both balanced and imbalanced minimax game, there will be a trade-off between accuracy and robustness, and the discrepancy is larger when robust overfitting happens. \n\n---\n\n**Q5.** Please state explicitly the limitations and potential negative impact.\n\n**A5.** Thanks for your suggestion. A main limitation of the proposed method is that as we regularize the model trainer to restore the balance of the minimax game, the training takes slightly longer to attain optimal performance, e.g., 110 epochs (AT) and 170 epochs (ours). However, in practice, people usually train longer in AT (typically 200 epochs) to attain better accuracy, and use early stopping to select the best checkpoint. As our method does not need early stopping techniques (without robust overfitting), and have neglectable computation overhead per epoch, the total training time is comparable with vanilla AT. \n\nAs for the social impact, since our method is designed for better defence of adversarial attack, it should have a positive impact by enhancing the security of machine learning models.\n\n---\n\nHope our elaborations above could address your concerns. Please let us know if there is more to clarify."}, {"review_id": "UtWc8Sohpi", "reviewer": "Reviewer_55GS", "summary": "This paper studies the robust overfitting phenomenon in adversarial training. Meanwhile, this paper focuses on a specific problem “the robust overfitting occurs when we use learning rate decay techniques.” This paper proposes a game perspective to explain the robust overfitting. It claims that the robust overfitting happens because of the imbalance between the attacker and the model after the LR decay. Moreover, this paper proposes ReBAT method to improve the robustness and mitigates the overfitting phenomenon.", "questions": "See weakness.", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "This paper studies a specific but interesting problem, the robust overfitting phenomenon. This paper explains that the overfitting happens because of the learned mapping of the non-robust features after LR decay.\n\nThis paper provides an intersting perspective to help us to understand the cause of the robust ovetfitting. In particular, the robust problem is a min-max problem, which could be regarded as a game. The robust overfitting is because of the breaks of the original equilibrium.\n\nThis paper provides extensive verification to support its explanation, which addresses most of my questions when I first review this paper.", "weaknesses": "1.For the Figure 2 (b), I guess the red line denotes  w/ LR decay and the blue line denotes w/o LR decay;\n\n2.For section 3.2.1, the authors share an interesting and important insight. I’m afraid it is a little trifling. Could the author provide a figure to demonstrate it?\n\n3.In section 4.1, the authors provide some techniques to address the robust overfitting, e.g., bootstrapping, small decay factor. From the theoretical analysis in this paper,  I guess there may exist other methods to achieve the balanced learning of robust and non-robust feature, and more discussions is helpful.  Could the author provide some theoretical analysis about the proposed method for the re-balance?\n\n4.I find the proposed method ReBAT may cause more computation overhead to get a convergent result. Could the author discusses the limitations of this work?\n\n5.Could the author share more discussions about the game idea?  For example, in multi-task learning, could it be regarded as a game, where there are multiple players?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 55GS for your careful reading and appreciation on the novelty and solidness of our work. Below, we address your main concerns of this work.\n\n---\n**Q1.** For Figure 2 (b), I guess the red line denotes w/ LR decay and the blue line denotes w/o LR decay.\n\n**A1.** Thank you for pointing it out. You are correct. We will correct it in the revision.\n\n---\n**Q2.** For section 3.2.1, the authors share an interesting and important insight. Could the author provide a figure to demonstrate it?\n\n**A2.** Following your suggestion, we plot an intuitive figure Fig A (Rebuttal PDF) to illustrate each individual stage during the robust overfitting. Specifically, Fig A (a) shows that before LR decay it achieves a balance between the trainer and the attacker, and the weak trainer will NOT fit the non-robust features in misclassified adversarial training examples (blue→orange). Fig (b) shows that after LR decay, the trainer endowed with stronger fitting ability memorizes the adversarial examples to their original labels (blue) by drawing a more complex decision boundary (from a feature view, the non-robust features are falsely memorized altogether). Fig (c) shows that this new decision boundary introduces easy-to-reach shortcuts to attack test examples as it falsely maps y’-class features (the orange region) to the y class. As a result, the training robustness rises but test robustness degrades. This is in line with our analysis in Sec 3.2.1 that after decay, the falsely memorized non-robust features open shortcuts for test-time attack and lead to robust overfitting.\n\n---\n**Q3.** From the theoretical analysis in this paper, I guess there may exist other methods to achieve the balanced learning of robust and non-robust feature, and more discussions are helpful. Could the author provide some theoretical analysis about the proposed method for the re-balance?\n\n**A3.** Indeed, there are other kinds of strategies to restore the balance of minimax training, including some tricks proposed in prior works. We have provided an overview of these strategies from our theoretical perspective in **Section 5**, where we explain how existing techniques (10+) can help restore the balance from three different aspects: data regularization, training regularization, and stronger attacker. \n\nHere, we provide a more detailed explanation of the proposed methods (bootstrap, small decay, and stronger attack). As shown in the theory of Section 3.2, robust overfitting rises when the model trainer memorizes false non-robust features contained in adversarial examples. Therefore, there are two ways to alleviate this effect, either to regularize the trainer (cannot fit), or to strengthen the non-robust features (cannot be fitted). The bootstrap regularization enforces a flat landscape of model weights (simpler decision boundary), such that the non-robust features cannot be memorized via drawing complex decision boundaries. Similarly, the small LR decay factor regularizes the optimizer’s local fitting ability, which also prevents memorizing the non-robust features. Stronger attacker, on the other side, makes the non-robust features too strong to be memorized.\n\nHope the explanations above helps address your questions and we will add them in the revision.\n\n---\n**Q4.** I find the proposed method ReBAT may cause more computation overhead to get a convergent result.\n\n**A4.** To ease your concerns, we calculate the per-epoch training cost of different methods. From the table below, we can see that the per-epoch training cost of ReBAT is almost the same as vanilla AT (129.4s v.s. 125.3s), lower than many other advanced AT methods (e.g., AWP). Thus, in this view, ReBAT is rather computationally efficient. \n\nAs you mentioned and can be observed from the training process in Figure B (Rebuttal PDF), ReBAT takes slightly longer training time to converge to the best performance (it peaks at around the 170th epoch while AT peaks at around the 110th epoch), because it adopts smaller LR decay and regularization to avoid robust overfitting. **But it achieves much higher robustness and does not suffer from robustness deterioration.** Further considering that in practice, people usually train longer in AT (typically 200 epochs) to attain better accuracy and use early stopping to select the best checkpoint, the two methods actually require similar total training epochs in practice. \n\nTherefore, we believe that the computation cost of ReBAT is comparable to vanilla AT while attaining much better robustness.\n\n*Training time comparison on CIFAR-10 with PreActResNet-18 backbone.*\n\n| Method | Training Cost (per epoch) |\n| --- | --- |\n| PGD-AT | 125.3 |\n| AWP | 135.6 |\n| ReBAT | 129.4 |\n\n---\n**Q5.** Could the author share more discussions about the game idea?\n\n**A5.** There is a body of literature that analyzes AT using game theory, e.g., [1], usually regarding AT as a zero-sum game between two players. A recent work argues that AT should be treated as a non-zero-sum game [2]. For multi-objective AT, there could exist multiple attackers that constitute a game of multiple players, where the AT could behave very differently. \n\nDue to word limit, please refer to **A1 in our response to Reviewer EtSp** for deeper discussions on the connections between our work and those papers and new insights our work brings to the game idea. Existing game theory papers mainly study AT’s Nash equilibrium under toy models and **static minimax players**, but not **a dynamic game in practical AT algorithms** as in our understandings in robust overfitting. We will elaborate this part in the revision.\n\n**References:**\n\n[1] Balcan et al. Nash Equilibria and Pitfalls of Adversarial Training in Adversarial Robustness Games. AISTATS, 2023.\n\n[2] Robey et al. Adversarial Training Should Be Cast as a Non-Zero-Sum Game. *arXiv preprint arXiv:2306.11035.* 2023.\n\n---\nHope our elaborations and new results above could address your concerns. Please let us know if there is more to clarify."}, {"review_id": "IGAOFLJvFk", "reviewer": "Reviewer_EtSp", "summary": "This paper empirically shows that robust overfitting is caused by the over-memorization of the non-robust features after learning rate decay. To mitigate the issue of robust overfitting, the authors propose to use a stronger training attack, a smaller learning rate decay rate, and a bootstrapped adversarial training loss. The comprehensive empirical results validate the effectiveness of the proposed method in mitigating robust overfitting and even improving robustness.", "questions": "1. What is the effect of the hyper-parameter \\lambda?\n\n2. Could you provide the result evaluated on CIFAR-10 in the experiments part? I am afraid that I did not find the corresponding results.\n", "limitations": "The proposed method can effectively mitigate the issue of robust overfitting.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper provides comprehensive verifications for their proposed reason for robust overfitting. The authors clearly show the effect of the false memorization of the adversarial non-robust features after learning decay in robust overfitting.\n\n2. The empirical experiments on various datasets and networks are comprehensive. The results support the authors’ claim.\n", "weaknesses": "1. Besides extensive empirical results, it would be better for the authors to provide some analyses from a theoretical perspective (possibly using game theory which could be related to the empirical results in this paper). \n\n2. The proposed method could hurt the natural test accuracy to some extent.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer EtSp for appreciating the comprehensiveness and solidness of the verification of our understanding. Below, we address your main concerns.\n\n---\n\n**Q1.** It would be better for the authors to provide some analyses from a theoretical perspective (possibly using game theory which could be related to the empirical results in this paper).\n\n**A1.** There is a body of literature that analyzes AT from a game theory perspective, see [1-5]. However, existing theory papers only analyze AT’s Nash equilibrium under toy models (e.g., Gaussian features and linear models), and **static** minimax players. **None of the existing literature considers a dynamic game and could explain robust overfitting from a game perspective, particularly in practical AT algorithms, as done in our work.** Nevertheless, we find some interesting connections and new insights between them that worth noting.\n\n**Connections.** Under these simplified assumptions, prior works show that although the Nash equilibrium of the AT objective exists and is robust, **the current alternating optimization of AT may fail to converge to the Nash equilibrium**, see, e.g., a recent work [5]. The key reason is that the trainer can falsely fit the non-robust features [5], which is in a similar spirit to our analysis that robust overfitting is caused by the falsely memorized non-robust features after LR decay. **This shows that our explanation is not only in line with the cutting-edge theory of AT, but also further explains robust overfitting from a game perspective for the first time.**\n\n**New insights.** We also notice that existing AT game theory papers cannot explain robust overfitting, and from the perspective of our theory, this is possible because they overlook the dynamic change of the AT players. Thus, our understanding of AT from a dynamic game can inspire more in-depth theoretical characterization of AT from the game theory perspective.\n\nAs our work mainly focuses on the understanding robust overfitting of practical AT algorithms, and it is generally hard to theoretically analyze the training dynamics of practical DNNs, we leave more theoretical investigations under strong assumptions to future work. Following your suggestion, we will add this discussion on the relation to game theory in the revision.\n\n**References:**\n\n[1] Bulo et al. Randomized prediction games for adversarial machine learning. TNNLS, 2016.\n\n[2] Pal & Vidal. A game theoretic analysis of additive adversarial attacks and defenses. NeurIPS, 2020.\n\n[3] Bose et al. Adversarial example games. NeurIPS, 2020.\n\n[4] Pinot et al. Randomization matters how to defend against strong adversarial attacks. ICML, 2020.\n\n[5] Balcan et al. Nash Equilibria and Pitfalls of Adversarial Training in Adversarial Robustness Games. AISTATS, 2023.\n\n---\n\n**Q2.** The proposed method could hurt natural accuracy to some extent.\n\n**A2.** Due to the well-known trade-off between accuracy and robustness, better robustness often comes with lower accuracy. Nevertheless, Table 1 (quoted below) shows that our ReBAT version achieves **both better accuracy and robustness** **than PGD-AT, TRADES, and AWP** on CIFAR-10. The ReBAT[strong] version utilizes stronger attacker, so it scarifies accuracy to trade for higher robustness. **Table 8 (Appendix C.4)** shows that further incorporating the knowledge distillation (KD) technique can further boost the clean accuracy of ReBAT while maintaining high robustness. Therefore, ReBAT achieves a good trade-off between accuracy and robustness as it can attain good clean accuracy while significantly improve model robustness. \n\n*Performance of best-epoch models on CIFAR-10 with PreActResNet-18. (Quoted from Table 1)*\n\n| Method | Natural Accuracy | AutoAttack |\n| --- | --- | --- |\n| PGD-AT | 81.62 | 47.51 |\n| TRADES | 80.45 | 48.09 |\n| AWP | 81.11 | 50.09 |\n| ReBAT (ours) | 81.86 | 51.13 |\n| ReBAT[strong] (ours) | 78.71 | 51.49 |\n| ReBAT + KD  (ours) (Table 8) | 84.20 | 50.72 |\n\n---\n\n**Q3.** Effect of the hyper-parameter $\\lambda$.\n\n**A3.** Here, $\\lambda$ controls the strength of the KL regularization, and a larger $\\lambda$ imposes stronger regularization on model flatness. In **Table 7 in Appendix C.2**, we include a detailed analysis of the influence of $\\lambda$. We quote some results below. We can see that under a small decay rate ($d=1.5$), a medium $\\lambda$ attains the best performance, while too large $\\lambda$ hurts both natural accuracy and robustness severely with too much flat regularization. When the decay rate is larger, e.g., $d=4$, a larger $\\lambda$ can significantly improve model robustness by regularizing the model’s fitting ability, but meanwhile, its natural accuracy also degrades. In view of this analysis, in practice, we simply choose a small decay factor $d=1.5$ and a medium $\\lambda=1.0$ as it can attain both good accuracy and robustness.\n\n*Analysis of the decay factor ($d$) and regularization strength ($\\lambda$) on CIFAR-10 with PreActResNet-18 (best-epoch results). (Quoted from Table 7)*\n\n| $d$ | $\\lambda$ | Natural Accuracy | AutoAttack |\n| --- | --- | --- | --- |\n| 1.5 | 0.0 | 81.90 | 50.81 |\n|  | 1.0 | 81.86 | 51.13 |\n|  | 4.0 | 79.68 | 50.52 |\n| 4.0 | 0.0 | 83.05 | 50.38 |\n|  | 1.0 | 82.46 | 50.87 |\n|  | 4.0 | 80.99 | 51.00 |\n\n---\n\n**Q4.** Missing CIFAR-10 results in the experiments part.\n\n**A4.** This is caused by a typo. Table 1 actually is the CIFAR-10 results with two different backbones. The correct caption should be “Comparing our method with several training methods on CIFAR-10 under the perturbation norm $ε_\\infty = 8/255$ based on the PreActResNet-18 and WideResNet-34-10 architectures.” We will correct it in the revision. Thanks for pointing it out.\n\n---\n\nHope our elaborations and new results above could address your concerns. Please let us know if there is more to clarify."}, {"review_id": "NjZGPVfRsN", "reviewer": "Reviewer_kwEx", "summary": "This paper explains the phenomenon of robust overfitting in adversarial training from a minimax game perspective. The author considers AT as a minimax game between the model trainer and the attacker, pointing out the imbalance between them leads to the network memorizing non-robust features, causing robust overfitting. Based on these explanations, the author proposes several measures to rebalance the minimax game, thereby mitigating robust overfitting and improving adversarial robustness.", "questions": "Please refer to comments in weakness.", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. It is interesting to study AT from a minimax game perspective.  \n2. The paper proposes multiple measures to alleviate robust overfitting and enhance adversarial robustness.", "weaknesses": "1. The motivation is unclear. The author's explanation of the robust overfitting process is based on some observation-driven analysis, which are difficult to be convincing. For example, the attacker injects non-robust features for misclassification, and the cause of robust overfitting is the network's memorization of non-robust features. What exactly are the false non-robust mapping and the falsely memorized non-robust features? Can the authors use the intuitive and precise statement to explain the mechanism of robust overfitting?\n\n2. The method's novelty is limited. The author claims that previous attempts to change attacker strength have not focused much on robust overfitting. However, there is existing research in this area:   \nYu C, Zhou D, Shen L, et al. Strength-Adaptive Adversarial Training, arXiv preprint arXiv:2210.01288, 2022.\n\n3. The experimental results are incomplete and not significant. 1) Did the author confirm that the robustness of MLCAT in Table 1 is lower than AWP? 2) The author introduces multiple measures to mitigate robust overfitting and reports their combined performance. However, what are the individual performances of each technique? Considering that even combining multiple existing techniques for robust overfitting mitigation can further improve robustness, it is necessary to report the experimental results of each individual technique applied to Standard AT.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer kwEx for your time and efforts in reviewing this work. Below, we address your main concerns of this work.\n\n---\n\n**Q1.** What exactly are the false non-robust mapping and the falsely memorized non-robust features? Can the authors use the intuitive and precise statement to explain the mechanism of robust overfitting?\n\n**A1.**  To address your concerns, we add both intuitive illustrations and formal definitions below. We will append these discussions in the revision to make it clearer.\n\n**1) Intuitive Understandings** \n\nFollowing your suggestion, we plot an intuitive figure Fig A (in Rebuttal PDF) to illustrate each individual stage during the robust overfitting. Due to the space limit, please refer to A2 of Reviewer 55GS for detailed descriptions.\n\n**2) Precise Definitions** \n\nIn this paper, we mainly follow Ilyas et al’s definitions of robust and non-robust features, and extract the features following the same procedure. Thus, “non-robust mapping” and “non-robust features” refer to the same thing, because Ilyas et al define **robust/non-robust features are defined as mappings** $f:X\\to \\mathbb{R}$  (Sec 2). The key difference is that they only extract features only from the final model, while we study features learned at different training stages. \n\nTo state it precisely, a training configuration $t\\in\\mathcal{T}$ (LR, epoch, etc) specifies a hypothesis class of features/mappings that it can robustly fit on training data, i.e.,\n\n$H_t=${$f_\\theta:X\\to R \\|$ $E_{x, y\\sim D_{train}}\\inf_{\\|x-\\bar x\\|\\leq\\varepsilon}[y \\cdot f_\\theta(x)]>0$ $\\ (f_{\\theta}$ is robust on training data), and $\\theta\\in{\\Theta}$ is attainable under the configuration $t$}.\n\nWe say a non-robust feature $f$ is falsely memorized if $f\\notin H_{before}, f\\in H_{after}$, where $H_{before},H_{after}$ refer to the hypothesis classes before and after LR decay, respectively. In other words, the non-robust $f$ is falsely memorized by the model (as a robust feature) under the after-LR-decay trainer. This mismatch could happen because the trainer only sees training examples and it can overfit them under an imbalanced minimax game. Since this feature is essentially a non-robust feature (in the population sense), this feature still behaves non-robust on test data, and even introduce shortcuts to test-time attack (verified in Sec 3.2.2). \n\n**3) Empirical Evidence**\n\nNotably, our explanation of robust overfitting is further justified by four extensive experiments in Section 3.2.2, where **we extract non-robust features from models before and after LR decay following Ilyas et al.’s procedure**, and compare their influence on robustness (Verification 1), target-class information (Verification 2), class correlation (Verification 3), and class symmetry (Verification 4). The phenomena fully support our dynamic game perspective of AT.\n\n---\n\n**Q2.** The method's novelty is limited. The author claims that previous attempts to change attacker strength have not focused much on robust overfitting. However, there is existing research in this area (Yu et al., SAAT). \n\n**A2.** Thank you for pointing out this recent work, and we will add it for discussion in revision. Like ours, SAAT also considered stronger attack as a way to counter robust overfitting. However, we remark that there are several key differences:\n\n- **Different perspectives.** The **main focus/contribution in our work is the minimax game understanding of robust overfitting (Sec 3)**. The three strategies are naturally motivated by our understanding to rebalance the minimax game. Instead, the design of SAAT is motivated directly by the influence of the perturbation budget on robustness disparity. Therefore, the two works clearly have different motivations and design principles.\n- **Different solutions.** Notably, stronger attack is **only one of the three strategies proposed in our work** (Sec 4), and it is **optional** and **NOT adopted in the final ReBAT method** (bootstrap+small decay) because it degrades clean accuracy a lot. Besides, ReBAT attains **very good accuracy and robustness *without stronger attack*, and it can still outperform AWP+SAAT** (e.g., clean 79.49 v.s. 81.86, AA 49.29 v.s. 51.22) with neglectable overfitting.\n- **Different strategies for adjusting attacker.** The two are also different in the proposed stronger attack. SAAT gradually increases $\\varepsilon$ along training and adjusts adaptively for different samples. Instead, ours is much simpler and easy to use: **we only apply larger perturbations after LR decay to counter stronger trainer, and use a constant perturbation budget for all samples** (Sec 4.2)**.** As shown in Table B in the attached PDF, this simple strategy achieves comparable or even better performance than SAAT (78.17 v.s. 76.37 natural acc, 50.99 v.s. 48.86 best AA, 47.66 v.s. 47.17 last AA), and still brings slight better robustness when combined with other methods like AWP and ReBAT.\n\nThus, according to these key differences, our method is still quite different from SAAT and provides new perspectives and solutions for understanding and alleviating robust overfitting.\n\n---\n\n**Q3.** Did the author confirm that the robustness of MLCAT (Yu et al.) in Table 1 is lower than AWP?\n\n**A3.** Indeed, there is a transcription mistake in the WideResNet-34 results of MLCAT in Table 1. Thanks for pointing it out. We add the correct results in Table A (Rebuttal PDF) and will fix them in the revision. \n\n---\n\n**Q4.** What are the individual performances of each technique?\n\n**A4.** We note that we have provided a detailed analysis of each technique in **Figure 5 (Sec 4)** and included the quantitative results in **Appendix C**. We further summarize the results in Table C in Rebuttal PDF. We can see that each technique is useful for mitigating RO and combining them leads to better performance.\n\n---\n\nHope our elaborations and new results above could address your concerns. Please let us know if there is more to clarify."}]

Author rebuttal:
The Rebuttal PDF can be seen in the attached file, which contains 
- Figure A: an intuitive illustration for the proposed understanding of robust overfitting;
- Figure B: a plot of the training process comparing ReBAT with vanilla AT;
- Table A, B, C: additional comparison experiments between different AT methods.

Meta-review: The reviewers have generally agreed that this paper provides a valuable contribution to the understanding of robust overfitting in adversarial training. The authors' approach of analyzing the issue from a minimax game perspective is appreciated for its novelty and the insights it provides. The proposed ReBalanced Adversarial Training (ReBAT) method is also seen as a promising solution to mitigate robust overfitting. 

However, the reviewers have also pointed out some areas that could be improved. These include the need for clearer explanations and notations, more theoretical analysis, and a discussion on the limitations of the proposed method. The authors are encouraged to address these points in their final version of the paper.

Overall, the paper is accepted for its technical solidity, potential impact, and comprehensive empirical results. The authors are commended for their work and encouraged to continue their research in this area.
Decision: Accept (poster)