Understanding and Mitigating Robust Overfitting through the Lens of Feature Dynamics
Authors: Yifei Wang
Liangchen Li
Yisen Wang
Jiansheng Yang
Zhouchen Lin
Venue: Submitted to ICLR 2023 (2023)
Keywords: Adversarial Training
Robust Overfitting
Generalization
Robustness
Adversarial Attack

Abstract: Adversarial Training (AT) has become arguably the state-of-the-art algorithm for extracting robust features. However, researchers recently notice that AT suffers from severe robust overfitting problems, particularly after the learning rate (LR) decay, while the existing static view of feature robustness fails to explain this phenomenon. In this paper, we propose a new dynamic feature robustness framework which takes the dynamic interplay between the model trainer and the attacker into consideration. By tracing temporal and dataset-specific feature robustness, we develop a new understanding of robust overfitting from the dynamics of non-robust features, and empirically verify it on real-world datasets. Built upon this understanding, we explore three techniques to restore the balance between the model trainer and the attacker, and show that they could effectively alleviate robust overfitting and attain state-of-the-art robustness on benchmark datasets. Notably, different from previous studies, our interpretation highlights the necessity of considering the min-max nature of AT for robust overfitting.


Under review as a conference paper at ICLR 2023
UNDERSTANDING AND MITIGATING ROBUST OVERFIT- TING THROUGH THE LENS OF FEATURE DYNAMICS
Anonymous authors Paper under double-bling blind review
ABSTRACT
Adversarial Training (AT) has become arguably the state-of-the-art algorithm for ex- tracting robust features. However, researchers recently notice that AT suffers from severe robust overfitting problems, particularly after the learning rate (LR) decay, while the existing static view of feature robustness fails to explain this phenomenon. In this paper, we propose a new dynamic feature robustness framework which takes the dynamic interplay between the model trainer and the attacker into consideration. By tracing temporal and dataset-specific feature robustness, we develop new understanding of robust overfitting from the dynamics of non-robust features, and empirically verify it on real-world datasets. Built upon this understanding, we explore three techniques to restore the balance between the model trainer and the attacker, and show that they could effectively alleviate robust overfitting and attain state-of-the-art robustness on benchmark datasets. Notably, different from previous studies, our interpretation highlights the necessity of considering the min-max nature of AT for robust overfitting
INTRODUCTION
Overfitting seems to have become a history in the deep learning era. Contrary the traditional belief' in statistical learning theory that large hypothesis class will lead to overfitting, [Zhang et al. (2019) note that DNNs have good generalization ability test data even if they are capable of memorizing random training labels. Nowadays, large-scale training often does not require early stopping, and longer training simply brings better generalization (Hoffer et al. 2017).
However, researchers recently notice that in Adversarial Training (AT), overfitting is still severe issue on both small and large scale data and models (Rice al. 2020). AT is arguably the most effective defense method (Athalye et al.] 2018) against adversarially ly crafted perturbations to images (Szegedy et al. (0144). Specifically, given tramiin data Derain and model fo. AT can be formulated as a min-max optimization problem (Madry et al. 2018 Goodfellow et al. 2015):
min Er. Derain max (cF((o(()).)) =xE
where fce denotes the cross entropy (CE) loss, (z) = fz ||x- E1pp - S }} denotes the -p-norm ball with radius e. However, in practice, researchers notice this min-max training scheme suffers severely from the robust overfitting (RO) problem: after particular point (e.g.. learning rate decay), its training robustness will keep increasing (Figure Ta] red line) while its test robustness will begin to dramatically decrease (Figure lb] red line). This abnormal behavior of AT has attracted many interests recently. Previous work correctly points out that during AT, the robust loss landscape becomes much sharper, and RO can be largely surpassed by enforcing a smoother landscape with additional regularizations (Stutz et al., 2021 Chen et al.] 2021 Wu et al.) 2020). However, this phenomenological perspective does not explain the rise of either the sharp landscape or RO. Some researchers try to explain RO though the phenomenons of standard training (ST), such as random memorization (Dong et al. 2022) and double descent Dong et al. (2021). but they fail to characterize the uniqueness of AT from ST(why AT has the overfitiing issue while ST does not).
In this paper, we seek to establish a generic explanation for robust overfitting during AT. Different from previous attempts to relate the behaviors of AT to existing theories of ST, we believe that the reasons of RO should lic exactly in the differences between AT and ST. In particular, we emphasize a critical difference: RO usually happens after learning rate (LR) decay in AT, while in ST, LR decay does not lead to (severe) overfitting, so our paper focus the change of learning behaviors before and after the L.R decay. We notice that feature robustness actually changes during this process: a non-robust feature could become robust after LR decays, which is contrary to the static feature robustness framework developed by Ilyys al. 2019). This motivates to design dynamic feature

---

Under review as a conference paper at ICLR 2023
15
9214 m000
(a) training robustness dur- (b) test robustness during (c) robustness change of (d) relative change in test robustness with stronger non-robust features (e).
ing adversarial training adversarial training
non- non-robust features
Figure (a, b) Adversarial training results on CIFAR-10 using vanilla PGD-AT (Madry al. 2018). (c) After decay, non-robust features become significantly more robust on training set, but only increase slightly on test set. (d) With LR decay, injecting stronger non-robust features (larger e) induces severer degradation in test robustness, while the degradation is less obvious without decay.
robustness framework that takes the learning process into consideration, and discussion the dynamics of feature robustness. Moreover, this dynamic perspective also suggests a new understanding of robust overfitting. Specifically, we hypothesize that due to the strong local fitting ability endowed by smaller LR, the model learns false mapping of non-robust features contained in adversarial examples, which opens shortcuts for the test-time attacker and induces large test robust error. Accordingly, design series of experiments that empirically verify this understanding.
Based on this dynamic view of feature robustness, we investigate three new strategies to avoid fitting non-robust features: stronger model regularization, smaller LR decay, and stronger attacker. We show that all three strategies help mitigate robust overfitting to a large extent. Based on these insights, we propose Bootstrapped Adversarial Training (BoAT) that has neglectable (if any) degree of robust overfitting without using additional data augmentations, even if we train for 500 epochs. Meanwhile, BoAT attains state-of-the-an robustness (both best-epoch and final-epoch) on benchmark datasets including CIFAR- 10, CIFAR-100, and Iing-emm This suggests that with appropriate design of training strategies, AT could also enjoy similar "train longer, generalize better" property like ST. To summarize, our main contributions are:
We point out that the existing static robust feature framework fails to explain robust overfit- ting, particularly after the LR decay. Therefore, propose new dynamic robust feature framework to account for this change by taking the interplay among the model trainer and the attacker into consideration.
Based on our dynamic framework, we analyze the change of feature robustness during LR decay. We further propose a new understanding of robust overfitting through the change of non-robust features, and empirically verify it on three nontrivial implications.
From our dynamic perspective, we propose three effective approaches that mitigate robust overfitting by re-striking a balance between the model trainer and the attacker. Experiments show that proposed BoAT largely alleviates robust overfitting and attains -fhthe- art robustness.
2 A NEW DYNAMIC FRAMEWORK FOR ROBUST AND NON-ROBUST FEATURES
In this section, we explain how traditional static framework of feature robustness analysis fails to explain the robust overfitting problem in AT. Motivated by this fact, we establish dynamic view of feature robustness that centers on the adversarial learning process.
2.1 LIMITATIONS OF TRADITIONAL STATIC FEATURE ROBUSTNESS FRAMEWORK
The arguably most prevailing understanding of adversarial training is the feature robustness framework proposed by Ilyas el al. (Tlaas et al.) 2019). They regard natural images as a composition of both robust and non-robust features that can both generalize (i.e.. useful), and the use of non-robust features in standard training results in its adversarial vulnerability. Adversarial training is to prune the non-robust features and the learned robust model only makes use of robust features. Generally, for a distribution P and model class F. we can define the robustness of a feature f E F + V as a real value, and decide whether it is a (non-)robus feature based on a given threshold 7 E [0, 1]:
(lyas et al. 's definition) Rf, = B(era)> min 1[f(x + 8) = v|
(2)
BEA(s)
2

---

Under review as a conference paper at ICLR 2023
Limitations. However, notice that this feature robustness definition fails to explain robust overfitting. Because the robustness of feature f only depends on the attacker A and the distribution P that are both static across training, the robustness of f should also be unchanged under different decays. The core issue here is that they only consider the attacker A, while adversarial training is essentially a dynamic min-max game between two players, the attacker A (inner-loop) and the model trainer T (the outer-loop), both of which are typically based on first-order optimization algorithms. As result, feature robustness will dynamically vary depending on the relative strength between A and T. Another failure of llyas et al. (2019) is that they only consider the population robustness over the distribution P (Eq. 2 and regard robustness as an intrinsic data property. However, the large robust generalization gap in Figure Ia and Tb] shows that a model can behave quite differently even on the same distribution. Therefore, we should distinguish the robustness on the training and test dataset.
To summarize, in order to explain the robust overfitting phenomenon, we need to go beyond Ilyas et al.'s 's original framework, and incorporate the adversarial game between A and T on a specific dataset 'D into the characterization of robust features, as we will do in the next part.
2.2 THE PROPOSED DYNAMIC FEATURE ROBUSTNESS FRAMEWORK
Different from Myas et al. 2019), we define a feature g as a filter of the input g : X - X. Canonical data features (such as, shape, texture) naturally fall into this definition. More importantly, our to explicitly model the attacker A and
definition of feature is clasiffer-independenn which enables the trainer T as two independent factors that affect feature robustness. Specifically, we define the robustness of feature y as a function of the attack region A, the model trainer T, and the dataset D:
(our definition) Ra,T,p(g) = max ERDDD STETT
min llfg (x + 5) - v1 BEA
(3)
where FT C. F denotes the attainable subset of models under the trainer T (a specific training configuration), and FJ further denotes the subset of F that utilzes feature g Many choices of the trainer T affect have large impact on Fr,, such as the optimizer. learning rate and its schedule, weight decay, etc. In particular, as shown in Figure the change of learning rate has a huge impact on model robustness. Theoretically. [L et al. (2019) prove that different learning rates indeed have strong regularization effect on NN traming. Therefore, in this work, we mainly study the effect of LR decay on the change of feature robustness.
Besides the introduction of dataset D, a key difference from/lyas et al. (2019) is that in our framework, the robustness of feature is not decided as a priori. but dependent on dynamic game between A and T. Specifically, a feature is robust if the trainer T can find a model So that is resistant against the attacker A. otherwise, 9 is non-robust and discarded by the trainer. More importantly, this situation could change over time. When we adopt smaller LR after decay, the trainer poses different regularization effect on the model class FT. As pointed out by Li et al. (2019), small LR often favors susy-to-generalize and hard-to-fit patterns. In the context of adversarial training, these features are often non-robust features (before IR decay). Indeed, non-robust features enjoy good generalization (Iyas et al. 2019), but they are hard to fit under the attacker. However, with a smaller I.R, the trainer becomes more capable of memorizing these features by drawing more complex decision boundaries in a small local region. Therefore, due the change of relative strength between A and T, these non-robust features could become robust after LR decay.
To verify the change of robustness, following a similar procedure as Ilyas et al. (2019) (details in Appendix B.3) we extract non-robust features from models before LR decay, and examine their robustness after LR decay.. We also incorporate a control group that adopts the same training configuration except for the LR decay. Results are shown in Figure Ic Compared with the control group with constant LR (blue line), we can increase in feature robustness on the training set (48% + 77%). Therefore, indeed there are many non-robust features that become robust on the training set due to LR decay, as hypothesized above. However, we also notice that on the test set, robustness only changes slightly (25% - 30%), and the non-robust features largely remain non-robust. This indicates that the robustness obtained by these non-robust features after decay are not generalizable from the training set Derain to the test set Drest- The discrepancy in feature robustness shows that the rise of training robustness does not consistently
that LR decay
introduces a significant
Formally, V fg E 3b: X + Y, such that fg = h c Accordingly. given a threshold y. we call a feature g he F, roouutttt if RA,F.D(g) 2 7, and non-robust feature otherwise. We denote the of all (A, F, Dobuut features G F,
3

---

Under review as a conference paper at ICLR 2023
epeck 60
epeck 200
before LB decay
viloon LRdeay
wib LR: Joes
Liameyy
Prodicsal EA-A 11-4
und
A-A 2008
A A 137
(a) larget-class information in test robust features
(b) bilateral class correla-
(c) confusion matrix symmetry
tion
Figure 2: Empirical verification of our explanation for robust overfitting. (a) After LR decays, test non-robust features become less and less informative (from 64.79% to 40.87% of the classes to which they are misclassified. (b) Increasingly strong correlation between rrainingttime A + B misclassification and test-time B + A miselassification increase. (c) Test-time confusion matrix of adversarial examples indeed becomes symmetric with LR decay. More details in Section 2.3
lead to higher test robustness, leading to a large robust generalization gap. Nevertheless, we still do not fully understand why test robustness even degrades shortly after LR decay (Figure lb). Below, we show that it is also induced by fitting these non-robust features with smaller LR.
2.3 ROBUST OVERITTING FROM THE LENS OF DYNAMIC FEATURE ROBUSTNESS
In this section, we study how the change of feature robustness during LR decay induces the robust overfitting phenomenon. As discussed above, significant change during LR decay is that many non-robust features become robust on the training set, while remaining non-robust on the test set. To begin with, we design a controlled experiment showing that fitting these non-robust features on the training set is strongly correlated with the degradation of robustness on the test set.
Memorizing More Non-robust Features Brings More Harm to Test Robustness. To study the effect of non-robust features alone, we deliberately add non-robust features (generated with different perturbation budgets to the clean CIFAR- dataset, where larger s indicates stronger non-robust features in the synthetic dataset (details Appendix [.3).. As shown Figure [d] with LR decays, stronger non-robust features induce severe test robustness degradation, while it is much less severe without LR decay. This shows that the memorization of non-robust features induced by LR decays indeed hurts test robustness a lot. But how does this robust overfitting phenomenon happen?
A Hypothetical Explanation for Robust Overfitting, From the perspective of dynamic feature robustness, the essential role of LR decay is to break the original balance between the attacker A and the trainer T. The decay tips the scale towards the trainer who seizes more and more adversarial examples (containing non-robust features) and push up the training robustness. However, it is not no cost. In the analysis below, we show that it also sets trap for itself in the test-time attack. 1) Model Learns False Mapping of Non-robust Training Features after Decay. Consider an adversarial example x from class A that is misclassified to class B 7 A before decay. According to Ilyas et al. (2019), the non-robust feature g of x belongs to class B. After LR decay, the trainer T becomes more capable of memorization with a smaller LR, and we assume that it successfully learns to map this adversarial example x to its correct label A. Accordingly, the non-robust feature from class B in æ is also (falsely) mapped to class A. 2) False Non-robust Mapping Opens Shortcuts for Test-time Adversarial Attack. Considering a test sample f from class B, before LR decay, the attacker A has to add non-robust features from a different class, say A, in order to misclassify it. However, after decay, because the existence of false non-robust mapping, the attacker A can simply add a non-robust feature from the same class, denoted as such that it is misclassified to wrong class A. In other words, the injected non-robust features do not even have to come from different class, which makes it much easier for the attacker to find when starting from & (compared to non-robust features from other classes). Therefore, the attacker will have a larger attack success rate with the help of these false non-robust mappings, as they create easy-to-fin shortcuts for adversarial attack. Summary: when the balance breaks, the trainer has an (overly) strong fitting ability such that it learns false mapping of non-robust features. At test time, these non-robust features induce shortcuts for adversarial attack, which leads to robustness degradation. From a causal perspective, these non-robust features are spurious features (Zhang et al., 2022; Sagawa et al. 2020). Therefore, when they falsely memorized, would also induce significant performance degradation.
Empirical Verification. On real-world data like images, it is hard to know what the data distribution or the non-robust features actually are. Instead, we validate this theory by verifying some nontrivial yet testable implications of it. We refer the readers to Appendix B.3 for details of these experiments.
4

---

Under review as a conference paper at ICLR 2023
feef
2047
(a) robustness comparison
(b) flatness comparison
(c) decay factor
(d) attack strength
Figure Mitigating robust overfitting from several different perspectives. (a. b) The purposed BoAT loss successfully boosts model robustness and mitigates robust overfitting by bootstrapping adversarial loss landscape flatness between online and WA model. (c) Small LR decay factor leads to better best-epoch robustness of WA model with only slight overfitting. (d) Stronger training attacker also helps mitigate robust overfitting and boost robustness.
Lack of Target-class Features in Adversarial Examples. In our example above, the added non- robust feature g comes from the same class B as &, so the resulting adversarial example s' actually has little features of class A, even if it is (mis)classified to class A. Figure 2a shows that adversarial examples generated by models after LR decay indeed have less and less target-class features after the decay. 2) A Strong Bilateral Class Correlation. Our theory suggests that if overfitting happens, the training misclassification from class A + B (before decay) will induce increase in test misclassification from class B - A (after decay) as + B false non-robust mappings create B + A shortcuts. We verify this in Figure 2b. where we observe that this bilateral correlation indeed consistently increases after LR decay, while remaining very low (nearly no correlation) without decay. 3) Symmetrized Confusion Matrix. As a result of the bilateral correlation, interestingly, we can deduce that the A - B and B - A misclassification will have more similar error rates, which means that the confusion matrix of test robustness will become more symmetric. Indeed, as shown in Figure 2c the confusion matrix (denoted as C) becomes much more symmetric (smaller distance between A and ) after LR decay, particularly when compared to constant learning rate.
The three experiments above help justify our understanding of robust overfitting from the perspective of dynamic feature robustness. In the next section, we further show that this theory also sheds new lights on the design of adversarial training algorithms for alleviating robust overfitting.
3 MITIGATING ROBUST OVERATTING FROM A DYNAMIC PERSPECTIVE
In Section 2 we have introduced our dynamic feature robustness and explain robust overfitting from the shortcut effects of memorizing non-robust features. In this section, we further investigate how to alleviate robust overfitting. From the perspective of our theory, the main goal is to prevent the model trainer T from fitting non-robust features too quickly and too adequately. As AT is a dynamic game between the trainer T and the attacker A, we can either weaken the trainer's ''s fitting ability (Sections
3.1 & 3.2 or strengthen the attacker's attacking ability (Section 3.3 as we show below.
3.1
MODEL REGULARIZATION WITH BOOTSTRAPPED FLATNESS
In order to suppress robust overfitting of the model trainer T after LR decays, we need to put further regularization its local fitting ability, e.g., by enforcing better landscape flatness. In this way, we could prevent the learner from drawing complex decision boundaries and overfit to non-robust features. Indeed, Stutz et al. (20TT) find that models with less robust overfitting often enjoy better landscape flatness. Among existing works, there are mainly two approaches for better flatness. One is AWP (Wu al. (200)) by adversarially perturbing weights, and one is weight average (WA) (Izmailov) et al. 2018) with equal moving average of model parameters. Compared with AWP that introduces extra training steps and computation cost, WA is rather simple and efficient, as model average brings neglectable training overhead. Therefore, we focus on WA in this work.
Along the online model fe that is adversarially trained, we also maintain a WA model f.g that is an exponential moving average of the online parameters 0 é + (1-r) 0. where 7 [o, is the decay rate. Although WA can produce a flatter landscape (Izmailov et al., (008)), we notice that it is not enough to prevent robust overfitting. As shown in Figure 3al the WA model still overfits shortly after the online model overfits. |Rebuffi et al. (202T) further combine WA with CutMix (Yun et al.)
2019). but it still cannot alleviate robust overfitting under longer training (see Figure 4ap. Here, we consider alternative solution with new regularization loss.
5

---

Under review as a conference paper at ICLR 2023
We notice that when the online model fo deteriorates, the WA model, although flatter, will eventually overfit as it is only moving average of fa. Therefore, we also need improve the flatness of the online model as well. How to achieve this? Inspired by the latent bootstrap mechanism in self-supervised learning (Grill et al. 2020). we realize that the WA model f is flatter version of the online model So itself, and thus we can bootstrap its own flatness by aligning the online model with the WA model (as a target). Specifically, given an adversarial example z generated by PGD attack as in AT, we devise new regularization term that aligns their predictions using KL divergence,
(BAAT y; 0) = LCE (CF(f((z)))) + A KL (fo()))lf)z))
where À is coefficient balancing the CE loss (for better training robustness) and the regularization term (for better landscape flatness). We name it as Bootstrapped Adversarial Training (BoAT) loss. Because this loss is designed to alleviate robust overfitting after decay, we only apply WA and BoAT loss after decay to save computation. We also detach the gradients from the WA model as it only serves a flatter target. Compared with CE loss, the KL regularization provides more fine-grained supervision from a flatter model that helps avoid overfitting to training labels. As a result, the online model trained with the BoAT loss will have better flatness than the vanilla AT loss. A flatter online model will further improve the flatness of the WA model, which, in return, also benefits the flatness of the online model. In this way, we could improve the flatness of both online and WA models in a bootstrapped fashion, which, consequently, alleviates the robust overfitting problems of both models.
To verify the intuition above, we compare the AT loss and the BoAT loss under the same training configurations. From FigureBal we can see that contrary to vanilla AT that quickly degrades after decay, the online and WA robustness of the BoAT loss continues to gradually increase after decay. In particular, with AT loss, a typical behavior is that the online model will dramatically jump to high robustness after decay with a stronger fitting ability, but because it overfits to non-robust features, its robustness also dramatically degrades shortly afterwards. Instead, with the bootstrap regularization, BoAT's online model has a slow but steady improvement of robustness, which also contributes to better robustness of the WA model. At last, BoAT's WA model achieves better robustness than AT's best WA model, showing that alleviating robust overfitting indeed helpful for attaining better robustness. When further comparing their flatness at the last checkpoints (Figure (3b), we notice that both BoAT's online and WA models enjoy better flatness than AT's online and WA models, showing that our bootstrap mechanism effectively improves the flatness of both online and WA models.
3.2 SMALLER LEARNING RATE DECAY
The discussion above also suggests that a large LR decay can be harmful for robustness, as it induces severer robust overfitting. A canonical recipe of AT (Madry et al. 2018) is to adopt piecewise linear LR schedule with large decay (e.g., LR divided by d = 10) at certain epoch. As shown in Figure 3a the online robustness quickly jumps up by nearly 10%, and Rice et al. (2020) propose to early stop the training before it deteriorates. However, our discussion on the BoAT loss shows that a longer training without overfitting might bring further improvement over this jump-and and-early-stop strategy.
Apart from the regularization loss (Section 3.I. here we investigate the effect of different decay factors d on robust overfitting. In our dynamic robustness framework, the I.R decay plays an important role, as it breaks the balance between the trainer and the attacker by endowing the trainer with stronger local fitting ability (Section 2.2). However, we also find when it is relatively too strong, the trainer could memorize many non-robust features that are poisonous for its test robustness (Section 2.3 Therefore, a smaller decay rate with milder change of relative strength could help prevent robust overfitting. To verify this, we compare six decay factors d: 1, 2, 3, 4, 6, 10, where d = 10 corresponds to canonical decay and d online model using smaller decay indeed has less robust overfitting, though their best robustness is also lower than d = 10. However, interestingly, when combined with WA, the flatter WA model can benefit lot from smaller decay and improve both best-epoch and final-epoch robustness. Specifically, WA with a very small decay rate d = 2 attains better best-epoch robustness than d = 10 with only slight overfitting. Intuitively, because the WA model is moving average of many online models, for better robustness and flatness, would prefer a group of similar and growing (at least less overfitting) online models (under smaller decay) instead of a group of quickly overfitting models (under large decay). This investigation shows that with flatter landscape like WA, smaller decay is indeed beneficial for both attaining better robustness and alleviating robust overfitting.
corresponds to constant LR (no decay). As shown in Figure 3c| the
3.3 COUNTERING STRONGER MODEL TRAINER WITH STRONGER TRAINIGG ATTACKER
In the discussions above, we have shown that regularizing the trainer's ' local fitting ability with the BOAT loss smaller learning rate decay can both help prevent it from overfitting to non-
6

---

Under review as a conference paper at ICLR 2023
Table I: Comparing our method with several previous methods CIFAR-10 under the perturbation norm coo 8/255 based on PreActResNet-18 and WideResNNet33-100 architecture.
Pre-ctResNet-18
WideResNet-34-- 34--1
Method
Natural
PGD-20
AutoAttack
Natural
PGD-20 best
AutoAttack
best
final
best
final
best
final
best
final
final
best final
PGD-AT
82.08 83.98 52.64
47.06 47.72 42.60
86.09
86.81 56.45 48.16
52.11 45.47
TRADES
80.72 82.61
52.66
49.75 48.37 46.94
84.73
84.62
56.50 47.28
53.25
45,29
WA
83.37 85.07
55.24
47.79 49.92
43.82
87.84
86.83
56.21 49.18
52.41 46.36
KD+SWA
83.82 84.43
54.59
54.42 49.87
49.74
86.85
88.03
56.92 55.74
53.64 53,23
PGD-AT+TE
82.15 82.59
55.03
53.79 50.11
49.14
86.20
85.63
56.89
53.49
53.35 50.81
AWP
81.25 81.56
55.53
54.83
50.34
49.64
86.28
86.28
58.85
58.76
53.26
53.27
BoAT
82.20 82.06
56.59
56.55
51.39
51.36
85.21
85.41
59.82
59.30
54.94
54.69
WA+CutMix
80.18 80.18 56.15
56.11
49.87
49.87
85,41
88.30
60.50
59.32
55.16
54.10
BoAT+CutMix 78.75 78.75 56.30 56.27 50.41
50.39 86.49
86.86 61.72 61.56 56.13 56.23
Table 2: Comparing our method with several training methods on CIFAR-100 and Ting-ImageNet under the perturbation norm & 8/255 based on the PreActResNet- 18 architecture.
CIFAR-100 100
Inny-ImageNet PGD-20 best
Method
Natural
PGD-20
AutoAttack final
Natural
AutoAttack final
best
final
best
final
best
best
final
final
best
PGD-AT
55.52 57.35
29.58
23.02
24.53 20.21
45.54
48.73
22.00
18.08
17.30 14.68
TRADES
55.53 57.09
29.56 26.08
24.51 22.86
47.99
47.79
22.51 21.28
16.66 16.29
WA
57.96 58.81
30,90 24.04
25.95 21.02
48.76 50.00
24.66 19.82
19.76 15.82
KD+SWA
57.23 57.66
30.06
30.02
26.04 25.99
50.59
50.83
24.83 24.79
19.78 19.76
PGD-AT+TE
56.52 57.30
31.23
29.25 26.04 25.13
46.84
50.51
22.89
19.40
18.16 15,88
AWP
53.92 54.81
30.67
30.43
25.26
25.07
43.31
43.30
23.60
23.26
18.08
17.84
BoAT
56.44 56.51
32.70
32.38
27.59
27.52
47.79
48.20
25.21 24.52
20.41 19.88
WA+CulMix
56.84 56.96
32.35
32.34 26.39 26.35 46.58
46.87
24.97
24.87
18.97
18.87
BoAT+CutMix 55.74 55.88 33.03 33.02 27.21 27.20 46.02 46.15 25.04 25.02
19.51 19.50
robust features. As noted in our dynamic framework, AT a dynamic game between the trainer T and the attacker A. Another natural approach is to strengthen the attacker's attacking ability to counter the stronger trainer after decay, which would also help strike new balance between two players. To examine this, we apply four different training perturbation budgets after decay, 2/255 and increasing PGD steps
E 8/255. 10/255, 12/255, 16/255 (with fixed step size (00e/(8//55) where g = 8/255 is the default budget used before decay and in test-time attack. As can be seen from Figure 3d. models trained with stronger attacker (larger e) indeed has much less overfitting, which helps verily perspective. Besides, a too strong training attacker also prevents the trainer from learning useful robust features (under = 16/ (6/2. Empirically, we observe that selecting moderately larger attack strength, E 12 22, attains the optimal best-epoch robustness (see also Appendix B.4]. This shows that we need to re-design the training attacker according to the change of the model trainer.
understanding of robust overfitting through the dynamic game
3.4 OVERALL APPROACH
Above, we have introduced three effective techniques to mitigate robust overfitting from different perspectives: model regularization, learning rate decay, and attacker strength. At last, we combine the BoAT loss and smaller learning rate our final approach. We do not include stronger attacker because stronger attacker often induces a large decrease in natural accuracy. This is because stronger attacks will make all features less robust, which will decrease the number of usable features in the classifier and hurt natural accuracy, although could further bring slight increase of robustness. We include study on this trade-off in Appendix B.4
4 EXPERIMENTS ON BENCHMARK DATASETS
In this section, we evaluate the effectiveness of the proposed BoAT on several benchmark datasets using different network architectures.
7

---

Under review as a conference paper at ICLR 2023
45-
3310-
83
06.3
13
250
Epocc
(a) training longer with CutMix
(b) target-class information in test non-robust features
(c) bilateral class correlation
Figure 4: Empirical understandings on BoAT. (a) BoAT can benefit from training longer (with CutMix), (b) Compared to PGD-AT, test adversarial examples still have to contain fairly much target-class information (measured by accuracy) under BOAT. (c) BoAT does not result in strong bilateral class correlation as in PGD-AT.
Setup. We consider the classification tasks on CIFAR-10, CIFAR- 100 (Krizhevsky et al.) 2009). and
(Deng et al. (2009) with the PreActResNet-J8 18 (He et al. (016) and WideResNet-
34-10 (Zagoruyko & Komodakis 2016) architectures. Besides the vanilla PGD-AT (Madry et al.
2018), we include the following baseline methods for alleviating robust overfitting: WA (Izmailov
et al. (2018), KD+SWA (Chen et al. (022)) PGD-AT+TE (Dong et al. (022)) AWP (Wu et al. (2220) and WA+CutMix (Rebuffi et al.
[022T.. [
During evaluation, use PGD-20 (Madry et al. 2018) 8) and
AutoAttack (AA) (Croce & Hein (020) for the adversarial attack methods. We train each model for 200 epochs and record the best-epoch (selected based on PGD attack) and final-epoch robust test
accuracy. Please refer Appendix B.5 and B.6 for details and Appendix C for additional results.
4.1 RESULTS ON BENCHMARK DATASETS
Performance across Different Networks. In Table we evaluate BoAT against previous AT variants on two popular backbone networks: PreActResNNt--1 and WideResNet-34- 10. We can that without strong data augmentations like CutMix, BoAT attains state-of-the-art robustness on both models and outperforms previous methods by at least 1% against AutoAttack (AA), arguably the strongest white-box attacker. Meanwhile, BoAT almost alleviates robust overfitting NreActResNet- 18 with only 0.03% gap (51.39% best and 51.36% final), and the gap is also small (0.30%) on WideResNet. Because of this, remarkably, the final-epoch robustness of BoAT even consistently outperforms the best-epoch robustness of previous approaches using early stopping. Therefore, with BoAT, we might not need per-epoch validation and early stopping techniques any more, which helps save training computation cost, simplify the training recipe, and improve the applicability of AT.
has larger capacity, and existing WA+CutMix strategy (Rebutti et al.) 202T) still overfits by more than 1 1% (55.16% best and 54. 10% last). In comparison, -oAmix achieves 56. best robustness (chosen according to PGD) and 56.23% final robustness, showing that this combination indeed alleviates robust overfitting on WideResNet and attain state-of-the-ar robustness. In comparison, CutMix brings little improve- ment PreActResNet-118 mainly because this model has lower capacity and still underfits under strong augmentations, and requires longer training reach higher performance. Even though, BoAA+CCtMix still outperforms WA+CutMix by 0.5% AA robustness by the end of 200 epochs.
Strong Augmentations. WideResNet is more prone to overfit
Performance across Different Datasets. We also conduct experiments on two additional benchmark datasets, CIFAR- 100 and Liny-ImageNet, that are more complex than CIFAR-10. As shown in Table 2. our BoAT and BoAT+Cuumix still achieve the highest best and last robustness, demonstrating its scalability larger scale datasets. Similarly, CutMix does not help much as often leads to underfitting with stronger regularization effects within a relatively short training length.
4.2 EMPIRICAL UNDERSTANDINGS
BoAT Can Benefit from Longer Training. Table 1 suggests that when CutMix is applied to
PreActResNet-I1 18 model, demands longer training fully unleash the beast. This motivates us to examine the effect of longer training, e.g. 500 epochs. As shown in Figure 4a| longer training with WA+CutMix brings little improvement final robustness (49.87 (49.87% (200 epochs) epochs) under AA). Instead, BorT+CurMix achieves higher best robustness and the robustness keeps improving along training (> 1% improvement under AA), In particular, the additional 300 epochs improve the final AA robustness of BoAT+CutMix from 50.39% 11.28% This provides strong
49.90% (500
8

---

Under review as a conference paper at ICLR 2023
evidence on the effectiivness of BoAT against robust overfitting. It also shows that combining BoAT with strong data augmentations can benefit from long training as standard training.
Verification on Robust Overfitting Experiments. Beyond its success, we empirically study how BoAT manages to suppress robust overfitting through a lens of feature robustness. To this end, following the experiments in Section 2.3 we compare PGD-AT and BoAT in terms of 1) the amount of targtt-laass features of test adversarial examples and 2) the strength of bilateral class correlation. First, we can see that test adversarial examples still have to contain much target-class features to be misclassified (Figure (4b) and there is no longer a strong bilateral class correlation (Figure 4c) This confirms that BoAT indeed successfully prevent the model from learning false mapping of non-robust training features which creates shortcuts for test-time attack.
REVISITING EXISTING APPROACHE TO ALLEVIATE ROBUST OVERFITTING
Remind that our definition of feature robustness (Eq. 3) is composed of three factors: the dataset D, the model trainer T, and attacker A. Based on our understanding of robust overfitiing (RO), we can categorize existing AT methods that help alleviate RO into the following three classes.
Data Regularization. As RO is induced by non-robust features in training inputs, one way is to corrupt non-robust features with various input transformations. As can be seen from [lyas et al. (019))
non-robust features often correspond to local and high-frequency patterns in images. Therefore, stronger augmentations like CutMix will help mitigate RO, as analyzed by Rice al. (2020).
Recently, Li et al. (2022) show that random or even fixed masks can eliminate the catastrophic overfitting of FGSM-AT. As the FGSM attacker is much weaker than PGD, FGSM-AT would be more prone to overfit, and we reckon that the masks help corrupt non-robust features in original data. A complementary approach is to correct the supervision of the adversarial examples accordingly to avoid learning false label mapping (Section 2.3), e.g., the soft labels adopted in Dong et al. (2021). and the distillation-aliae methods KD+SWA (Chen et al. 2021) and TE (Dong et al.) 2022).
Training Regularization. A necessary condition of RO is that the model trainer is stronger than the attacker, which breaks their original balance. Therefore, we can prevent RO by regularizing the trainer to be relatively weaker on fitting non-robust features. One approach is directly promote model flatness, e.g. AWP (Wu et al. 20200) EMA Rebuffi et al. (2021). S SWA [Cen et al. (0021), ) and our BoAT loss (Section 3.I). Yu et al. (222)) propose regularize samples of small loss values (typically using non-ro Teatures). Another approach to adjust the learning rate schedule to
avoid dramatic change of robustness that could be irreparable, as studied in Section 3.2] Previously, Rice et al. (2020) and WWang& & Wang (022)) also study different types of learning schedules, while we focus on the effect of LR decay factors on robust overfitting. In particular, we firstly show that piecewise decay with small decay factor can be helpful for both better robustness and less RO.
Stronger Training Attacker. Another way restore the balance is to adopt stronger training attacker after LR decay, as we have explored in Section 3.3 Previously, from a curriculum training perspective, Cai et al. (2018), Qin et al. (2019) and Wang et al. (2019) propose to start training from a weaker attacker (e.g., E 0), and gradually lift its strength These strategies are designed as a warmup stage at the earlier training stages. Pang et al. (2020) point out that these warmup methods are hardly useful when measured by AA robustness. Instead, our method is designed from a dynamic game perspective, and we propose to adopt a stronger training attacker (s > 8/255) after the LR decay in order to counter robust overfitting. Besides, our method shows clear benefits improving best AA robustness well as mitigating robust overfitting.
the standard level (e.g., E = 8 3/255).
6 CONCLUSION
In this paper, we investigate the underlying cause of robust overfitting through the lens of dynamics feature robustness. To achieve this, we have proposed a new dynamic definition of feature robustness by taking the attacker A, the model M, and the dataset 2 into considcration. Using our new frame- work, we analyze the change of robust and non-robust features during the learning rate decay, and attribute robust overfitting to the falsely memorized non-robust features. Based on this understanding. we have proposed three new approaches to restore the balance between the model trainer and the attacker. Experiments Training (BoAT) largely resolves the overfitting issue, even under very long training. Meanwhile, it also achieves state-of-the oof-the-art art robustness even without early stopping, showing that adversarial training can also benefit from longer training as long as the overfitting issue is resolved.
benchmark datasets show that our proposed Bootstrapped Adversarial

---

Under review as a conference paper at ICLR 2023
REFERENCES
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In ICMI, 2018.
Qi-Zhi Cai, Min Du, Chang Liu, and Dawn Song. Curriculum adversarial training. In IJCAI, 2018.
Tianlong Chen, Zhenyu Zhang, Sijia Liu, Shiyu Chang, and Zhangyang Wang. Robust overfitting may be mitigated by properly learned smoothening. In ICLR, 2021.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In ICML, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Chengyu Dong, Liyuan Liu, and Jingbo Shang. Double descent in adversarial training: An implicit label noise perspective. arXiv preprint arXiv:21 10.00135, 2021.
Yinpeng Dong, Ke Xu, Xiao Yang. Tianyu Pang. Zhijie Deng, Hang Su, and Jun Zhu. Exploring memorization in adversarial training. In ICLR, 2022.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. In NeurlPS, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In ECCV, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.00 02553., 2015.
Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generaliza- tion gap in large batch training of neural networks. NeurIPS, 2017.
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In NeurlPS, 2019.
Pavel Izmailov. Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson, Av- eraging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.0555 1803005407 2018.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Yuanzhi Li, Colin Wei, and Tengyu Ma. Towards explaining the regularization effect of initial large learning rate in training neural networks. NeurlPS, 2019.
Zichao Li, Li Liu, Zeyu Wang, Yuyin Zhou, and Cihang Xie. Bag of tricks for fgsm adversarial training. arXiv preprint arXiv:220900266 2022.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018.
Tianyu Pang, Xiao Yang, Yinpeng Dong, Hang Su, and Jun Zhu. Bag of tricks for adversarial training. In ICLR, 2020.
Chongli Qin, James Martens, Sven Gowal, Dilip Krishnan, Krishnamurthy Dvijotham, Alhussein Fawzi, Soham De, Robert Stanforth, and Pushmeet Kohli. Adversarial robustness through local linearization. In NewrlPS, 2019.
Sylvestre-Alviss Rebuffi, Sven Gowal, Dan Andrei Calian, Florian Stimberg, Olivia Wiles, and Timothy A Mann. Data augmentation can improve robustness. In NeurlPS, 2021.
Leslie Rice, Eric Wong, and Zico Kolter, Overfitting in adversarially robust deep learning. In ICML, 2020.
10

---

Under review as a conference paper at ICLR 2023
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. In ICLR, 2020.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. ICLR, 2015.
David Stutz, Matthias Hein, and Bernt Schiele. Relating adversarially robust generalization to flat minima. In ICCV, 2021.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In ICIR, 2014.
Hongjun Wang and Yisen Wang. Self-ensemble adversarial training for improved robustness. In ICLR, 2022.
Yisen Wang, Xingjun Ma, James Bailey, Jinfeng Yi, Bowen Zhou, and Quanquan Gu. On the convergence and robustness of adversarial training. In JCML, 2019.
Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust general- ization. In NeurIPS, 2020.
Chaojian Yu, Bo Han, Li Shen, Jun Yu, Chen Gong, Mingming Gong, and Tongliang Liu. Under- standing robust overfitting of adversarial training and beyond. In ICML., 2022.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh. Sanghyuk Chun, Junsuk Choe. and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.
Hongyang Zhang. Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In ICML., 2019.
Yonggang Zhang, Mingming Gong, Tongliang Liu, Gang Niu, Xinmei Tian, Bo Han, Bernhard Schölkopf, and Kun Zhang. Adversarial robustness through the lens of causality. 2022.

---

Under review as a conference paper at ICLR 2023
A ADDITIONAL EXPERIMENT: ADVERSARIAL EXAMPLES (IN ADVERSARIAL TRAINING) CONTAIN NON-ROBUST FEATURES
At the end of Section 2.2 we craft adversarial examples model A and evaluate the misclassified ones on another model B with their correct labels to indicate the amount of features that have become robust. We argue that this is reasonable because adversarial examples used for AT' consist of robust features from the original label y and non-robust features from the misclassified label g. B must have robustified those features (non-robust on A) to correctly classify them. this section, we further provide a rigorous discussion on this through an additional experiment.
Experiment Design. Recall that Ilyas al. (2019) leverage targeted PGD attack (Madry et al. (018)) to craft adversarial examples on a standard (non-robust) classifier and relabel them with their target labels to build a non-robust dataset. Finding standard training on it yields good accuracy on clean test data, they prove that those adversarial examples contain non-robust features corresponding to the target labels (Section 3.2 of their paper).
clean training data
relabel as cat
evaluate natural accuracy
(untargeted) PGD attack
standard training
good accuracy (> 55%)
dog
cat
cat
robust features: dog non-robust features: dog
robust features: dog non-robust features: call
robust features: cat non-robust features: cat
Figure 5: Mining non-robust features from adversarial examples in adversarial training. 1) Craft adversarial examples å; using untargeted PGD attack on AT checkpoint. 2) Relabel them (only misclassified ones) with misclassified label ú to build a non-robust dataset {(ē,, 9t)}. 3) Perform standard training from the checkpoint. 4) Achieve good natural accuracy (> 55%).
However, different from their settings where non-robust features lie statically in the examples, we lay emphasis on mining non-robust features from adversarial examples generated on-the-fly ffy in AT. To this end, we first craft ad adversarial examples on the AT checkpoint at some epoch using untargeted PGD attack (Madry et al, 2018) to restore the real setting of AT, then relabel the misclassified ones i;; (e.g., dog) with their misclassified labels ÿi # yi (e.g., cat) build non-robust dataset {( ú:) Also, in order to capture non-robust features at exactly the training epoch these adversarial examples are used, we continue to perform standard training directly from the checkpoint on the non-robust dataset, and finally evaluate natural accuracy. See Figure 5 for an illustration of our experiment.
In details, we select the AT models at epoch 60 (before LR decay) and epoch 1,000 (after LR decay) to conduct the above experiment. We use untargeted PGD-20 with perturbation norm 6 16 255 to craft adversarial examples on those checkpoints from the training data of CIFAR-10 10 (Krizhevsky et al. 2009). The attack we adopt is a little bit stronger than the attack of PGD-10 with 6oc 8/255 that commonly used to generate adversarial examples in baseline adversarial training, because the training robust accuracy rises to as high as 94.67% at epoch 1,000 (Figure Lab and gives less than 2,700 misclassified examples, which is sigmificantly insufficient for further training. Using the stronger attack, we obtain success rate of 78.38% on the checkpoint at epoch 60 and success rate of 34.86% on the checkpoint at epoch 1,000. Since the attack success rates are different, we randomly select same number of misclassified adversarial examples for standard training for fair comparison. The learning rate is initially set to 0.1 and decays to 0.01 after 20 epochs for another 10 epochs of fine-tuning Given that the original checkpoints already have nontrivial natural accuracy, we also add two control groups that train with random labels instead of to exclude the influence that may brought by the original accuracy.
Results. As shown in Figure we find that standard training on the non-robust dataset {(2,, 0:)) successfully converges to fairly good accuracy (> 55%) on natural test images, i.e., predicting cats as cats, no matter from which AT checkpoints (either epoch 60 or epoch 1,000) the adversarial examples are crafted. Also, we can see that the nontrivial natural accuracy has nothing to do with the original accuracy of the AT checkpoints, soon as the standard training starts with random labels. This proves that adversarial examples in adversarial training do contain robust features w.r.f the classes to which they are misclassified, which strongly corroborates the validity of the experiments in Section 2.2
the accuracy plummets to around 10% (random guessing) as
12

---

Under review as a conference paper at ICLR 2023
80
70
Natural accuracy (%)
80
epoch 50 (misclassified labels)
30-
(misclassified labels)
epoch 60 (random labels)
(random
Epoch (ST)
Figure 6: Natural accuracy during standard training. Standard training the non-robust dataset built from the checkpoint at either epoch 60 or epoch 1,000 converges to fairly good natural accuracy (>> 55%). The failure of training with random labels proves that the good accuracy has nothing to do with the original natural accuracy of the AT checkpoint.
A Discussion on the Influence Brought by Robust Features During Standard Training. Since untargeted PGD attack cannot be assigned with a target label, we cannot guarantee that the misclassi- fied labels to be uniformly distributed regardless of the original labels (especially for real-world datasets). This implies that the non-robust features
completely decoupled from robust features, ie., training on {( 0:)} may take advantage ft;" robust features from y/ through the correlation between y, and D. However, we argue that robust features from y mingling with non-robust features from i only increases the difficulty of obtaining good natural aceuracy. This is because learning through the shortcut will only wrongly map the robust features from y; to D; that never equals to yi. but during the evaluation, each clean test example will always have robust and non-robust features from yı. and such wrong mapping will induce x; be misclassified some Ú; due the label of its robust features. As a result, despite the negative influence brought by robust features, we still achieve good natural accuracy at last, which even solidifies our conclusion.
B EXPERIMENT DETAILS
In this section, we provide more experiment details that are omitted before due to the page limit.
B. BASELINE ADVERSARIAL TRAINING
In this paper, we mainly consider classification task on CIFAR-10 (rrihevskyy et al. 2009). The dataset contains 60,000 32 x 32 RGB images from 10 classes. For each class, there are 5,000 images for training and 1,000 images for evaluation.
For baseline adversarial training. We use PreActResNet-18 18 (He et al. 2016) model as the classifier. We use PGD-10 10 attack (Madry al. 2018) with step size a 2/255 and perturbation norm Exc =8 =8/255 to craft adversarial examples on-the-fly. We use SGD optimizer with momentum 0.9, weight decay 5 10 and batch size 128 train the model for as many as 1,000 epochs. The learning rate (LR) is initially set to be 0.1 and decays to 0.01 at epoch 100 and further decays to 0.001 at epoch 150. For the version without LR decay used for comparison in paper, simply keep the LR be 0.1 during the whole training process.
Each model included in this paper is trained on a single NVIDIA GeForce RTX 3090 GPU. For PGD-AT, it takes about 3d 14h to finish 1,000 epochs of training.
B.2 LEARNING RATE DECAY INDUCES ROBUST OVERFITTING
In this paper, one of our most essential observations is that RO often happens after LR decays. We further verify that this is the general case instead of phenomenon that happens to occur only when
13

---

Under review as a conference paper at ICLR 2023
the LR first decays epoch 100 by allowing it to decay different times during the training. Figure
7 demonstrates that whenever the LR decays, the test robust accuracy immediately increases by more Than 5% and then slowly drops, implying that RO is indeed caused by LR decay.
Robust Accuracy (%)
52
50
48
44
42
0
50
100
150
200
Epoch
Figure 7: Test robust accuracy CIFAR-10 under the perturbation norm Eno 8/255 based on the PreActResNet-18 architecture. The gray vertical lines mark the epochs where the LR decays.
B.3 DYNAMIC FEATURE ROBUSTNESS
Features of the Training Data Become Robust After LR Decay. We craft adversarial examples on checkpoint before LR decay and evaluate the misclassified ones on checkpoint after LRR decay with their correct labels evaluate the amount of features that have become robust (see Appendix A for detailed discussions) at the end of Section 2.2 Following Appendix [A] we adopt PGD-20
attack with perturbation norm & 16 16/255 to craft adversarial examples which is stronger than the common attack setting we use in PGI-ATT We note that test adversarial examples crafted by stronger attack indicates stronger extraction of their non-robust features, so they are more indicative of feature robustification when still correctly classified.
Memorizing Non-robust Features Harms Test Robustness. At the beginning of Section 2.3 we
create synthetic datasets to demonstrate that memorizing the non-robust training features indeed harms test-time model robustness. To instill non-robust features into the training dataset, we minimize the adversarial loss w.t.. the training data in a way that just like PGD attack, with the only difference that we minimize the adversarial loss instead of maximizing it. Since we only use very small bound to be non-robust, For fair comparison,
perturbation norm 6 S 4/255, the added features we also perturb the training set with random uniform noise of the same perturbation norm to exclude the influence brought by (slight) data distribution shifts, We continue training from the 100-th baseline AT checkpoint (before LR decay) on each synthetic dataset for 10 epochs, and then evaluate model robustness with clean test data.
Verification:
Lack of Target-class Features. This is to say that when RO happens, expect that test adversarial examples become less informative of the classes which they are misclassified according to our theory. To verify this, we first craft adversarial examples on checkpoint 7' after RO begins, then evaluate the misclassified ones with their misclassified labels saved at epoch 60. As a result, the accuracy reflects how much features from ý the examples have to contain to be misclassified on T. All adversarial examples evaluated in the experiments in Section 2.3 are crafted using PGD-10 attack with perturbation norm & 8/255.
the checkpoint
Verification: Bilateral Class Correlation. quantitatively analyse the correlation strength of bilateral misclassification described in Section 2.3 empirical verification 2, we first summarize all y + By misclassification rates into two confusion matrices pirain ptest e CCC for the training and test data, respectively. Because we are mainly interested in the effect of the LR decay, we focus on the relative change on the test data, i.e., ptest ptest- ptest. According our analysis above, for each class pair (), there should be a strong correlation between the training misclassification of + j before LR decays, i.e., (P )ej. and the increase test misclassification of - i, i.e., A To examine their relationship, we plot the two variables ((Ptraim) )ij. A A and compute their Pearson correlation coefficient p.
14

---

Under review as a conference paper at ICLR 2023
Pootuce lhtd
Poddstd label
Thee lobel
Tree label
Tree ludl
Tree labe
True kubal spech
spoch
50
Prediced Jobe
Pridced
1pech
Trac lindl
Tree label
The lahel
Frad label
True labul
opech 120
epech 140
spach 160
epech 150
spech 200
Figure 8: More test-time confusion matrices during the first 200 epochs of the training. After LR decays (the second row), the confusion matrix A immediately becomes symmetric, as the spectral norm A -
--A1 All W.r.f. the matrix decreases from 2 350 before epoch 100 to S 150 after epoch 200.
+LAM
700
320
25400
1:C
SC3
4BC
357
1522
3000
P1
76
2002
200
True Vob
2500
800
D7T
P45
2002
IST
1200
103
37M
22
LMS
24h
1000
30
210
126
39
300
334
20
300
(a) train
(b) test
Figure 9: Confusion matrices of the training and the test data at an epoch before robust overfitting starts. They show nearly a same pattern of attacking preference among classes.
Verification: Symmetrized Confusion Matrix. In Section 2.3 we mention the growing symmetry of the test-time confusion matrix after LR decays as an evidence of the strengthening A - B and B + A correlation. Here we present more confusion matrices during the first 200 epochs of the training in Figure [8 and it very clear that the confusion matrices soon become symmetric after LR decays at epoch 100 when robust overfitting starts. For a deeper comprehension of this phenomenon, first visualize the confusion matrices of the training and the test data an epoch before robust overfitting starts in Figure
9 They exhibit nearly a same pattern of attacking preference among classes (e.g.. A + B) due to the bias rooted in the dataset, e.g., class 6 is intrinsically vulnerable in this case. For the test data, this intrinsic bias wouldn't 't be wiped out through learning due to the non-genere non-generalizabiiity of robustified training data features demonstrated in Section (i.e., A + B still holds); and for the training data, this biased feature memorization will open shortcuts for test-time adversarial attack as discussed in Section 2.3 (i.e., B + A begins). Combining both A B and B + A. we arrive at the symmetry of test-time confusion matrix.
Changing LR Decay Schedule. Our discussion above is based on the piecewise LR decay schedule, in which the sudden decay of LR most obviously reflects our understandings. Besides, we also explore using other decay schedules. including Cosine/Linee decay schedules will affect the observations and claims we made in this paper. For each schedule, we train the model for 200 epochs following the settings in Rice et al. (2020). As demonstrated in Figure T0a] we arrive at the same finding as Rice et al. (2020) that with Cosine/I Linear LR decay schedule, the training still suffers from severe RO after epoch 130. Then, we rerun the three empirical experiments Section 2.3 and find that under both two LR decay schedules 1) the test adversarial examples indeed contain less and less target- sarget-class information as the training goes (Figure [0b).. 2) the bilateral class correlation becomes increasingly strong (Figure 10c) and 3) the confusion matrix
LR decay, to check whether different
15

---

Under review as a conference paper at ICLR 2023
mantevttee (h)
Cosire Liswer
Casre Lines
Peeeroon coweficceet
Costne
-2-
iineer
satest aooroooraa scorscy
2-3
42-
0.2
100
203
340
230
250
300
100
140
130
1.0
172
100
190
200
Spocn
Opoc1
Opocn
(a) test robust accuracy during train- ing
(b) target-class information in test robust features
(c) bilateral class correlation
Figure 10: Empirical verification of our explanation for robust overfitting when Cosine/Lincal LR decay schedule is applied. (a) With Cosine/L.nnet LR decay schedule, the training still suffers from severe RO. (b) After RO begins, non-robust features the test data become less and less informative of the classes to which they are misclassified. (c) Increasingly strong correlation between training-time A + B misclassification and test-time B A misclassification increase.
epoch
epuch 200
before LR decay
Cosine
Linear
True labe!
True label
True labo
Predicted label A ll 477.42
Predicted label
Predicted label l:= 164.18
A| AfIh- 132.12
Figure 11: Test-time confusion matrix also becomes symmetric and implies that the bilateral correla- tion also exists when Cosine/Linear LR decay schedule applied.
indeed becomes symmetric (Figure II). The results are almost the same as the results achieved when we adopt piecewise LR decay schedule, indicating that our understandings in the cause of RO is fundamental and regardless of whatever LR decay schedule is used.
B.4 THE EFFECT STRONGER TRAINING ATTACKER
Stronger Attack for Stronger Defense. In Section 3.3 we point out that using a stronger attacker in AT is able mitigate RO to
extent by neutralizing the learning power when it is overly strong. To achieve the results reported in Figure 3d we craft adversarial examples on-the-fly with more PGD iteration steps when & is larger (see Table and further evaluate the best and last robustness of the WA models against PGD-20 and AA.
Table 3: Training with stronger attack and evaluating model robustness on CIFAR-10 under the perturbation norm 6xo = 8/255 based on the PreActResNett-11 18 architecture.
Natural final
PGD-20
AutoAttack
Attack Strength
best
diff
best
final
diff
best
diff final
8/255, PGD- 10 (baseline)
83.37 85.07 -1.70
55.24
47.79 7.45 49.92
43.82 6.10
10/255, PGD-12
80.66 82.48 -1.82
56.63
50.63 6.00
50.89
46.13 4.76
6
12/255, PGD-15
78.17 80.25 -2.08
57.09
53.13 3.96
50.99
47.66 3.33
14/255. PGD-17
73.92 76.70 -2.78
56.42
54.04
2.38
50.28
48.53 1.75
E 16/255, PGD-20
69.51 73.11 -3.60
55.09
54.27 0.82 49.58 48.53 1.05
Although RO is only partially mitigated and natural accuracy decreases when stronger attacker is applied, is surprising find from Table that that an attacker of appropriate strength may significantly boost the best robustness. This suggests using a stronger attack could potentially be an interesting new path to stronger adversarial defense, and we leave it for future work.
16

---

Under review as a conference paper at ICLR 2023
Using Stronger Training Adversarial Attack in BoAT. Above show BoAT is almost capable of eliminating RO. However, since we also show leveraging stronger adversarial attack in AT may help mitigate RO while boosting model robustness above, it is natural to further combine BoAT with this training scheme. According Figure 12, though slightly stronger attack (e.g.. = 9/255) may marginally improves the best and last robust accuracy, heavily degrades natural accuracy, particularly when much stronger attack is used. We deem that this is because it breaks the balance that BoAT has achieved: with relatively weak learning ability of the trainer T, stronger attacker A naturally dominates the adversarial game and results in an underfitting state that harms both robust and natural accuracy.
Accuracy (96)
best natural
last natural
best robust
last robust
55
(x 255)
Figure 12: Using different adversarial attack strength in BoAT.
B.5 MITIGATING ROBUST OVERFITTING WITH BOAT
Datasets. Beside CIFAR-10. 10, we also include CIFAR-100 Krizhevsky et al. 2009 and Tiny- ImageNet (Deng al. 2009) for evaluation of the effectiveness of BoAT. CIFAR- shares the same training and test images with CIFAR-10, but it classifies them into 100 categories, i.e., 500 training images and 100 test images for each class. Liny-mmaeNNet is subset of ImageNet (Deng al. 2009) which contains labeled 64 64 RGB images from 200 classes. For each class, it includes 500 and 50 images for training and evaluation respectively.
Training Strategy. For CIFAR-10 and CIFAR-100, we follow exactly the same training strategy as introduced in B.1 For Imy-eme we follow the learning schedule of of[Chen al. (2021). in which the model is trained for a total of 100 epochs and the LR decays twice (by 0.1) at epoch 50 and 80.
Choices of Hy peperparaaetrrs. For KD+SWA (Chen AWP (Wu et al. 2020) and WA+CutMix (Rebutti et al. 2021). ). we strietly follow their original settings of hyperparameters. SWA/WA well the BoAT regularization purposed in Section 3.] starts at epoch 105 (5 epochs later than the first LR decay where robust overfitting often begins), and following (Rebuffi et al. (021) we choose the EMA decay rate of WA to be y = 0.999. Please refer to Table for our choices of the decay factor d and regularization strength Å. We notice that since CutMix improves the difficulty of learning, the model demands a relatively larger decay factor to better fit the augmented data. For ImageNN, we also apply larger À after the second LR decay to better maintain the flatness of adversarial loss landscape and control robust overfitting.
2021 PGD-AT+TE (Dong et al. 2022
Table 4: Choices of hyperparameters when training models on different datasets using different network architectures with BoAT.
Network Architecture Method
CIFAR-IO/CIFAR-1100
Tiny-ImaeeNet
BoAT
d d= 1.5, 1.0
d = 1, 2.0, 2 10.0
PreA
BoAT+Cutmix
d 4.0,
2.0
6.0, 1.5
BoAT
d 1.3, 0.5
BoAT+Cuumix
d= 4.0, X = 2.0
17

---

Under review as a conference paper at ICLR 2023
B.6 TRAINING ROBUST ACCURACY
Figure shows the robust accuracy change on the training data during the training, Compared with vanilla PGD-AT that yields training robust accuracy over 80% epoch 200, BoAT manages sup- press to only 65%. It successfully prevents the model from learning the non non-robust features w.r.t. the training data too fast and too well, and therefore significantly reduces the robust generalization gap (from 35% 9%) and mitigates robust overfitting.
BoAT
Robust Accuracy (%)
70
60
40
80
100
120
160
180
200
Epoch
Figure 13: Training robust accuracy of PGD-AT and BoAT on CIFAR-10 under the perturbation norm 6oo 8/255 based on the PreActResNet-188 architecture.
C MORE EXPERIMENTS ON BOAT
In this section, we conduct extensional experiments on the proposed BoAT method to further demonstrate its effectiveness, efficiency and flexibility.
C.I ABLATION STUDY
Effectiveness BoAT Loss. In Section 3.1 Figure 3a has demonstrated the effectiveness of BoAT loss. furthrr report the detailed robustness against PGD-20 and AA, showing that the purposed regularization not only boosts the best robustness by large margin (0.67% higher against AA) but also significantly suppresses RO (1.52% v.s. 6.10% against AA). To achieve the reported robustness, we first use A 10.0 after the first LR decay and then apply X2 = 60.0 after the second LR decay better maintain the flatness of adversarial loss landscape and control robust overfitting.
Table 5: Comparing model robustness w/ and w/o BoAT loss CIFAR-10 under the perturbation
norm Eoo 8/255 based on the Pre 18 architecture.
Natural
PGD-20
AutoAttack
Method
best
final
dill
best
final
diff
best
final
diff"
WA BoAT(d = 10)
83.37 85.07
-1.70
55.24
47.79
7.45
49.92
43.82 60
81.54 82,42 -0.88
55.58
53.74
1.84
50.59
49.07 1.52
Larger Decay Factor Goes with Stronger BoAT Regularization. Below we show that when relatively large decay factor is applied, i.e., the model has overly strong fitting ability that results in robust overfitting, a large regularization coefficient À should be choosen for better performance. Table reeveals this relationship between d and X. When d = 1.3, even À as small as 1.0 will harm both the best and last robustness as well as natural accuracy, as d 1.3 is already too small decay factor that makes the model suffering from underfitting and naturally requiring no more flatness regularization. On the other side, when d is relatively large, even strong regularization of À 4.0 not adequate to fully suppress robust overfitting. Besides, comparing the situation of À >> 0 and = 0 for fixed d(2 1.5), we emphasize that the purposed BoAT loss again exhibits its apparent effectiveness in simultaneously boosting the best robustness and mitigating robust overfitting.
18

---

Under review as a conference paper at ICLR 2023
Table 6: Changing decay factor d and regularization strength À and evaluating model robustness on CIFAR-10 10 under the perturbation norm Eoo = 8/255 based on the PreActResNet-18 architecture.
Natural final
PGD-20
AutoAttack
Method
best
diff
best
final
diff
best
final
diff
BoAT(d = 1.3, = 0.0) 81.17
81.27
-0.10
56.43
56.23
0.20
50.80
50.75 0.05
BoAT(d = 1.3, À 1.0)
80.67
80.63
0.04
56.27
56.15
0.12
50.74
50.65 0.09
(o 1.3, À 4.0)
78.50
78.36
0.14
55.10
55.04
0.06
50.31
50.30 0.01
Bo 1.5,
0.0)
82.04
82.57
-0.73
56.33
56.02
0.31
50.93
50.69 0.24
BoAT(d = 1.5, À = 1.0) 82.20
82.06
0.14
56.59
56.55
0.04
51.39
51.36 0.03
Bo = 1.5, À 4.0) 79.68
79.88
-0.20
55.72
55.65
0.07
50.52
50.50 0.02
(d 4.0, À 0.0) 83.05 85.38
-2.33
55.98
50.87 5.11
50.38
46.95 3.43
(oAT(d = 4.0,
1.0) 82.46
84.84
-2.38
55.86
52.02 3.84
50.87
47.88 2.99
BoAT(d = 4.0, À = 4.0) 80.99 84.07 -3.08
56.06
53.58 2.48
51.00
49.02 1.98
C.2 IMPROVING NATURAL ACCURACY WITH KNOWLEDGE DISTILLATION
Chen et al. 22021) propose adopt knowledge distillation (KD) (Hinton et al. 2015 to mitigate RO and it worth mentioning that their method achieves relatively good natural accuracy according to Table and Since our BoAT method is orthogonal to KD, we propose to combine our techniques simplify their method by using only non-robust standard classifier as teacher (ST teacher) instead of using both ST teacher and a AT teacher, because i) large sum of computational cost for training the AT teacher will be saved, ii) our main goal is improve natural accuracy the ST teacher matters more and iii) BoAT already use the WA model as a very good teacher. This gives the final loss function as
with KD to further improve natural accuracy. Specifically,
(BoNT+KD v; =(1 AST) BBAAT (2. 0) + AST - KL (fo(t))(fsr())
(5)
where fsT indicates the ST teacher and AST is a trade-off parameter.
Table 7: Combining our methods with knowledge distillation and evaluating model robustness on CIFAR-10 under the perturbation norm = = 8/255 based on the PreActRReNNet118 architecture.
Natural best final
PGD-20
AutoAttack final diff
Method
diff
best
final
diff
best
BoAT(AST 0.0)
82.20 82.06 0.14
56.59
56.55 0.04
51.39 51.36 0.03 51.11 0.02
BoAT+KD(AAT 0.4)
83.88 83.74 0.14
55.69
55.47 0.22
51.13
BoAT+KD(AsT = 0.6) 84.66 84.54 0.12
54.44 54.38 0.06 50.54 50.65 -0.11
Table 7 compares the performance of BoAT+KD when different ÄST is applied, and clearly large ÀST results in improvement in natural accuracy and decreases robustness (may due to the theoretically principled trade-off between natural accuracy and robustness (Zhang et al. (019)). However, it is still noteworthy that when ÄST = 0.4, notable increase in natural accuracy (+> 1.70%) is achieved the cost of only a small slide of ~ 0.25% in robustness against AutoAttack. Moreover, we emphasize that RO is almost completely eliminated regardless of the trade-off, which is the main concern of this paper and demonstrates the superiority of our method against previous ones.
C.3 CHANGING LEARNING RATE DECAY SCHEDULE
In the previous experiments we only investigate the piecewise L.R decay schedule. However, a natural idea would be using mild LR decay schedules, e.g., Cosine and Linear decay schedule, instead of suddenly decaying it by a factor of d at some epoch in the piecewise decay schedule. Therefore, experiment with Cosine and Linear decay schedule and summarise the results in Table To be specific, the LR still decays to 0.01 epoch 150 and 0.001 epoch 200, though the two decay stages (from epoch 100 to 150 and 150 to 200) are designed to be gradual following the Cosine/Linear schedule. We also gradually increase the strength of BoAT regularization from zero the L.R
19

---

Under review as a conference paper at ICLR 2023
gradually decreases, following the "larger decay factor goes with stronger BoAT regularization" principle that we introduced in Appendix C.I
Table 8: Comparing our method with WA on CIFAR-1O under the perturbation norm e 8/255 based on PreActResNet-18 architecture, when Cosine/Linear LR decay schedule are applied.
Cosine
Linear
Method
Natural
PGD-20 AutoAttack
Natural
PGD-20
AutoAttack best final
best
final
best
final
best
final
best
final
best
final
WA(d=10)
82.22 85.17 56.22 47.96 50.79 50.79 43.87 82.28 85.27
56.37 48.17 17 50.83 44.37
BoAT(d=10)
82.16 82.17
56.24 54.38 51.01 49.76 82.03 82.03
56.53 54.67 51.17 49.63
can be concluded from Table X that simply changing the LR decay schedule indeed improves the best robust accuracy from 49.92% to 50.79% and 50.83% against AutoAttack respectively, but provides no help all in mitigating RO as the last robust accuracy is still around 44% against AutoAttack. We also note that in this situation, the application of BoAT loss not only significantly mitigates RO but also further improves the best model robustness, which also proves its effectiveness.
C.4 USING VALIDATION SET
Following Wu et al. (2020): : Dong al. (202) and Yu et (202)) we report the test accuracy on the best checkpoint that achieves the highest robust test accuracy. Another way to evaluate model robustness is selecting the model that achieves the highest robustness validation set as the best checkpoint. Therefore, following Rice et al. (2020), we hold out 1,000 examples from the CIFAR-10 training set for validation purposes and rerun the experiments using PGD-AT and BoAT.
Table 9: Selecting models of best robustness base on their performance on validation set and evaluating robustness on CIFAR-10 10 under the perturbation norm & 8/255 based on the PreActResNet-188 architecture.
Natural
PGD-20
AutoAttack final diff"
Best Model Selection
Method
best
final
diff
best
final
diff'
best
PGD-AT 81.61 84.67 3.06
52.51
46.20 6.31
47.54
42.31 5.23 51.13 51.22 -0.09
Using Validation Set
BoAT
81.86 81.91 -0.05
56.36
56.12 0.24
PGD-AT
82.08 83.98
-1.90
52.64
47.06 5.58
47.72
42.60 5.12
Highest on Test Data
BoAT 82.20 82.06 0.14
56.59
56.55 0.04
51.39 51.36 0.03
From Table g we can see that the selection criterion seems to make little difference. In our BoAT, changing test to validation only causes very slight drop in natural (0.1%) and AA (0.2%) accu- racy. Therefore, our validation-based best-epoch robustness (51.13% against AutoAttack) is still significantly better than all the other best-based methods Table
C.5 RESULTS ON MORE NETWORK ARCHITECTURES
In previous experiments we compare methods based on PreAAtRReNNt-118 and WedeResseet33--10 architecture, and here we also adopt VGG-16 (Simonyan & Zisserman, 2015) and MobileNetV2 (Sandler al. 2018) architecture. The significant improvement against baseline PGD-AT in both best and final robust accuracy and in mitigating RO reported in Table further demonstrates that method is applicable to wide range of network architectures.
C.6 TRAINING EFFICIENCY
We also test and report the training time (per epoch) several methods evaluated in this paper. For fair comparison, all the compared methods are integrated into a universal training framework and each test runs single NVIDIA GeForce RTX 3090 GPU.
From Table TT we can see that BoAT requires nearly no extra computational cost compared with vanilla PGD-AT (136.2s v.s. 131.6s per epoch), implying that it is an efficient training method in practical. We also remark that KD+SWA, one of the most competitive methods that aims address
20

---

Under review as a conference paper at ICLR 2023
Table 10: Comparing method with PGD-AT on CIFAR-10 under the perturbation &oo 8/255 based on VGG- 16 and MobileNetV2 architecture.
VGG-16
MobileNetV2 PGD-20
Method
Natural
PGD-20
AutoAtiack
Natural
AutoAttack
best
final
best
final
best
final
best
final
best
final
best
final
PGD-AT 78.75 81.95 50.76
44.58
44.56
39.96
80.12 81.03
46.01 45.27 53.18 52.57 47.66 47.35
51.56 50.67
BoAT
78.37 78.65
53.66 53.41
47.65 47.55
78.98 80.81
Table 1I: Combining training time per epoch CIFAR-10 under the perturbation norm Eoc 8/255 based on the PreActResNet-I1 architecture.
Method
Training Time per Epoch (s)
PGD-AT
131.6
WA
132.1
KD+SWA
131.6+16.88 15.5411.
AWP
142.8
MLCAT
353.3
BoAT
136.2
WA+CutMix
168.6
BoAT+CutMix
173.1
the RO issue, is really computationally efficient requires pretrain robust classifier and non-robust one as AT teacher and ST teacher respectively.
C.7 COMPARISON WITH MLCAT
Recently, eu al.
2022
propose their understandings in robust overfitting and propose method Here we compare our work with theirs in details.
MLCAT to alleviate RO.
2022 analyze robust overfitting by investigating the roles of easy and hard samples and propose to regulanze samples of small loss values (typically using non-robust features). Nevertheless, their discussions cannot fully explain how robust overfitting rises from easy samples. In comparison, provide generic explanation for the rise of robust overfitting from feature dynamics perspective, and we verify this understanding through extensive experiments. Our proposed BoAT is also designed from a bootstrapping perspective, which different from theirs. At last, understandings can also help understand how Yu et al. (2022)'s 's method works: by regularizing small-loss data, they enhance robust features and diminish non-robust features, which helps alleviate robust overfitting.
Methodologies, Yu et al.
Table 12: Comparing method with MLCAT IEFAR-10/CIFAR.CO under the perturbation norm Eoc = 8/2 8/255 based on the PreActResNet-18 architecture. The reported robust accuracy is evaluated with Auto/ttack.
CIFAR-10
CIFAR-100 final diff
Method
best
final
diff
best
MLCATTs
28.12 26.93 1.19
13.41
11.35 2.06
MLCATwP
50.70
50.32
0.38
25.66
25.28 0.38
BoAT
51.39 51.36
0.03
27.59
27.52 0.02
Performance and Efficiency. Table 12 shows that 1) our BoAT method is consistently superior (t 1 %% robust improvements against AutoAttack) to MLCAT across different architectures and different backbone networks, and 2) BoAT has weaker robust overfitting than MI.AT. Table [T] further shows that 3) BoAT is much more training efficient by requiring only 1/3 training time of MLCAT. Therefore, BoAT is both more effective, more efficient, and has less overfitting than MLCAA:
21

---

Under review as a conference paper at ICLR 2023
D VISUALIZING ADVERSARIAL EXAMPLES
In Figure 14 We visualize some adversarial examples generated in the experiments (following the settings introduced in Appendix B.3 in Section
and 2.3
"Cat"
"Deer"
"Dog"
"Frog"
"Horse"
original
add adverssuria
Figure 14: Adversarial examples generated the experiments in Section
and 2.3
22


Reviews:
[{"review_id": "afePIJ5UAr", "reviewer": "Reviewer_vLyN", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the robust overfitting (RO) problem for adversarial training (AT) which usually happens after the learning rate (LR) decay. In particular, it hypothesizes that RO is caused by a loss of balance (trainer becomes stronger than attacker) after LR decay. It verifies the hypothesis by looking into target-class information in test robust features, bilateral class correlation, and confusion matrix symmetry for models with and without LR decay. Based on this understanding, the authors propose three techniques (model flatness regularization, smaller LR decay and stronger attacker during training) to rebalance the trainer and the attacker. The proposed method BoAT is evaluated against several baselines on two model architectures and three datasets.", "summary_of_the_review": "Overall, the paper provides neat observations on the RO problem in AT and verifies the hypotheses. Its proposed three techniques mitigating RO seems to be functional for alleviating RO but might not be useful as general techniques to improve robustness.\n", "strength_and_weaknesses": "## Pros\n\n-neat observations and formulation of the RO problem\n\n-the experiments mostly verify the proposed hypotheses\n\n## Cons\n\n-BoAT has much lower natural accuracy compared with KD+SWA\n\nThe results in Table1 seem to suggest BoAT simply trades off natural accuracy and adversarial accuracy compared with existing methods (e.g., KD+SWA) so it might not be very useful in the sense of improving the natural-adversarial accuracy trade-off. I would like to see how the trade-off of natural accuracy and adversarial accuracy look like for different hyper-parameters of BoAT and baselines, and which one lies on the Pareto front.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "VPn3-SnFZNQ", "reviewer": "Reviewer_yb7Z", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work aims to provide a new perspective to explain the robust overfitting and improve the adversarial robustness built upon the proposed viewpoint. To do so, the authors revisit and generalize the static framework of feature robustness. Built upon the proposed framework, the authors claim that the balance between the attacker and the trainer is the key to understanding robust overfitting. Consequently, the authors propose three methods inspired by the framework and demonstrate the effectiveness of these (two, as one is dropped in the main page) methods.", "summary_of_the_review": "The proposed framework is novel to me, the proposed methods are simple yet effective, and the conducted experiments are convincing.", "strength_and_weaknesses": "Strength:\n- The considered symmetrized confusion matrix is interesting and convincing, which is novel to me. \n- The designed objective function is inspiring. Specifically, the authors introduce a KL term, minimizing the KL divergence between the learnable model f_{\\theta} and the detached model f_{\\phi}, which is actually equal to assigning a soft label to each adversarial example. Thus, it is promising and exciting to explore a novel approach to designing different assignments of soft labels. And, maybe, it has potential benefit to the research of certifiable defense.\n- The paper is well-written, easy to follow, and has solid experimental analysis.\n- The proposed method is simple yet effective, verified by the experiments.\n\nWeakness:\n- A stronger attack for stronger defense is not employed due to its harmful impacts on natural accuracy, but I cannot find any explanations about the observation from the proposed perspective. It is necessary to add the corresponding explanation using the proposed view because methods inspired by the proposed framework are dropped from the main page, making the framework less convincing.\n- The definition of Eq. (3) seems confusing. The trainer T is directly applied to update the model, i.e., changing the learned representations, thus, introducing the trainer T, given the representation, seems meaningless, but the trainer T is important for the rest of the work. Hence, the authors should clarify the point. Similarly, the utilized F_f^T is also confusing.\n\nSuggestions:\n1 The claimed theory seems to be a hypothesis, as merely experimental analysis is given in the current version.\n\n2 According to the claimed perspective -- the main goal is to prevent the model trainer T from fitting non-robust features too quickly and too adequately --, a straightforward approach can be that the trainer decreases the learning rate progressively from the previous epoch decaying the learning rate by 0.1. Thus, the corresponding experiments are required. However, it is lacking in the current version. \n\n3 Different from the section on the method, the section on explanation seems weak, I suggest employing more references to convince the claim, the related perspective is the spurious feature perspective, such as CausalADV and group DRO.\n\n4 The ‘feature’ considered in Eq. (2) corresponds to the data space, while the ‘feature’ considered in this work Eq. (3) is more about the representation space, thus, I suggest the authors make these two features distinguishable as they are conceptually different.\n\n[CausalADV] Adversarial robustness through the lens of causality. Zhang et al. 2021\n\n[group DRO] Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. Sagawa et al., 2020\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NyEgbZqpTk_", "reviewer": "Reviewer_kLGk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Summary\n\nBased on former findings, this paper first proposes a new definition of robust features with taking dynamics into consideration. Focused on the phenomenon that robust overfitting usually starts after learning rate decay, the authors provide an explanation of robust overfitting based on several experiments. New approaches to mitigate robust overfitting are proposed and validated. The main difference from previous studies is their emphasis on the necessity of considering the min-max nature of AT in robust overfitting. ", "summary_of_the_review": "Good paper with novelty and experiment limitations.", "strength_and_weaknesses": "\nPros\n\n1. The background and motivation of the paper are well-written, and the validation experiments of the proposed hypothesis are convincing.\n2. The paper is well-written, clear, and easy to follow.\n\n\n\nCons.\n\n1. The proposed dynamic feature robustness framework needs to be more formally and clearly defined, especially the difference from the former work. For instance, it is encouraged to provide a formal mathematical definition of dynamic, robust, and non-robust features.\n2. The mitigation approaches ”adding regularization loss”, “using stronger attacker” and “decreasing learning rate” are well-established tricks and therefore this paper does not bring technical novelty.\n3. The proposed methods are only evaluated on 2 architectures in the same model family, which is highly insufficient. More network backbones like MobileNet and VGG should be included.\n4. Although the proposed methods lead to improved robustness, it suffers from natural accuracy degradations. It is hard to say whether the BoAT is better compared to other approaches. Also, this needs to be mentioned in the front of this paper, otherwise only mentioning the robustness benefits is misleading.\n5. The paper mentions that non-robust features will be much more robust after the learning rate decay. I am very curious about whether the non-robust features will also approach robust features visually.\n6. \"RO usually happens after the learning rate decay in AT\". Actually, this claim is insufficiently supported. Since most previous papers decay the learning rate at the 100th epoch. In order to well support this claim, the authors need to conduct experiments with LR decay at different epochs and show the consistency of RO observations.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7_PWcFwqgk", "reviewer": "Reviewer_zGeX", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper cites the feature robustness framework of Ilyas et al. 2019 as its starting point. It claims that framework fails to explain overfitting in adversarial training (Note: that paper wasn't claiming to explain overfitting). The paper then proposes a narrower definition of feature robustness (where an attacker and a trainer are given). They claim that it explains overfitting.  \n\nThe paper, I think, mischaracterizes Ilyas et al. 2019 definition of feature robustness when describing Limitations. \n\nThe writing is not particularly clear on why the change in definition explains overfitting, which was the original motivation for the paper. The paper acknowledges past work that robust overfitting can be largely \"surpassed\" (suppressed?) by enforced a smoother landscape with additional regularizations. In the end, it seems the proposed schemed BoAT also ends up doing something similar (Section 3.1: \"In order to suppress robust overfitting of the model trainer T after LR decays, we need to put further regularization on its local fitting ability, e.g., by enforcing better landscape flatness.\").\n\nSo, I am still trying to figure out the key contribution of the paper and new teachings. It could be that the theoretical arguments need improvement to explain overfitting.  \n\nThe BoAT training method is a potential contribution since the claim is that it eliminates overfitting even over 500 epochs. But, the authors should compare that with other approaches such as simply picking an earlier saved model (prior to overfitting) and prior techniques that introduce additional regularizations to smooth the landscape (many of which they cite in the 2nd para of the Intro). \n\nThe other issue is efficiency of BoAT training. How long does the training take compared to existing techniques, e.g.,  early stop? I am trying to figure if BoAT is proposed as a practical technique that practitioners should use or is simply a tool to help explain overfitting. \n\nThe Intro states: \n\n\"Specifically, we hypothesize that due to the strong local fitting ability endowed by smaller LR, the model learns false mapping of non-robust features contained in adversarial examples, which opens shortcuts for the test-time attacker and induces large test robust error. \"\n\nBut, why doesn't standard training also learn false mappings on non-robust features and also cause overfitting? \n\nIn the end, the paper started with an interesting promise of explaining the difference in overfitting between standard training and adversarial training. But, that difference in behavior still remains unclear after reading the paper.\n", "summary_of_the_review": "The paper needs to be better written with a  more careful discussion of related work to better position the paper. The paper is also missing some related work that also attempts to explain overfitting and propose techniques to prevent overfitting.  \n\nAdditional comments for the authors after the rebuttal: \n\nI thank the reviewers for detailed explanations. I think the algorithm is a good contribution. But, I still feel that the overall pitch in the Intro and Section 2 should be changed in a future version of the paper. I am not sure Ilyas et al. is the correct baseline to criticize, since it was not directed at providing an explanation on robust overfitting. Other papers that attempt to explain that in other ways would be a better starting point.  In case you use Ilyas et al., perhaps a better pitch would be to say that even if a feature is fundamentally robust,  it can be perceived as non-robust by a given model because of inability to find the correct decision boundary for that feature -- even if one exists.  That way, perhaps Ilyas et al. are not necessarily wrong -- static and dynamic views become more in harmony. I don't think the paper proves that their view was necessarily wrong, but the paper suggests so, which I think is an issue with the current thrust. \n\nAnother area that the paper could  do a better job in  would be to better explain why normal training does not overfit, but adversarial does. Are there situations where normal training would also overfit, based on the understanding provided by the work? Even normal data should have some non-robust features, no -- or is that fundamentally missing in normal datasets?  More in-depth discussion would be useful. \n\nOverall, I am not changing my evaluation, but  I encourage authors to rethink the overall pitch for presenting the Boat algorithm and add more depth for an improved paper in the future.\n\n\n\n\n\n", "strength_and_weaknesses": "\nStrengths: \n\nI am still trying to figure that out. I wish the paper was clearer in writing. It seems the authors are trying to draw a connection between feature robustness (using their definition, not Ilyas et al.), learning rate decay,  and overfitting. And why the authors think that standard training doesn't run into overfitting but adversarial training does at a fundamental level.\n\nIn the end, it is not clear if BoAT is a better algorithm over prior methods for adversarial training.\n\nWeaknesses: \n\nEnglish writing is poor, making the paper difficult to follow. The abstract could be better written to begin with. \n\nLimitations on page 3: I didn't understand your argument. I didn't find in Ilyas a discussion or claim regarding overfitting.  Furthermore,  during model training, isn't the classifier changing? And also isn't f changing, if if is part of the model? If it is some abstract function f independent of the trained model, then why you do believe that robustness of f is changing during model training?\n\nFurthermore, I don't see an attacker A in definition (2). \n\nI don't agree with the characterization of  Ilyas et al. 2019 that they only consider attacker A and fail to view adversarial training as a dynamic min-max game. For instance, in Robust Training para on page 4 of Ilyas et al., they explicitly discuss adversarial training a min-max optimization problem.  \n\nFurthermore, the definition I Alias et al. considers the adversarial space, but does not explicitly mention an attacker strategy. I am still trying to understand your criticism of Ilyas et al. 2019.\n\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3sIdz6UVMl", "reviewer": "Reviewer_p7yt", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper begins with an argument that the robustness of a model should also be conditioned on the training process. They then focus on  the one specific aspect of the training process -- learning rate (LR) decays. They found that after LR decays, the confusion matrix becomes more symmetric. The author claimed that this observation helps to understand robust overfitting. Then, they proposed the BoAT loss, which seems to perform better than some prior work in their experiment.", "summary_of_the_review": "This paper is not well written. The flow of the paper is also not logical (e.g. the relationship between the Symmetrized Confusion Matrix and robust overfitting is not explained). There are major flaws in the experiment setup (e.g. selecting the best epoch based on the performance on test data), and the experimental result cannot support all their claims.", "strength_and_weaknesses": "Weakness:\n- The first paragraph in page 5 is confusing.\n  - In Figure 2, there are no measurement on feature, only on classes. But the conclusion is on target-class features. What is the definition of non-robust feature?\n  - The relationship between the confusion matrix being symmetric and robust overfitting is unclear.\n  - What does A->B and B->A mean in Figure 2?\n  - What does accuracy with misclassified labels in Figure 2a mean?\n  - Is the confusion matrix computed with adversarial attack?\n- Some other parts of the paper that are confusing:\n  - Should the output of f be a feature or a label? In the text, it's called feature while in equation 2 and 3, they are compared with label y.\n  - The authors claimed \"False Non-robust Mapping Opens Shortcuts for Test-time Adversarial Attack\". What is a shortcut? Could you measure this \"shortcut\"? Are there any hypothesis on why this shortcut is related to LR decay?\n- What is the learning rate schedule used in Figure 2? Does different scheduling or decay method matters?\n- The method used in [1] should also be compared. Their method is effectively make the model locally smooth, which in some way also improves landscape flatness.\n- For Equation (4), is the adversarial example computed on the cross-entropy loss or on the BoAT loss? The inner minimization should be included in the equation to make things more clear.\n- How is the lambda in Equation (4) chosen?\n- This paper discussed a lot on robust overfitting, while in section 3 and 4, there is not discussion on the generalization gap comparing with other methods.\n- The hyper parameter choices in Table 4 seems arbitrary.\n- I think it is arguable whether adversarial training is SOTA algorithm for adversarial robust training. I would suggest checking out some of the methods in https://robustbench.github.io/\n- In the paragraph before Section 4.1: \"We train each model for 200 epochs and record the best-epoch (selected based on PGD attack)and final-epoch robust test accuracy.\". One should not just select the best epoch on test data. This should be done on a validation set.\n\n[1] Zhang, Hongyang, et al. \"Theoretically principled trade-off between robustness and accuracy.\" International conference on machine learning. PMLR, 2019.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject