SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yifei Wang $^{1*}$
yifei_wang@pku.edu.cn
Liangchen  $\mathbf{Li}^{2*}$
11673@cam.ac.uk
Jiansheng Yang
yjs@math.pku.edu.cn
Zhouchen Lin3,4,5
zlin@pku.edu.cn
Yisen Wang $^{3,4\dagger}$
yisen.wang@pku.edu.cn
<sup>1</sup> School of Mathematical Sciences, Peking University
$^{2}$  Department of Engineering, University of Cambridge
3 National Key Lab of General Artificial Intelligence,
$^{4}$  Institute for Artificial Intelligence, Peking University
5 Peng Cheng Laboratory
Adversarial Training (AT) has become arguably the state-of-the-art algorithm for extracting robust features. However, researchers recently notice that AT suffers from severe robust overfitting problems, particularly after learning rate (LR) decay. In this paper, we explain this phenomenon by viewing adversarial training as a dynamic minimax game between the model trainer and the attacker. Specifically, we analyze how LR decay breaks the balance between the minimax game by empowering the trainer with a stronger memorization ability, and show such imbalance induces robust overfitting as a result of memorizing non-robust features. We validate this understanding with extensive experiments, and provide a holistic view of robust overfitting from the dynamics of both the two game players. This understanding further inspires us to alleviate robust overfitting by rebalancing the two players by either regularizing the trainer's capacity or improving the attack strength. Experiments show that the proposed ReBalanced Adversarial Training (ReBAT) can attain good robustness and does not suffer from robust overfitting even after very long training. Code is available at https://github.com/PKU-ML/ReBAT.
Overfitting seems to have become a history in the deep learning era. Contrary to the traditional belief in statistical learning theory that large hypothesis class will lead to overfitting, Zhang et al. [58] note that DNNs have good generalization ability on test data even if they are capable of memorizing random training labels. Nowadays, large-scale training often does not require early stopping, and it is observed that longer training simply brings better generalization [19]. However, researchers recently notice that in Adversarial Training (AT), overfitting is still a severe issue on both small and large scale data and models [38]. As the most effective defence against adversarial perturbation [2], AT solves a minimax optimization problem with training data  $\mathcal{D}_{\mathrm{train}}$  and model  $f_{\theta}$  [15, 30, 49]:
$$
\min  _ {\theta} \mathbb {E} _ {\bar {x}, y \sim \mathcal {D} _ {\text {t r a i n}}} \max  _ {x \in \mathcal {E} _ {p} (\bar {x})} \ell_ {\mathrm {C E}} \left(f _ {\theta} (x), y\right), \tag {1}
$$
where  $\ell_{\mathrm{CE}}$  denotes the cross entropy (CE) loss function,  $\mathcal{E}_p(\bar{x}) = \{x\mid \| x - \bar{x}\| _p\leq \varepsilon \}$  denotes the  $\ell_p$ -norm ball with radius  $\varepsilon$ . However, contrary to standard training (ST), AT suffers severely from
†Corresponding Authors: Yisen Wang (yisen.wang@pku.edu.cn) and Zhouchen Lin (zlin@pku.edu.cn)
(a) training robustness during adversarial training
(b) test robustness during adversarial training
Figure 1: (a, b) Training/test robust accuracy on CIFAR-10 using vanilla PGD-AT [30] w/ and w/o LR decay. (c) RO happens whenever LR decays.
(c) test robustness when decay at different time
robust overfitting (RO): after a particular point (e.g., learning rate decay), its training robustness will keep increasing (Figure 1a, red line) while its test robustness decreases dramatically (Figure 1b, red line). This abnormal behavior of AT has attracted much interest. Previous works find that the AT loss landscape becomes much sharper during RO [42, 8, 53], but we are still unclear about what causes the sharp landscape. Some researchers try to explain RO through known ST phenomena, such as, random label memorization [12] and double descent [11]. However, these theories cannot explain why only AT overfits while ST hardly does.
Instead, we argue that the reason of RO should lie exactly in the difference between AT and ST. Sharing similar training recipes and loss functions, their key difference is that AT adopts a bilevel minimax objective, compared to the min-only ST objective. We can think of AT as a minimax game between two players (Eq. 1): the model trainer  $\mathcal{T}$  (minimizing loss via updating  $\theta$ ) and the attacker  $\mathcal{A}$  (maximizing loss via updating  $x$ ), both adopting first-order algorithms in practice. With a proper configuration of both players, AT can strike a balance during which model robustness hinges on a constant level. In comparison, we also notice that when one player (typically the trainer) changes its original strategy (e.g., decaying learning rate), the original balance of the game is broken and RO also occurs subsequently. We further verify this observation through a controlled experiment. As shown in Figure 1c, if we do not perform LR decay, RO will not occur till the end; but whenever we perform LR decay, RO will immediately happen. This provides strong evidence showing that breaking the balance between minimax layers can cause robust overfitting.
To gain further insights into this balance-imbalance process, in Section 3, we provide a generic explanation for RO from the perspective of feature learning. Specifically, when the balance breaks after the LR decay, the trainer with a smaller LR is endowed with a stronger local fitting ability capable of memorizing adversarial examples such that the attacker can no longer succeed in attack. However, during this process, the model learns false mapping of non-robust features contained in the training adversarial examples, which opens shortcuts for the test-time attacker and induces large test robust error. Accordingly, we design a series of experiments that align well with our analysis.
The minimax game perspective also indicates a simple fix to RO, that is to adjust the ability of the two minimax players to a new balance by modifying their strategies, namely, rebalancing. In this work, we explore three new strategies for rebalancing the game after LR decay: bootstrapping objective, smaller LR decay factor, and stronger attacker. The former two restrict the local fitting ability of the trainer while the third enhances the ability of the attacker. We show that all the three strategies help mitigate robust overfitting to a large extent. Based on these insights, we propose ReBalanced Adversarial Training (ReBAT) that achieve superior robustness with a negligible (if any) degree of robust overfitting, even if we train it further to 500 epochs. Meanwhile, ReBAT attains superior robustness (both best-epoch and final-epoch) on benchmark datasets including CIFAR-10, CIFAR-100, and Tiny/ImageNet. This suggests that with appropriately balanced training strategies, AT could also enjoy similar "train longer, generalize better" property like ST. To summarize, our main contributions are:
- We propose to understand adversarial training (AT) as a dynamic minimax game between two actual players: the model trainer  $\mathcal{T}$  and the attacker  $\mathcal{A}$ . We characterize the balance status of the minimax game from the perspective of non-robust features.
- We show robust overfitting is caused by the imbalance between two minimax players induced by the LR decay, and explain this process as the false memorization of non-robust
features. We design a series of experiments to verify this explanation and provide a holistic characterization of robust overfitting from the views of both players.
- From our dynamic perspective, we propose three effective approaches that mitigate robust overfitting by re-striking a balance between the model trainer and the attacker. Experiments show that our proposed ReBAT method largely alleviates robust overfitting while attaining good robustness on different network architectures and benchmark datasets.
Adversarial Attack. Given an image classifier  $f$ , adversarial attack [43] is proposed to perturb the image by an imperceptible noise such that the image is misclassified by  $f$ . A well-known white-box attacker  $\mathcal{A}$  is PGD (Projected Gradient Descent) [30], which uses multi-step gradient ascent steps to maximize the cross entropy loss while projecting intermediate steps to the feasible region:
$$
\tilde {x} _ {k + 1} = x _ {k} + \alpha \nabla_ {x _ {k}} \ell_ {\mathrm {C E}} \left(f _ {\theta} \left(x _ {k}\right), y\right), x _ {k + 1} = \mathcal {P} _ {\mathcal {E} _ {p} (\bar {x})} \left(\tilde {x} _ {k + 1}\right), k = 0, \dots , K - 1. \tag {2}
$$
where  $x_0 = \bar{x} +\varepsilon ,\varepsilon \sim U[0,\varepsilon ]$  , and  $\mathcal{E}_p(\bar{x}) = \{x|x\in [0,1]^d,\| x - \bar{x}\| \leq \varepsilon \}$  is the feasible region.
Adversarial Training. To counter misclassification caused by the attacker, adversarial training (AT) [43, 15, 30, 38, 50, 3] aims to build a robust classifier through the minimax objective in Equation 1. In practice, AT iterates between two steps: 1) solve the inner-loop maximization w.r.t. input  $x$  using a few (e.g., 10) PGD steps (Eq. 2) to generate a minibatch of adversarial examples  $\{\hat{x}\}$ , and then 2) solve the outer-loop minimization w.r.t. NN parameters  $\theta$  with SGD as the trainer  $\mathcal{T}$ . Therefore, AT can be seen as a minimax game between two players (specifically, two first-order optimization algorithms), the attacker  $\mathcal{A}$  and the trainer  $\mathcal{T}$ , with the objective function  $\ell_{\mathrm{CE}}(f_{\theta}(\hat{x}),y)$ .
Robust and Non-robust Features. The arguably most prevailing understanding of adversarial training is the framework proposed by Ilyas et al. [21], where they regard training data as a composition of robust and non-robust features that are both useful for classification. Consider a binary classification task, a feature  $f: \mathcal{X} \to \mathbb{R}$  is categorized as 1) a useful feature if  $\mathbb{E}_{\bar{x},y}[y \cdot f(x)] \geq \rho$ ; 2) a robust feature if it is useful for  $\rho > 0$  and  $\mathbb{E}_{\bar{x},y}\inf_{x \in \mathcal{E}_p(\bar{x})}[y \cdot f(x)] \geq \gamma$ ; 3) a non-robust feature if it is useful for  $\rho > 0$  and not robust for any  $\gamma \geq 0$ . We can also choose a threshold  $\gamma$  to determine its belonging class since there is no clear dividing line between robust and non-robust features in practice. According to Ilyas et al. [21], standard training (ST) adopts non-robust features for better accuracy, which also leads to adversarial vulnerability. Instead, AT only uses robust features and becomes resistant to adversarial attack. Our work also adopts the notion of non-robust features. But compared with their framework which mainly considers the ideal, static solution to the minimax problem (a learned robust classifier), our work zooms into the dynamic interplay of the two actual players (how a classifier learns the robust & non-robust features in AT) to understand RO.
3 A Minimax Game Perspective on Robust Overfitting
In this section, we propose a minimax game perspective to understand the cause of robust overfitting. We first show the minimax game in Equation 1 achieves a balance when the trainer  $\mathcal{T}$  is not capable of memorizing the non-robust features, and then demonstrate through several verification experiments that the break of this balance, which leads to memorization of the non-robust features, actually induces shortcuts for test-time adversarial attack that results in RO.
3.1 AT Strikes a Balance when Non-robust Features Cannot be Memorized
First, we look at the balanced status of adversarial training. From Figure 1a (blue line), we observe that without LR decay, the two players can maintain a constant robustness level throughout training. Inside the equilibrium, the attacker  $\mathcal{A}$  injects non-robust features from a different class  $y^\prime$  to  $x$  in order to misclassify an input  $(x,y)$  [21]3, while the trainer can only correctly classify a part of them using robust features. In other words, the game can strike a balance when there is a limit of the trainer's fitting ability that after a certain robustness level, it can no longer improve robustness by fitting the non-robust features that the attacker can modify (i.e., non-robust features are not memorizable).
To verify this point, we collect misclassified adversarial examples (containing non-robust features w.r.t.  $y'$  if misclassified to  $y'$ ) on an early checkpoint (60th epoch) and evaluate their accuracy on
3We refer the reader to Appendix B for simple a verification and to Fowl et al. [14] for a further discussion.
(a) non-robust feature memorization
(b) influence of stronger non-robust features (larger  $\varepsilon$ ) on test robustness
(c) target-class non-robust features
Figure 2: (a) Almost non-generalizable non-robust features in training data is memorized only after LR decay. (b) With LR decay, injecting stronger non-robust features (larger  $\varepsilon$ ) induces severer degradation in test robustness, while the degradation is less obvious w/o LR decay. (c) After LR decay, test adversarial examples contains less and less target-class non-robust features.
Robust overfitting happens!
Figure 3: An illustration example of the rise of robust overfitting. Here, the attack (from the clean example to the adversarial example) follows the direction to the nearest decision boundary.
a later checkpoint ( $\geq$ 150th epoch) w.r.t. their true labels  $y$  (detailed settings in Appendix C.2). If the adversarial non-robust features can be memorized, we would expect high accuracy on these adversarial examples on the later checkpoints. However, in practice, the blue solid line in Figure 2a shows the accuracy is still relatively low ( $< 25\%$ ), suggesting that these non-robust features are almost not memorized under the same training configuration without LR decay.
3.2 Balance  $\rightarrow$  Imbalance: Robust Overfitting as a Result of Imbalanced Minimax Game
Next, we look at how LR decay breaks the balance of the minimax game. As pointed out by the theoretical work [28], small LR favors easy-to-generalize and hard-to-fit patterns, which correspond to non-robust features in AT [21]. Thus, with a smaller LR, the trainer becomes more capable of memorizing these features by drawing more complex decision boundaries in a small local region. Indeed, we observe a significant increase in training robust accuracy in Figure 1a. Moreover, we can see from the red line in Figure 2a that most of the pre-LR-decay adversarial examples (with adversarial non-robust features) are correctly classified after LR decay, indicating stronger memorization of non-robust features. Therefore, we claim that  $LR$  decay induces an imbalance between the attacker  $\mathcal{A}$  and the trainer  $\mathcal{T}$ : the boosted local fitting ability of  $\mathcal{T}$  overpowers the attack ability of  $\mathcal{A}$  that it is capable of memorizing the adversarial non-robust features that  $\mathcal{A}$  can leverage.
Meanwhile, as the red dashed line in Figure 2a implies, it turns out that these memorized non-robust features in training data do not generalize to improve test robustness, resulting in a large robust generalization gap (see Figure 1). Next, we further reveal how such memorization harms test robustness by inducing robust overfitting.
3.2.1 How Imbalance Leads to Robust Overfitting
Under the imbalanced minimax game, the stronger trainer pushes up training robustness by memorizing the non-robust features in the training adversarial examples. However, the trainer's (overly) strong fitting ability actually learns false mappings of non-robust features which induce shortcuts for
test-time adversarial attack that lead to robustness degradation. Below, we explain how this happens during adversarial training, and an (over-simplified) illustration is shown in Figure 3.
1) Model Learns False Mapping of Non-robust Training Features after Decay. Consider a training adversarial example  $x'$  from class  $y$  that is misclassified to class  $y' \neq y$  before LR decay. According to [21], the non-robust feature  $g$  of  $x'$  belongs to class  $y'$ , i.e.,  $g \in \mathcal{F}_{y'}$ . After LR decay, the trainer  $\mathcal{T}$  becomes more capable of memorizing with a smaller LR and it successfully learns to map this adversarial example  $x'$  to its correct label  $y$ . Accordingly, as we argued above, the non-robust feature  $g \in \mathcal{F}_{y'}$  in  $x$  is also (falsely) mapped to class  $y$ , i.e.,  $\varphi(g) = y$ .
2) False Non-robust Mapping Opens Shortcuts for Test-time Adversarial Attack. Now we consider a clean test sample  $\hat{x}$  whose true label is  $y'$ . Before LR decay, in order to misclassify it, the attacker  $\mathcal{A}$  has to add non-robust features from a different class, say  $g' \in \mathcal{F}_{\hat{y}'}$ ,  $\hat{y}' \neq y'$ . However, after LR decay, because of the existence of false non-robust mapping  $\varphi$ , the attacker  $\mathcal{A}$  can simply add the non-robust feature from the same class, i.e.,  $g \in \mathcal{F}_{y'}$  such that  $\varphi(g) = y$ ,  $y \neq y'$  to misclassify  $\hat{x}$  to  $y$ . In other words, the injected non-robust features do not even have to come from a different class to misclassify  $\hat{x}$ , which are much easier for the attacker to find when starting from  $\hat{x}$  (compared with non-robust features  $g'$  from other classes). Therefore, the attacker will have a larger attack success rate with the help of these falsely memorized non-robust features, as they create easy-to-find shortcuts for test-time adversarial attack.
Remark. To be rigorous, we give a formal definition for these non-robust features. Denote the hypothesis class of features that the model can robustly fit on training data under configuration  $t$  as
$$
\mathcal {H} _ {t} = \left\{f _ {\theta}: \mathcal {X} \rightarrow \mathbb {R} \mid \mathbb {E} _ {\bar {x}, y \sim \mathcal {D} _ {t r a i n}} \inf  _ {x \in \mathcal {E} _ {p} (\bar {x})} [ y \cdot f _ {\theta} (x) ] > 0, \right. \tag {3}
$$
where  $f_{\theta}$  is robust on training data, and  $\theta \in \Theta$  is attainable under the configuration  $t$ .
We say a non-robust feature  $f$  is falsely memorized if  $f \notin \mathcal{H}_{\text{before}}$ ,  $f \in \mathcal{H}_{\text{after}}$ , where  $\mathcal{H}_{\text{before}}$ ,  $\mathcal{H}_{\text{after}}$  refer to the hypothesis classes before and after LR decay, respectively. In other words, the non-robust  $f$  is falsely memorized by the model (as a robust feature) under the after-LR-decay trainer. This mismatch could happen because the trainer only sees training examples and it can overfit them under an imbalanced minimax game. Since this feature is essentially a non-robust feature (in the population sense), this feature still behaves non-robust on test data and introduce shortcuts to test-time attack, as we elaborate above.
3.2.2 Verification
To further validate our explanation above, we design a series of experiments regarding the influence of memorization of the adversarial non-robust features after LR decay (Verification I & II). Guided by our explanation, we also observe some new intriguing phenomena of RO, which also help verify our minimax perspective (Verification III & IV). Please refer to Appendix C.2 for more details.
Verification I: More Non-robust Features, Worse Robustness. To study the effect of non-robust features, we deliberately add non-robust features of different strengths to the clean data (a larger  $\varepsilon$  indicates stronger non-robust features) and perform AT on these synthetic datasets. As shown in Figure 2b, we can see that 1) with LR decay, stronger non-robust features induce severer test robustness degradation, and 2) it is much less severe without LR decay. This directly shows that the memorization of non-robust features induced by LR decay indeed hurts test robustness a lot.
Verification II: Vanishing Target-class<sup>4</sup> Features in Test Adversarial Examples. In our example above, the added non-robust feature  $g$  comes from the same class  $y'$  as  $\hat{x}$ , so the resulting adversarial example  $\hat{x}'$  actually contains little features of class  $y$ , even if misclassified to  $y$ . Figure 2c shows that the test adversarial examples generated after LR decay indeed have less and less target-class features as RO becomes more and more severe, which proves the shortcuts indeed exist to make test-time adversarial attack easier.
Verification III: Bilateral Class Correlation. Our theory suggests that if RO happens, the training misclassification from class  $y \rightarrow y'$  (before decay) will induce an increase in test misclassification from class  $y' \rightarrow y$  (after decay), as the  $y \rightarrow y'$  false non-robust mappings create  $y' \rightarrow y$  shortcuts. We verify this in Figure 4a, in which we observe that this bilateral correlation indeed consistently increases after LR decay, while remaining very low (nearly no correlation) without decay.
4By target-class we mean the class to which an adversarial example is misclassified.
(a) bilateral class correlation
(b) confusion matrix symmetry
epoch 200
Figure 4: (a) Increasingly strong correlation between training-time  $y \rightarrow y'$  misclassification and test-time  $y' \rightarrow y$  misclassification changes (due to RO). (b) Test-time confusion matrix of adversarial examples becomes symmetric with LR decay.
(a) different LR decay factor
(b) different attack strength
Figure 5: (a) RO becomes increasingly severe as larger LR decay factor is adopted. (b) Varying the attack strength from extremely weak to extremely strong, the degree of RO first increases and then decreases after a turning point around  $\varepsilon = 10 / 255$ .
Verification IV: Symmetrized Confusion Matrix. As a result of the bilateral correlation, we can deduce a very interesting phenomenon that the  $y \to y'$  and  $y' \to y$  misclassification will have more similar error rates, which means that the confusion matrix of test robustness will become more symmetric. Indeed, as shown in Figure 4b, the confusion matrix (denoted as  $A$ ) becomes much more symmetric (smaller distance between  $A$  and  $A^{\top}$ ) after LR decay than without LR decay.
3.3 A Holistic View of Robust Overfitting from the Minimax Game Perspective
Based on our understanding of RO through the lens of the minimax game, we give a holistic view of RO, revealing the influence of both the trainer and the attacker.
Trainer. We first fix the attacker strength to be PGD-10 attack with  $\varepsilon = 8 / 255$  and change the LR decay factor  $d$  from 1 (w/o LR decay) to 20 at epoch 100, and we observe that stronger LR decay induces a monotonically increasing degree of RO (Figure 5a). This agrees well with our explanation that smaller LR induces a severer imbalance and can memorize more non-robust features that eventually harm test robustness.
Attacker. We then fix the training configuration and vary the attack strength from PGD-0 with  $\varepsilon = 0 / 255$  (equivalent to standard training (ST)) to PGD-25 with  $\varepsilon = 20 / 255$ . Different from previous beliefs that RO will become severer with stronger attack [54], we find an intriguing inverted U curve in Figure 5b: the degree of RO firstly rises with stronger attackers, peaks at around  $\varepsilon = 10 / 255$ , and decreases under even stronger attackers ( $\varepsilon > 10 / 255$ ). Our theory can naturally explain this phenomenon: 1) when using a weak attacker (small  $\varepsilon$ ), training robustness is very high and there are only a few non-robust features. Thus memorizing them only leads to slight or no RO; 2) a very strong attacker (e.g.,  $\varepsilon > 10 / 255$ ) can also counter the strong fitting ability of the post-decay trainer, which also prevents the memorization of non-robust features and alleviates RO.