Exploring Memorization in Adversarial Training
Authors: Yinpeng Dong
Ke Xu
Xiao Yang
Tianyu Pang
Zhijie Deng
Hang Su
Jun Zhu
Venue: ICLR 2022 Poster (2022)
Keywords: Adversarial examples
adversarial training
memorization
robust overfitting

Abstract: Deep learning models have a propensity for fitting the entire training set even with random labels, which requires memorization of every training sample. In this paper, we explore the memorization effect in adversarial training (AT) for promoting a deeper understanding of model capacity, convergence, generalization, and especially robust overfitting of the adversarially trained models. We first demonstrate that deep networks have sufficient capacity to memorize adversarial examples of training data with completely random labels, but not all AT algorithms can converge under the extreme circumstance. Our study of AT with random labels motivates further analyses on the convergence and generalization of AT. We find that some AT approaches suffer from a gradient instability issue and the recently suggested complexity measures cannot explain robust generalization by considering models trained on random labels. Furthermore, we identify a significant drawback of memorization in AT that it could result in robust overfitting. We then propose a new mitigation algorithm motivated by detailed memorization analyses. Extensive experiments on various datasets validate the effectiveness of the proposed method. 

Published as a conference paper at ICLR 2022
EXPLORING MEMORIZATION IN ADVERSARIAL TRAINING
Yinpeng Dong Ke Xu", Xiao Yang', Tianyu Pang's Zhijie Deng' Hang Su Jun Zhu 1.23. 1 Dept. of Comp. Sci. and Tech., Institute for AI, Tsinghua-Bose Bosch Joint MI. Center, THBI Lab BNRist Center, Tsinghua University, Beijing, China; RealAI; Peng Cheng Laboratory; 4 CMU
(dongy--peng, suhanges, doaz)) @mail tainggua. www.gh.......de.ooooooooooom.eee edu.en, kxcl @androw. amu.edu
ABSTRACT
Deep learning models have propensity for fitting the entire training set even with random labels, which requires memorization of every training sample. In this pa- per, explore the memorization effect in adversarial training (AT) for promoting a deeper understanding of model capacity, convergence, generalization, and espe- cially robust overfitting of the adversarially trained models. We first demonstrate that deep networks have sufficient capacity to memorize adversarial examples of training data with completely random labels, but not all AT algorithms can con- verge under the extreme circumstance. Our study of AT with random labels mo- tivates further analyses on the convergence and generalization of AT. We find that some AT approaches suffer from a gradient instability issue and most recently sug- gested complexity measures cannot explain robust generalization by considering models trained on random labels. Furthermore, we identify a significant drawback memorization in AT that could result in robust overfitting. We then propose a new mitigation algorithm motivated by detailed memorization analyses. Extensive experiments on various datasets validate the effectiveness of the proposed method.
INTRODUCTION
Deep neural networks (DNNs) usually exhibit excellent generalization ability in pattern recognition tasks, despite their sufficient capacity to overfit memorize the entire training set with completely random labels (Zhang et al., 2017). The memorization behavior in deep learning has aroused tremen- dous attention identifying the differences between learning on true and random labels (Arpit al., 2017; Neyshabur et al., 2017), and examining what and why DNNs memorize (Feldman, 2020; Feld- man & Zhang, 2020; Maennel et al., 2020). This phenomenon has also motivated a growing body of works model capacity (Arpit et al., 2017; Belkin et al., 2019), convergence (Allen-Zii et al., 2019; Du et al., 2019; Zou et al., 2020), and generalization (Neyshabur et al., 2017: Bartlett et al., 2017), which consequently provide better understanding of the DNN working mechanism.
In this paper, we explore the memorization behavior for a different learning training (AT). Owing the security threat of adversarial examples, i.e., maliciously generated in- puts by adding imperceptible perturbations to cause misclassification (Szegedy et al., 2014; Goodfel- low et al., 2015), various defense methods have been proposed to improve the adversarial robustness of DNNs (Kurakin et al., 2017; Madry et al., 2018; Liao et al., 2018; Wong & Kolter, 2018; Cohen al., 2019; Zhang et al., 2019b; Pang et al., 2019; 2020; Dong et al., 2020a). AT is arguably the most effective defense technique (Athalye et al., 2018; Dong et al., 2020b), in which the network is trained on the adversarially augmented samples instead of the natural ones (Madry et al., 2018).
adversarial
Despite the popularity, the memorization behavior in AT is less explored. Schmidt et al. (2018) show that a model is able to fully (over)fit the training set against an adversary, i.e., reaching almost 100% robust training accuracy, while the performance on test data is much inferior, witnessing significant generalization gap. The overfitting phenomenon in AT' further investigated in Rice et al. (2020). However, it is not clear whether DNNs could memorize adversarial examples of training data with completely random labels. Answering this question could help to examine the effects of memo- rization in AT under the "extreme" circumstance and facilitate a deeper understanding of capacity, convergence, generalization, and robust overfitting of the adversarially trained models. In general,
"Jun Zhu is the corresponding author. This work was done when Ke Xu was visiting Tsinghua University.

---

Published as a conference paper at ICLR 2022
Standard Training
Adversarial Training
True Labels
Trained
Labels
Trained
Figure I: A conceptual illustration of decision boundaries learned via standard training and adversarial training with truc and random labels, respectively. The model needs significantly more complicated decision boundary to memorize adversarial examples of training data with random labels.
difficult for classifier memorize adversarial examples with random labels since the model en- tails much more complicated decision boundary, as illustrated in Fig. 1. Even though the networks have sufficient capacity, may not necessarily converge. Therefore, we aim to comprehensively study this problem and explore how the analysis can motivate better algorithms.
Our contributions. We first empirically investigate the memorization behavior in AT by perform- ing PGD-AT (Madry et al., 2018) and TRADES (Zhang al., 2019b) with random labels sampled uniformly over all classes. Different from standard training (ST) that can easily memorize random labels (Zhang et al., 2017), AT may fail to converge, with PGD-AT being a typical example. Never- theless, TRADES can converge under this circumstance. demonstrates that DNNs have sufficient capacity memorize adversarial examples of training data with completely random labels. This phenomenon is commonly observed on multiple datasets, network architectures, and threat models.
The memorization analysis has further implications for understanding the convergence and gener- alization of AT. We conduct convergence analysis on gradient magnitude and stability to explain the counter-intuitive different convergence properties of PGD-AT and TRADES with random la- bels since they behave similarly when trained on true labels (Rice et al., 2020). We corroborate that PGD-AT suffers from gradient instability issue while the gradients of TRADES are relatively stable thanks to its adversarial loss. Moreover, by considering models trained on random labels. our gen- eralization analysis indicates that several recently suggested complexity measures are inadequate to explain robust generalization, which is complementary to the findings in ST (Neyshabur et al., 2017). Accordingly, an appropriate explanation robust generalization remains largely under-addressed.
Lastly, but most importantly, we identify significant drawback of memorization in AT that it could result robust overfitting (Rice et al., 2020). We argue that the cause of robust overfitting lies the memorization of one-hot labels in the typical AT methods. The one-hot labels can be inappropriate or even noisy for some adversarial examples because some data naturally lies close to the decision boundary, and the corresponding adversarial examples should be assigned low predictive confidence (Stutz et al., 2020; Cheng et al., 2020). To solve this problem, we propose a new mitigation algorithm that impedes over-confident predictions by regularization for avoiding the excessive memorization of adversarial examples with possibly noisy labels. Experiments validate that our method can eliminate robust overfitting a large extent across multiple datasets, network architectures, threat models, and AT methods, achieving better robustness under variety of adversarial attacks than the baselines.
BACKGROUND
2.1 ADVERSARIAL TRAINING
Let ((x,, 3/4 1 denote a training dataset with n samples, where x, Rd is natural example and y/ E (1, CJ} its true label often encoded as an one-hot vector 1y, with totally C classes. Adversarial training (AT) can be formulated as a robust optimization problem (Madry al., 2018):
)
min
max (X(,))
C(fo(x))))) 91).
where fo is a DNN classifier with parameters that predicts probabilities over all classes, the classification loss (i.e., the cross-entropy loss as L(fo(x),)) y) = - 1, log fx))... and S(x) {x' under the formorm threat
||x' - xII <) is adversarial region centered at with radius models (e.g., E and foc. norms that we consider). The robust optimization problem (1) is solved by using adversarial attacks to approximate the inner maximization and updating the model parameters
2

---

Published as a conference paper at ICLR 2022
0 via gradient descent. A typical method uses projected gradient descent (PGD) (Madry al., 2018) for the inner problem, which starts at a randomly initialized point in S(x,) and iteratively updates the adversarial example under the fo noom threat model by
X = (x)) (x² + a sign( (Vx[(J(x)))) vk)),
(2)
where IT)) is the projection operator and a is the step size.
Besides PGD-AT, another typical AT method is TRADES (Zhang et al., 2019h), which balances the trade-off between robustness and natural accuracy by minimizing a different adversarial loss
min
c(fo(x)).
max x(ES(x.)
)(fe(x)))lf((xx))
(3)
where is the clean cross-entropy loss on the natural example, is the Kullback-Leibi diver- gence, and (3 is balancing parameter. The inner maximization of TRADES is also solved by PGD.
Recent progress of AT includes designing new adversarial losses (Mao et al., 2019; Qin al., 2019; Pang et al., 2020; Wang et al., 2020; Dong et al., 2020a) and network architecture (Xie et al., 2019), training acceleration (Shafahi et al., 2019; Zhang et al., 2019a; Wong et al., 2020), and exploiting more training data (Hendrycks et al., 2019; Alayrac et al., 2019; Carmon et al., 2019; Zhai et al., 2019). Recent works highlight the training tricks in AT (Gowal et al., 2020; Pang et al., 2021).
2.2 RELATED WORK ON DNN MEMORIZATION
has been observed that DNNs can easily memorize training data with random labels (Zhang et al., 2017), which requires "rethinking" of conventional techniques (c.g., VC dimension) explain gen- cralization. Arpit et al. (2017) identify qualitative differences between learning on true and random labels. Further works attempt to examine what and why DNNs memorize (Feldman, 2020; Feld- man & Zhang, 2020; Maennel et al., 2020). Motivated by the memorization phenomenon in deep learning, convergence of training has been analyzed in the over-parameterized setting (Allen-Zhu et al., 2019; Du et al., 2019: Zou et al., 2020), while generalization has been studied with numerous theoretical and empirical complexity measures (Neyshabur et al., 2015; 2017; Bartlett et al., 2017; Novak et al., 2018; Arora et al., 2018; Cao & Gu, 2019; Jiang et al., 2020; Chen et al., 2020).
In contrast, the memorization behavior in AT has been less explored. The previous works demon- strate that DNNs can fit training data against an adversary (Madry et al., 2018; Schmidt et al., 2018; Rice et al., 2020), e.g., achieving nearly 100% robust training accuracy against a PGD adversary, but this behavior is not explored when trained random labels. This paper is dedicated to investigating the memorization in AT under the extreme condition with random labels, while drawing connections capacity, convergence, generalization, and robust overfitting, with the overarching goal of better understanding the AT working mechanism.
MEMORIZATION IN AT AND IMPLICATIONS
In this section, we first explore the memorization behavior in AT through an empirical study. Our analysis raises new questions about the convergence and generalization of AT, many of which cannot be answered by existing works. Thereafter, we provide further analytical studies on the convergence and generalization of AT by considering models trained random labels particularly.
3.1 AT WITH RANDOM LABELS
We explore the memorization behavior of PGD-AT (Madry et al., 2018) and TRADES (Zhang al., 2019b) as two studying cases. The experiments are conducted on CIFAR- 10 (Krizhevsky & Hinton, 2009) with a Wide ResNet model (Zagoruyko & Komodakis, 2016) of depth 28 and widen factor 10 (WRN-28-10) 10). Similar to Zhang et al. (2017), we train network on the original dataset with true labels and on copy of the dataset in which the true labels are corrupted by random ones. For training and robustness evaluation, 10-step loo PGD adversary with 8/255 and 2/255 is adopted. For TRADES, the PGD adversary maximizes the KL divergence during training, while maximizes the cross-entropy loss for robustness evaluation,
common practice (Zhang et al., 2019b). We set = 6.0. In the sequel, we denote accuracy of a classifier against the 10-step PGD adversary as "robust accuracy", and accuracy on natural examples "natural accuracy".
3

---

Published as a conference paper at ICLR 2022
1.0
1.0
Tainiig Accuncy
0.5
Traiuing Aucumey
0.8
0.8
Gonerizzzionn
0.6
lalils tostural
0.4
true lobels (jokrst ace.]
prodoo lsbels (paruml
0.4
tuee label (natural do0.)
0.4
PGD-AT (selurafaze.)
0.2
modoom lsbels (mobus scc)
0.2
mue lobels (robust soc.)
nandor labels (patuaal
0.2-
PGD AT trobust (ce))
TRADES (utturfooe))
0.0
acc.
0.0
TRADES (mbbust apc
100
125
150
25
75 Epuch
100
125
150
0.0
1.0
Epecht
Noine Rune
(a) PGD-AT
(b) TRADES
(c) Generalization gap growth
Figure 2: (a) and (b) show the natural and robust training accuracies of PGD-AT and TRADES, respectively,
when trained on true random labels. (c) shows the generalization gap under varying levels of label noise.
Fig. 2(a) and Fig. 2(b) show the learning curves of PGD-AT and TRADES without explicit regular- izations. Both methods achieve almost 100% natural and robust training accuracies when trained on true labels. When the labels are random, we observe the totally different behaviors between PGD- AT and CRADEESPGD--E accuracies. This phenomenon is somewhat striking because PGD-AT and TRADES perform simi- larly on true labels (Rice et al., 2020). We find that the different memorization behaviors between PGD-AT and TRADES when trained on random labels can commonly be observed across a variety of datasets, model architectures, and threat models (shown in Appendix A. D), indicating that it is a general phenomenon of memorization in the two AT methods. Therefore, our finding is:
fails to converge while TRADES still reaches nearly 100% training
DNNs have sufficient capucity to memorize adversarial examples of training data with completely random labels, but the convergence depends on the AT algorithms.
Partially corrupted labels. We then inspect the behavior of AT under varying levels of label noise from 0% (true labels) to 100% (completely random labels). The generalization gap (i.e., difference between training and test accuracies) presented in Fig. 2(C) grows steadily as we increase the noise rate before the network fails to converge. The learning curves provided in Appendix A.
Explicit regularizations. We study the role of common regularizers in AT memorization, including data augmentation, weight decay, and dropout (Srivastava et al., 2014). We train TRADES on true and random labels with several combinations of regularizers. We observe the explicit regularizers do not significantly affect the model's ability memorize adversarial examples, similar the finding in ST (Zhang et al., 2017; Arpit et al., 2017). The detailed results are provided in Appendix A.I.
3.2
CONVERGENCE ANALYSIS OF AT WITH RANDOM LABELS
Since we have observed counter- intuitive fact that PGD-AT and TRADES exhibit different conver- gence properties with random labels, it is necessary to perform a convergence analysis to understand this phenomenon. Note that our finding can hardly be explained by previous works (Gao et al., 2019; Wang et al., 2019; Zhang et al., 2020).
We first study the effects of different training settings on PGD-AT with random labels. We conduct experiments to analyze each training factor individually, including network architecture, attack steps, optimizer, and perturbation budget. We find that tuning the training settings cannot make PGD-AT converge with random labels (Appendix A.2 details the results). Based on the analysis, we think that the convergence issue of PGD-AT could be result of the adversarial loss function in Eq. (1) rather than other training configurations. Specifically, TRADES in Eq. (3) minimizes a clean cross-entropy (CE) loss on natural examples, making DNNs memorize natural examples with random labels before fitting adversarial examples. As seen in Fig. 2(b), at the very early stage of TRADES training (the first 25 epochs), the natural accuracy starts to increase while the robust accuracy does not. However, PGD-AT in Eq. (1) directly minimizes the CE loss on adversarial samples with random labels, which can introduce unstable gradients with large variance, making it fail to converge. To corroborate the above argument, analyze the gradient magnitude and stability below.
Gradient magnitude. First, we calculate the average gradient norm of the adversarial loss in Eq. (1) w.r.t. model parameters over each training sample for PGD-AT, and similarly calculate the average gradient norm of the clean CE loss (the first term) and the KL loss (the second term) in Eq. (3) w.r.t. parameters for TRADES to analyze their effects, respectively. We present the gradient norm along with training in Fig. 3(a). We can see that at the initial training epochs, the gradient norm of the KI. loss in TRADES is much smaller than that of the CE loss, which indicates that the CE
4

---

Published as a conference paper at ICLR 2022
181
IOE-TT TRADES
-NDAA STRADIS
Tado of Gfeader Marr
-5.
Codaa Seaturtt
0.4
POD-AT
02
IDD-AT TRADES
kmo
350
300
603
1000
1.002
-0.025
0.000
5.23
0.85
10
(a)
(b)
(a)
Figure 3: (a): Gradient norm of PGD-AT and TRADES Figure 4: (a): The &g distance between the gradients along the training process. (b): The ratio of the gra- at 0 and 0 + Ad of different losses, where 0 are ini- dient norm of PGD-AT and TRADES during the first tialized, À E [-0.05, 0.05]. (b): The cosine similarity 1000 training iterations.
between the gradients in cach two successive epochs.
loss dominates TRADES training initially. With the training progressing, the KL loss has a larger gradient norm, making the network memorize adversarial examples. However, it is still unclear why PGD-AT does not rely on a similar learning tactic for convergence. To make direct comparison with TRADES, we rewrite the adversarial loss of PGD-AT in Eq. (1) as
max C(fo(x)) y1) (x) v.) + R(x,, Vs. 0), x(E((x.)
where R(x,, Vi. 0) denotes the difference between the CE loss on adversarial example x and that on natural example x, Hence we can separately calculate the gradient norm of ((fo(x)) yi) and R(x,, Vi. 0) w.r.t. parameters 0 to find out the effect of R(x;, y: 0) on training. Specifically, we measure the relative gradient magnitude, i.e., in PGD-AT we calculate the ratio of the gradient norm VVR((x.9)0) while in TRADES, we similarly calculate the ratio of the gradient norm of the KL loss to that of the CE loss. Fig. 3(b) illustrates the ratio of PGD-AT and TRADES during the first 1000 training iterations. The ratio of PGD-AT's is consistently higher than that of TRADES, meaning that R(x, Vs: 0) has non-negligible impact on training.
Gradient stability. Then, we analyze the gradient stability to explain why PGD-AT cannot con- verge. We denote the adversarial loss of PGD-AT as J y, 0) maxx(eS(x) ((f((x)))))))) xx),)) with the subscript omitted for notation simplicity. We have a theorem on gradient stability.
Theorem 1. Suppose the gradient of the clean cross-entropy loss is locally Lipschitz continuous
(o
ve[(fo(x), 2)lla K - -x|pp=
(4)
for any x Rd. x E S(x), and any 0, where is the Lipschitz constant. Then we have (oJ(xxy, VoJ(x, y. 02)112 S | (x), v) (x),
2cK.
(5)
We provide the proof in Appendix B, where we show the upper bound in Eq. (5) is tight. Theorem indicates that the gradient of the adversarial loss J(x, y, 0) of PGD-AT will change more dramati- cally than that of the clean CE loss (x) y). When 0₃ and 0g are close, the difference between the gradients of L at 0, and O2 is close to 0 due to the semi-smoothness of ever-parameterized DNNs (Allen-Zhu et al., 2019), but that of J relatively large due 2 K in Eq. (5). To validate this, we visualize the change of gradient when moving the parameters O along random direction with magnitude X. In particular, we set 0 as initialization, is sampled from a Gaussian distri- bution and normalized filter-wise (Li et al., 2018). For PGD-AT and TRADES, we craft adversarial examples on-the-fly fly for the model with + Ad and measure the change of gradient by the & dis- tance to gradient 0 averaged over all data samples. The curves gradient change of PGD-AT, TRADES, and the clean CE loss are shown in Fig. 4(a). In a small neighborhood of 0 (i.e., small A) the gradient of PGD-AT changes abruptly while the gradients of TRADES and the clean CE loss are more continuous. The gradient instability leads to a lower cosine similarity between the gradient directions w.r.t. the same data in each two successive training epochs of PGD-AT, as illustrated in Fig. 4(b). Therefore, the training of PGD-AT would be rather unstable that the gradient exhibits large variance, making fail to converge.
Clean CE loss helps PGD-AT converge. To further verify our argu- ment, we add the clean CE loss into the PGD-AT objective to resemble the learning of TRADES with random labels, as
10210 500
110
min Σ (1->)) 7) ((fo(x)) 1.) +y max ((o((().u)) (6) 0
102 125
x(E((x.)
Figure S: AT by Eq. (6)
5

---

Published as a conference paper at ICLR 2022
0.25
Nunce Wough Lomn Landoage
Mode mniiad lhels
Mode oniced cadeer
0.22
Spera
0.15
24
100
20
80 100
20
60
80
53
100
20
60
93
100
62
33
93 90
100
Cap
Raliant Conera Eostice Cap
Clpp
Rohoet Gww Cup
Figure 6: The results on four complexity measures of the adversarially trained models w.r.t. robust generaliza- tion gap. The training settings of these models are provided in Appendix A.3.
where y is gradually increased from 1. By using Eq. (6), the gradient would be stabler at the initial stage and training on random labels can successfully converge, as shown Fig. 5.
In summary, our convergence analysis identifies the gradient instability issue of PGD-AT, provides new insights on the differences between PGD-AT and TRADES, and partially explain the failures of AT under other realistic settings beyond the scope of this section as detailed Appendix A.2.
3.3 GENERALIZATION ANALYSIS OF AT WITH RANDOM LABELS
As our study demonstrates DNNs' ability to memorize adversarial examples with random labels, we labels and how to
raise the question of whether DNNs rely on similar memorization tactic explain/ensure robust generalization. Although many efforts have been devoted to studying robust generalization of AT theoretically or empirically (Yin et al., 2018; Schmidt et al., 2018: 8; Bubeck al., 2019; Tu et al., 2019; Wu et al., 2020), they do not take the models trained on random labels into consideration. As it easy to show that the explicit regularizations are not the adequate explanation of generalization in ST (Zhang et al., 2017; Arpit et al., 2017) and AT (see Appendix A.3), people resort to complexity measures of model to explain generalization (i.e., a lower complexity should imply smaller generalization gap). Here show how the recently proposed complexity measures fail to explain robust generalization when comparing models trained on true and random labels.
We consider several norm-based and sharpness//latness-beee of a network by 0 ==
measures. We denote the parameters
The norm-based measures include spectral norm Traugia II [Will
and l, norm "inogin -i=11 Willi of model parameters, where Ymargin is margin on model output to make them ccale-insensitive (Neyshabur et al., 2017). The spectral norm appears in the theoretical robust generalization bounds (Yin al., 2018; Tu et al., 2019) and is related the Lipschitz constant of neural networks (Cisse et al., 2017). The ε norm is adopted to reduce the robust generalization gap (Yin et al., 2018). The sharpness/flatness-aase measures include the curvature of input loss landscape
et al., 2019) as the dominant eigenvalue of the Hessian matrix, as well as the flatness of weight loss landscape (Wu al.. 2020) related to the change of adversarial loss when moving the weights along a random direction. Fig. 6 plots the four complexity measures w.r... robust generalization gap of several models trained with various combinations of regularizations on true random labels. The results show that the first three measures can hardly ensure robust generalization, that lower complexity does not necessarily imply smaller robust generalization gap, e.g., the models trained random labels can even lead to lower complexity than those trained on true labels. Among them, the flatness of weight loss landscape (Wu et al., 2020) is more reliable.
In summary, the generalization analysis indicates that the previous approaches, especially various complexity measures, cannot adequately explain and ensure the robust generalization performance in AT. Our finding of robust generalization in AT is complementary ST (Zhang et al., 2017; Neyshabur et al., 2017; Jiang et al., 2020). Accordingly, robust generalization of adversarially trained models remains an open problem for future research.
that of standard generalization in
4 ROBUST OVERFITTING ANALYSIS
Rice al. (2020) have identified robust overfitting as dominant phenomenon in AT, i.e., shortly after the first learning rate decay, further training will continue to decrease the robust test accuracy. They further show that several remedies for overfitting, including explicit ε and &g regularizations, data augmentation, etc., cannot gain improvements upon early stopping. Although robust overfitting has been thoroughly investigated, there still lacks an explanation of why occurs. In this section, we draw a connection between memorization and robust overfitting in AT by showing that robust over- fitting is caused by excessive memorization of one-hot labels in the typical AT methods. Motivated by the analysis, we then propose an effective strategy to eliminate robust overfitting.

---

Published as a conference paper at ICLR 2022
1.0
1.0
4.0
Model
0.8-
Bobaat Tox Astunay
0.8
Socond Model
3.0
Moveebbl
2.0-
0.4-
0.2-
roboaa
0.2-
1.0-
natioal bos pon.
0.0
0.0
20
200
100
150
100
200
Epacis
Epocha
Sample
(a)
(C)
Figure 7: (a): The accuracy curves PGD-AT with true labels to reproduce robust overfitting, (b): The robust test accuracy of PGD-AT under various perturbation budgets e. (c): The adversarial loss of two independently
trained networks by PGD-AT on 500 samples sorted by the loss of the first model.
4.1 EXPLAINING ROBUST OVERHITTING
The typical AT approaches (e.g., PGD-AT, TRADES) commonly adopt one-hot labels as the targets for training, as introduced in Sec. 2. The one-hot labels could be inappropriate for some adversarial examples because it difficult for network to assign high-confident one-hot labels for all perturbed samples within the perturbation budget e (Stutz et al., 2020; Cheng et al., 2020). Intuitively, some examples may naturally lie close to the decision boundary and should be assigned lower predictive confidence for the worst-case adversarial examples. It indicates that one-hot labels of some training data may be noisy in AT After certain training epoch, the model memorizes these "hard" training examples with possibly noisy labels, leading to the reduction of test robustness, as shown in Fig. 7(a).
Thus, we hypothesize the cause of robust overfitting lies in the memorization of one-hot labels.
Our hypothesis is well supported by two pieces of evidence. First, we find that when the perturbation budget is small, robust overfitting does not occur, as shown in Fig. 7(b). This observation implies that the one-hot labels are more appropriate as the targets for adversarial examples within smaller neighborhood while become noisier under larger perturbation budget and lead to overfitting. Sec- ond, we validate that the "hard" training examples with higher adversarial loss values are consistent across different models. We first train two independent networks (using the same architecture and different random seeds) by PGD-AT and calculate the adversarial loss for each training sample. We show the adversarial losses on 500 samples sorted by the loss of the first model in Fig. 7(c). It can be seen that the samples with lower adversarial losses of the first model also have relatively lower losses of the second one and vice versa. We further quantitatively measure the consistency of the ad- versarial losses of all training samples between the two models using the Kendall's rank coefficient (Kendall, 1938), which is 0.85 in this case. A similar result can be observed for two different model architectures (see Appendix C. The results verify that the "hard" training examples with possibly noisy labels are intrinsic of a dataset, supporting our hypothesis on why robust overfitting occurs.
4.2 MITIGATING ROBUST OVERFITTING
Based on the above analysis, we resort to the methods that are less prone to overfit noisy labels for mitigating robust overfitting in AT. Although learning with noisy labels has been broadly studied in ST (Natarajan et al., 2013; Patrini et al., 2017; Jiang et al., 2018; Han et al., 2018; 88; Zhang & Sabunnu, 2018), we find that most of these approaches are not suitable for AT. For example, a typical line of methods filter out noisy samples and train the models on the identified clean samples (Jiang et al., 2018; Han et al., 2018; Ren et al., 2018). However, they will neglect portion of training data with noisy labels, which can lead to inferior results for AT due to the reduction of training data (Schmidt et al., 2018). Table 2 shows the results to validate this.
To address this problem, we propose to regularize the predictions of adversarial examples from being over-confident by integrating the temporal ensembling (TE) approach (Laine & Aila, 2017) into the AT frameworks. TE maintains ensemble prediction of each data and penalizes the difference between the current prediction and the ensemble prediction, which is effective for semi- semi-supervised learning and learning with noisy labels (Laine & Aila, 2017). We think that TE is suitable for AT since it enables to leverage all training samples and hinders the network from excessive memoriza- tion of one-hot labels with a regularization term. Specifically, we denote the ensemble prediction of
Note that we argue the onc-hot labels are noisy when used in AT, but do not argue the ground-truth labels of the dataset noisy, which is different from previous work (Sanyal al., 2021).
7

---

Published as a conference paper at ICLR 2022
Table 1: Test accuracy (%) of several methods on CIFAR-10, CIFAR- 100, and SVHN under the Eor norm with e 8/255 based on the ResNet-1 18 architecture. We choose the best checkpoint according to the highest robust accuracy on the test set under PGD-10.
Natural Accuracy Best
PGD-10
PGD-
C&W-1000
AutoAttack Final Diff'
Method
Final
Diff
Best
Final
Diff
Best
Final
Diff
Best
Final
Diff
Best
PGD-AT
83.75
84.82
-1.07
52.64
44.92
7.72
51.22
42.74
8.48
50.11 43.63 7.48 47.74 41.84 5,90
PGD-AT+TE
82.35
82.79 -0.44
55.79
54.83 0.96
54.65 53.30 1.35 52.30 51.73 0.57 50.59 49.62 0.97
TRADES
81.19
82.48
-1.29
53.32
50.25
3.07
52.44
48.67
3.77
49.88
48.
1.74 49.03
46.80 2.23
TRADES+TE
83,86
83.97
-0.11
55.15 54.42 0.73 53.74 53.03 0.71 50.77
50.63 0.14
49.77 49.20 0.57
The evaluation results CIFAR-IO.
Natural Accuracy Final
PGD-10
PGD-1000
C&W-1000
AutoAttack
Method
Best
Diff
Best
Final Diff
Best
Final Diff
Best
Final
Diff
Best
Final Diff'
PGD-AT
57.54
57.51 0.03
29.40
21.75 7.65
28.54 20.63 7.91
27.06 21.17 5.89 24.72 19.34 5.38
PGD-AT+TE
56.45 57.12 -0.67
31.74 30.24 1.50
31.27 29.80 1.47 28.27 27.36 0.91 26.30 25.34 0.96
TRADES
57.98
56.32
1.66
29.93
27.70 2.23
29.51
26.93 2.58 25.46
24.42
1.04
24.61
23,40 1.21
TRADES+TE
59.35
58.72
0.63
31.09 30.12 0.97
30.54 29.45 1.09 26.61 25.94
0.67
25.27
24.55 0.72
(h) The evaluation results on CIFAR-100.
Natural Accuracy Final
PGD-10
PGD-1000
C&W-1000
AutoAttack
Method
Best
Diff
Best
Final Diff
Best Final Diff
Best
Final
Diff
Best
Final Diff
PGD-AT
89.00 90.55 -1.55
54.51
46.97 7.54
52.22
42.85 9.37
48.66 44.13 4.53 46.61 38.24 8.37
PGD-AT+TE
90.09 90.91 -0.82
59.74 59.05 0.69
57.71 56,46 1.25 54.55 53.94 0.61 51.44 50.61 0.83
TRADES
90.88
91.30
-0.42
59.50
57.04 2.46
52.78
52.78 50.17 2.61 (52.76
50.53
2.23
40.36
38,88 1.48
TRADES+
89.01 88.52 0.49 59.81
58.49 1.32
58.24 56.66 1.58 54.00 53.24 0.76 51.45 50.16 1.29
(c) The evaluation results on SVHN.
a training sample x; as P, which is updated in each training epoch as P, f 7)-P; 1 fo where 17 is the momentum term. The training objective of PGD-AT with TE can be expressed as
min max (E(fo(x)),u vi) + (l) P.112). x(E((x))
where P; is the normalization of P; as a probability vector and w is a balancing weight. TE can be similarly integrated with TRADES with the same regularization term. The network would learn to fit relatively easy samples with one-hot labels in the initial training stage, as shown in Fig. 7(a). After the learning rate decays, the network can keep assigning low confidence for hard samples with the regularization term in Eq. (7) and avoid fitting one-hot labels. Therefore, the proposed algorithm enables to learn under label noise in AT and alleviates the robust overfitting problem.
5 EMPIRICAL EVALUATION ON MITIGATING ROBUST OVERFITTING
In this section, we provide the experimental results on CIFAR-10, CIFAR- 100 (Krizhevsky & Hin- ton, 2009), and SVHN (Netzer et al., 2011) datasets to validate the effectiveness of our proposed method. Code is available at https: http//gttttt gighub..................... com/ dongyp1 3 memorization
Training details. We adopt the common setting that the perturbation budget is 8/255 under the Eo norm in most experiments. We consider PGD-AT and TRADES two typical AT baselines and integrate the proposed TE approach into them, respectively. We use the ResNet- 18 (He et al., 2016) model as the classifier in most experiments. In training, we use the 10-step PGD adversary with a = 2/255. The models are trained via the SGD optimizer with momentum 0.9, weight decay 0.0005, and batch size 128. For CIFOR-10/100, we set the learning rate as 0.1 initially which decayed by 0.1 at 100 and 150 epochs with totally 200 training epochs. For SVHN, the learning rate starts from 0.01 with a cosine annealing schedule for a total number of 80 training epochs. In our method, We set 7 = 0.9 and = 30 along Gaussian ramp-up curve (Laine & Aila, 2017).
Evaluation results. We adopt PGD-10, PGD-1000, 1000, C&W-1000 (Carlini & Wagner, 2017), and AutoAttack (Croce& & Hein, 2020b) for evaluating adversarial robustness rigorously. AutoAttack is a strong attack to evaluate model robustness, which is composed of an ensemble of diverse attacks, including APGD-CE (Croce& & Hein, 2020b), APGD-DIR (Croce & Hein, 2020b), FAB (Croce
8

---

Published as a conference paper at ICLR 2022
& Hein, 2020a), and Square attack (Andriushchenko et al., 2020). To show the performance of robust overfitting, we report the test accuracy on the best checkpoint that achieves the highest robust test aceuracy under PGD-10 and the final checkpoint, as well as the difference between these two checkpoints. The results of PGD-AT, TRADES, and the combinations of them with our proposed approach (denoted PGD-AT+TE, and TRADES+TE) on the CIFAR-10, CIFAR-100, and SVHN datasets are shown in Table 1.
We can observe that the differences between best and final test accuracies of our method are reduced to around 1%. while the accuracy gaps of PGD- AT and TRADES are much larger. It indicates that our method largely eliminates robust overfitting. Due to being less affected by robust over- fitting, our method achieves higher robust accuracies than the baselines. We also show the learning curves of these methods in Fig. 8. We consis- tently demonstrate the effectiveness of our method on different network architectures (including WRN-34-10 and VGG-16) and threat models (including &g norm), which will be shown in Appendix C.2.
1.9
1.0
0.8
0.8
Avcuncy
0.6
0.6
FGD AT (saturd aco))
TRADES (catural acc.)
0.2
PGD AT Golead aoc.)
0.2
TRADES (ndraa) exc.)
7GD-AT TE (satund ane
TRADES TE (manuaa acc.)
PGD crbbatt acc.)
TRADES TE (ubuat soc.)
50
100
150
200
50
100
150
200
Epuchs
Epuchs
Figure 8: The natural and robust test accuracy curves (under PGD- 10) of PGD-AT, TRADES, and their extensions by integrating the proposed TE approach. The models are trained on CIFAR-10 un- der the for norm with = 8/255 based on the ResNet-18 18 archi- tecture.
Discussion and comparison with related works. Our method is kind of similar to the label smooth- ing (LS) technique, which studied in AT (Pang et al., 2021). Recent works have also introduced the smoothness in training labels and model weights (Chen et al., 2021: Huang et al., 2020), which can alleviate robust overfitting to some extent. The significant difference between our work and them is that we provide reasonable explanation for robust overfitting werfitting-one-hoo labels are noisy for AT, while previous methods did not give such an explanation and could be viewed as solutions to our identified problem. To empirically compare with these methods, we conduct experiments on CIFAR-10 10 with the ResNet- network. Under the PGD-AT framework, we compare with the baseline PGD-AT, PGD-AT+LS, self self-adaptive training (SAT) (Huang et al., 2020), and knowledge distillation with stochastic weight averaging (KD- (WA) (Chen et al., 2021). We also adopt the Co- teaching approach (Han al., 2018) adapted PGD-AT, which jointly trains two models using the filtered samples given by each other. The results under the adopted attacks are presented in Table 2. Although various techniques can alleviate robust overfitting, our method achieves better robustness than the others, validating its effectiveness. For Co-teaching, though robust overfitting is alleviated, the performance is worse than our proposed method due to the reduction of training data.
Table 2: Test accuracy (%) of the proposed method and other methods on CIFAR-10 under the la norm with 8/255 based on the ResNet- 18 architecture.
Natural Accuracy
PGD-10
PGD-1000
C&W-1000
AutoAttack Final
Method
Best Final Diff
Best Final Diff'
Best
Final Diff
Best
Final
Diff
Best
Diff
PGD-AT
83.75 84.82 -1.07 52.64 44.92 7.72 51.22 42.74 8.48 50.11 43.63 7.48 47.74
41,84 5.90
PGD-AT+LS
82.68 85.16 -2.48 53.70 48,90 4.80 52.56 46.31 6.25
50.41 46.06 4.35
49.02 44.39 4.63
SAT
82.81
81.86 0.95 53.81 53.31 0.50 52.41 52.00 0.41
51.99
51.71 0.28
50.21
49.73 0.48
KD-SWA
84.84
85.26 -0.42 54.89 53.80 1.09 53.31 52.45 0.86
51.48
50.91 0.57
50.42
49.83 0.59
Co-teaching
81.94
82.22 -0.28
51.27
50.52 0.75 50.15 49.12
1.03
50.85
49.86 0.99
49.60
48.49 1.11
PGD-AT+TE
82.35 82.79
-0.44
55.79
54.83
0.96
54.65 53.30
1.35
52.30
51.73 0.57
50.59
49.62 0.97
CONCLUSION
In this paper, we demonstrate the capacity of DNNs to fit adversarial examples with random labels by exploring memorization in adversarial training, which also poses open questions on the convergence and generalization of adversarially trained models. We validate that some AT methods suffer from gradient instability issue and robust generalization can hardly be explained by complexity measures. We further identify significant drawback of memorization in AT related to the robust overfitting phenomenon robust overfitting is caused by memorizing one-hot labels in adversarial training. We propose a new mitigation algorithm address this issue, with the effectiveness validated extensively.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENTS
This work was supported by the National Key Research and Development Program of China (2020AAA0105000, 2020AAA0104304, 2020AAA0106302), NSFC Projects (Nos. 61620106010, 02061136001, 61621136000 136008, 62076147, U19B2034, U181 1461, U19A2081), 1), Beijing NSF Project (No. JQ19016), Beijing Academy Artificial Intelligence (BAAI), Tsinghua-Alibabe Joint Re- search Program, Tsinghua Institute for Guo Qiang, Tsinghua-OPPO Joint Research Center for Fu- ture Terminal Technology.
ETHICS STATEMENT
The existence of adversarial examples can pose severe security threats to machine learning and deep learning models when they are deployed real-world applications. The vulnerability to adversarial examples could also lower the confidence of the public machine learning techniques. Therefore, is important to develop more robust models. As the most effective method for promoting model robustness, adversarial training (AT) has not been fully investigated. This paper aims investigate the memorization effect AT to facilitate better understanding of working mechanism. Some findings in this paper can be analyzed more deeply, including theoretical analysis of AT convergence, generalization, etc., which we leave to future work.
REPRODUCIBILITY STATEMENT
Most of the experiments are easily reproducible. We provide the code for reproducing the results at com/ homm//dnggp13333 hatggyypppmmmoo///. /ww.ri/mmorieooioo/mmmmmmoo za -on-AT.
REFERENCES
Jean-Baptiste Alayrac, Jonathan Uesato, Po-Sen Huang, Alhussein Fawzi, Robert Stanforth, and Pushmeet Kohli. labels required for improving adversarial robustness? In Advances in Neural Information Processing Systems (NeurIPS), pp. 12214-12223, 2019.
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. convergence theory for deep learning via over- parameterization. In International Conference Machine Learning (ICML), pp. 242-252, 2019.
Maksym Andriushchenko. Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square at- tack: query-efficient black-box adversarial attack via random search. In European Conference on Computer Vision (ECCV), pp. 484-501, 2020.
Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via compression approach. International Conference on Machine Learning (ICML), pp. 254-263, 2018.
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxin- der S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, closer look at memorization in deep networks. In International Conference on Machine Learning (ICML), pp. 233 242,22222222 2017.
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of se- curity: Circumventing defenses to adversarial examples. In International Conference on Machine Learning (ICML), pp. 274-283. 2018
Peter L Bartlett, Dylan Foster, and Matus Telgarsky. Spectrally-normalizedd margin bounds for neural networks. In Advances in Neural Information Processing Systems (NIPS), pp. 6240-6249, 2017.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences, 15849-) 2019.
10

---

Published as a conference paper at ICLR 2022
Sébastien Bubeck, Yin Tat Lee, Eric Price, and llya Razenshteyn. Adversarial examples from com- putational constraints. In International Conference on Machine Learning (ICML), pp. 831-840) 2019.
Yuan Cao and Quanquan Gu. Generalization bounds of stochastic gradient descent for wide and deep neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In JEEE Symposium on Security and Privacy (SP), pp. 39-57, 2017.
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, Percy Liang, and John C Duchi. Unlabeled data improves adversarial robustness. In Advances in Neural Information Processing Systems (NeurIPS), pp. 11192-11208 2019.
Tianlong Chen, Zhenyu Zhang, Sijia Liu, Shiyu Chang, and Zhangyang Wang. Robust overfitting may be mitigated by properly learned smoothening. In International Conference on Learning Representations (ICIR), 2021.
Zixiang Chen, Yuan Cao, Difan Zou, and Quanquan Gu. How much vverparrameterization is suf- ficient to learn deep relu networks? In International Conference on Learning Representations (ICLR), 2020.
Minhao Cheng, Qi Lei, Pin-Yu Chen, Inderjit Dhillon, and Cho-Jui Hsich. Cat: Customized adver- sarial training for improved robustness. arXiv preprint arXiv:2002.06789 2020.
Moustapha Cisse, Piotr Bojanowski, Edouard Grave, Yann Dauphin, and Nicolas Usunier. Parseval networks: Improving robustness to adversarial examples. In International Conference on Machine Learning (ICML), pp. 854-863, 2017.
Jeremy M Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. International Conference on Machine Learning (ICML), pp. 1310-1320, 2019.
Francesco Croce and Matthias Hein. Minimally distorted adversarial examples with fast adaptive boundary attack. In International Conference on Machine Learning (ICML), pp. 2196-220 2205, 2020a.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-fiee attacks. In International Conference on Machine Learning (ICML), pp- 2206-2216, 2020b.
Yinpeng Dong, Zhijie Deng, Tianyu Pang, Jun Zhu, and Hang Su. Adversarial distributional training for robust deep learning. In Advances in Neural Information Processing Systems (NeurIPS), pp. 8270-8283, 2020a.
Yinpeng Dong, Qi-An Fu, Xiao Yang, Tianyu Pang, Hang Su, Zihao Xiao, and Jun Zhu. Benchmark- ing adversarial robustness image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 321-331, 2020b.
Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In International Conference on Machine Learning (ICML), pp. 1675-1685, 2019.
Vitaly Feldman. Does learning require memorization? a short tale about a long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium Theory of Computing, pp. 954 959, 2020.
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. In Advances in Neural Information Processing Systems (NeurIPS), pp. 2881-2891, 2020.
Ruiqi Gao, Tianle Cai. Haochuan Li, Cho-Jui Hsieh, Liwei Wang, and Jason D Lee. Convergence of adversarial training in overparametrized neural networks. In Advances in Neural Information Processing Systems (NeurIPS), pp. 13029-130400 2019.

---

Published as a conference paper at ICLR 2022
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations (ICLR), 2015.
Sven Gowal, Chongli Qin, Jonathan Uesato, Timothy Mann, and Pushmeet Kohli. Uncovering the limits of adversarial training against norm-bounded adversarial examples. arXiv preprint 20Xi::000000359,, 2020.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in Neural Information Processing Systems (NeurIPS), pp. 8527-8537, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre-training can improve model robustness and uncertainty. In International Conference on Machine Learning (ICML), pp. 2712-2721, 2019.
Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinherger. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4700-4708, 2017.
Lang Huang, Chao Zhang, and Hongyang Zhang. Self-adaptive training: beyond empirical risk minimization. In Advances in Neural Information Processing Systems (NeurIPS), pp. 19365- 19376, 2020.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentorne:: Learning data- driven curriculum for very deep neural networks on corrupted labels. In International Conference Machine Learning (ICML), pp. 2313, 2018.
Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantas- tic generalization measures and where find them. In International Conference Learning Representations (ICLR), 2020.
Maurice G Kendall. A new measure of rank correlation. Biometrika, 30(1/2):81-93, 1938.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical report, University of Toronto, 2009.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning at scale. In International Conference on Learning Representations (ICLR), 2017.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In International Conference on Learning Representations (ICLR), 2017.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss land- scape neural nets. In Advances in Neural Information Processing Systems (NeurIPS), pp- 6389-6399, 2018
Fangzhou Liao, Ming Liang, Yinpeng Dong, Tianyu Pang, Xiaolin Hu, and Jun Zhu. Defense against adversarial attacks using high-level representation guided denoiser. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7778-1787, 2018.
Chen Liu, Mathieu Salzmann, Tao Lin, Ryota Tomioka, and Sabine Süsstrunk. On the loss landscape of adversarial training: Identifying challenges and how to overcome them. In Advances Neural Information Processing Systems (NeurIPS), 2020a.
Shengehao Liu, Dimitris Papailiopoulos, and Dimitris Achlioptas. Bad global minima exist and sgd can reach them. In Advances in Neural Information Processing Systems (NeurIPS), 2020b.
12

---

Published as a conference paper at ICLR 2022
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations (ICLR), 2018.
Hartmut Maennel, Ibrahim Alabdulmohsin, Ilya Tolstikhin, Robert JN Baldock, Olivier Bousquet, Sylvain Gelly, and Daniel Keysers. What do neural networks learn when trained with random labels? In Advances in Neural Information Processing Systems (NeurlPS), pp. 19693-19704, 2020.
Chengzhi Mao, Ziyuan Zhong, Junfeng Yang, Carl Vondrick, and Baishakhi Ray. Metric learning for adversarial robustness. In Advances in Neural Information Processing Systems (NeurIPS), pp- 480-491, 2019.
Scyed-Mohsen Moosavi-Dezfoolia Alhussein Fawzi, Jonathan Uesato, and Pascal Frossard. Ro- bustness via curvature regularization, and vice versa. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9078-9086, 2019.
Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. In Advances in Neural Information Processing Systems (NIPS), pp. 1196-1200 1204, 2013.
Yurii E Nesterov. A method for solving the convex programming problem with convergence rate (1/k2 2). In Dokl. akad. nauk Sssr, volume 269, pp. 543-547, 1983.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning. 2011.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neural networks. In Conference on Learning Theory (COLT), pp. 1376-1401, 2015.
Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring gener- alization in deep learning. In Advances in Neural Information Processing Systems (NIPS), pp- 5947-5956, 2017.
Roman Novak, Yasaman Bahri, Daniel A. Abolafia, Jeffrey Pennington, and Jascha Sohl-Dickstein. Sensitivity and generalization neural networks: an empirical study. In International Conference Learning Representations (ICLR), 2018.
Tianyu Pang, Kun Xu, Chao Du, Ning Chen, and Jun Zhu. Improving adversarial robustness via promoting ensemble diversity. In International Conference Machine Learning (ICML), pp- 4970-4979, 2019.
Tianyu Pang, Kun Xu, Yinpeng Dong, Chao Du, Ning Chen, and Jun Zhu. Rethinking softmax cross- entropy loss for adversarial robustness. In International Conference on Learning Representations (ICLR), 2020.
Tianyu Pang, Xiao Yang, Yinpeng Dong, Hang Su, and Jun Zhu. Bag tricks for adversarial training. In International Conference on Learning Representations (ICLR), 2021.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1944-1952, 2017.
Chongli Qin, James Martens, Sven Gowal, Dilip Krishnan, Krishnamurthy Dvijotham, Alhussein Fawzi, Soham De, Robert Stanforth, and Pushmeet Kohli. Adversarial robustness through local linearization. In Advances in Neural Information Processing Systems (NeurIPS), pp. 13847- 13856, 2019.
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In International Conference on Machine Learning (ICML), pp. 4334 4343, 2018.
13

---

Published as a conference paper at ICLR 2022
Leslie Rice, Eric Wong, and Zico Kolter. Overfitting in adversarially robust deep learning. In International Conference on Machine Learning (ICML), pp. 8093-81080 8104, 2020.
Amartya Sanyal, Puneet K. Dokania, Varun Kanade, and Philip Torr. How benign is benign overfit- ting? In International Conference on Learning Representations (ICLR), 2021.
Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. Adver- sarially robust generalization requires more data. In Advances in Neural Information Processing Systems (NeurIPS), pp. 5014-5026, 2018.
Ali Shafahi, Mahyar Najibi, Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! In Advances in Neural Information Processing Systems (NeurIPS), pp. 3358-3369, 2019.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, lly Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929 1958, 2014.
David Stutz, Matthias Hein, and Bernt Schiele. Confidence-calibrate adversarial training: Gener- alizing unseen attacks. In International Conference Machine Learning (ICML), pp. 9155- 9166, 2020.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations (ICLR), 2014.
Zhuozhuo Tu, Jingwei Zhang, and Dacheng Tao. Theoretical analysis of adversarial learning: A minimax approach. In Advances in Neural Information Processing Systems (NeurIPS), pp. 12280- 12280-12290, 2019.
Yisen Wang. Xingjun Ma, James Bailey, Jinfeng Yi, Bowen Zhou, and Quanquan Gu. On the convergence and robustness of adversarial training. In International Conference Machine Learning (ICML), pp. 3886-6595, 2019.
Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving adversarial robustness requires revisiting misclassified examples. In International Conference on Learning Representations (ICLR), 2020.
Eric Wong and Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International Conference on Machine Learning (ICML), pp. 5286-5295, 2018.
Eric Wong, Leslic Rice, and J. Zico Kolter. Fast is better than free: Revisiting adversarial training. In International Conference on Learning Representations (ICLR), 2020.
Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust gen- eralization. In Advances in Neural Information Processing Systems (NeurIPS), pp. 2958-2969, 2020.
Cihang Xie and Alan Yuille. Intriguing properties of adversarial training. In International Confer- ence on Learning Representations (ICLR), 2020.
Cihang Xie, Yuxin Wu, Laurens van der Maaten, Alan Yuille, and Kaiming He. Feature denoising for improving adversarial robustness. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 501-509, 2019.
Dong Yin, Kannan Ramchandran, and Peter Bartlett. Rademacher complexity for adversarially robust generalization. International Conference on Machine Learning (ICML) pp. 7085-7094, 2018.
Fisher Yu, Dequan Wang. Evan Shelhamer, and Trevor Darrell. Deep layer aggregation. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2403- 2412, 2018
14

---

Published as a conference paper at ICLR 2022
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Proceedings the British Machine Vision Conference (BMVC), 2016.
Runtian Zhai, Tianle Cai, Di He, Chen Dan, Kun He, John Hopcroft, and Liwei Wang. Adversarially robust generalization just requires more unlabeled data. arXiv preprint arXiv:1906.0055: 2019.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Rep- resentations (ICLR), 2017.
Dinghuai Zhang, Tianyuan Zhang, Yiping Lu, Zhanxing Zhu, and Bin Dong. You only propagate once: Painless adversarial training using maximal principle. In Advances in Neural Information Processing Systems (NewrIPS), pp. 227-238, 2019a.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P Xing. Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. International Conference on Machine Learning (ICML), pp. 7472-7482, 2019b.
Yi Zhang, Orestis Plevrakis, Simon S Du, Xingguo Li, Zhao Song, and Sanjeev Arora. Over- parameterized adversarial training: An analysis overcoming the curse dimensionality. Ad- vances in Neural Information Processing Systems (NeurIPS), pp. 679-688, 2020.
Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In Advances in Neural Information Processing Systems (NeurIPS), pp- 8778- 8788, 2018
Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu. Gradient descent optimizes over- parameterized deep relu networks. Machine Learning, 109(3):467-492, 2020.
15

---

Published as a conference paper at ICLR 2022
A ADDITIONAL EXPERIMENTS ON MEMORIZATION IN AT
In this section, we provide additional experiments on the memorization behavior in AT. All of the experiments are conducted on NVIDIA 2080 Ti GPUs. The source code of this paper is submitted as the supplementary material, and will be released after the review process.
A.1 MEMORIZATION OF PGD-AT AND TRADES
A.1.1 DIFTERENT TRAINING SETTINGS
We first demonstrate that the different memorization behaviors between PGD-AT and TRADES can be generally observed under various settings.
1.0
0.8
mmma
Training Accuracy
Training Accuracy
0.6
labels (natural acc.)
0.4
true labels (robust acc.)
0.4
true labels (natural acc.)
random labels (natural acc.)
true labels (robust acc.)
0.2
0.2
random labels (robust acc.)
random labels (natural acc.)
random labels (robust acc.)
0.0
0.0
0
25
50
75 Epachs
100
125
150
25
50
75
100
125
150
Epochs
(a) PGD-AT
(b) TRADES
Figure A.1: The natural and robust training accuracies of PGD-AT and TRADES CIFAR-100 when trained on true or random labels.
1.0
1.0
0.8
0.8
Training Accuracy
Training Accurucy
0.6
0.6
true labels (natural acc.)
0.4
labels (robust acc.)
0.4
true labels (natural acc.)
random labels (natural acc.)
true labels (robust acc.).
0.2
0.2
random labels (robust acc.)
random labels (natural acc.)
random labels (rubust acc.)
0.0
0
50
75
100
125
150
0
50
75
100
125
150 175 200
Epochs
Epochs
(a) PGD-AT
(b) TRADES
Figure A.2: natural robust training accuracies PGD-AT and TRADES on SVHN when trained on true or random labels.
Datasets. Similar to Fig. 2, we show the accuracy curves of PGD-AT and TRADES when trained on true random labels CIFAR-100 (Krizhevsky & Hinton, 2009) in Fig. A.I and on SVHN (Netzer et al., 2011) I) Fig. A.2. We consistently observe that PGD-AT fails to converge with random labels, while TRADES can successfully converge, although it does not reach 100% accuracy SVHN.
Model architectures. We then consider other network architectures, including the DenseNet-122 121 model (Huang al., 2017) and the deep layer aggregation (DLA) model (Yu et al., 2018). The corresponding results are shown Fig. A.3. The similar results can be observed, although it may take more training epochs make TRADES converge with the smaller DenseNet-121 network.
16

---

Published as a conference paper at ICLR 2022
1.0
0.8
Training Accuracy
Training Accuracy
0.6
PGD-AT AT (natural acc.)
PGD-AT (natural acc.)
0.4
PGD-AT (robust acc.)
PGD-AT (robust acc.)
TRADES (natural acc.)
TRADES (natural acc.)
0.2
TRADES (robust acc.)
TRADES (robust acc.)
0.0
50
100
200
250
75 Epochs
100
125
150
Epochs
(a) DenseNet-121
(b) DLA
Figure A.3: The natural and robust training accuracies of PGD-AT and TRADES on CIFAR-I0 with different architectures when trained on random labels.
1.0
0.8
Training Accuracy
PGD-AT (natural acc.)
0.4
PGD-AT (robust acc.)
TRADES (natural acc.)
0.2
TRADES (robust acc.)
0.0
25
50
75 Epuchs
100
125
150
Figure A.4: The natural and robust training accuracies of PGD-AT and TRADES CIFAR-10 under the fz-norm threat model when trained on random labels.
Threat models. We further consider the fo-norm threat model, in which we set = 1.0 and a = 0.25 in the 10-step PGD adversary. The learning curves of PGD-AT and TRADES are shown in Fig. A.4, which also exhibit similar results.
Perturbation budget. We study the memorization behavior in AT with different perturbation bud- gets c. In Fig. A.5, we show that when the perturbation budget c = 16/255 (E norm), TRADES trained on random labels can still converge. But when set a larger budget (e.g., both PGD-AT and TRADES cannot obtain near 100% robust training accuracy. We also find under this condition, even AT trained on true labels cannot get 100% robust training accuracy, indicating that the gradient instability issue discussed in Sec. 3.2 results in the convergence problem.
32/ 22/255),
In summary, our empirical observation that PGD-AT and TRADES perform differently when trained on random labels is general across multiple datasets, network architectures, and threat models.
A.1.2 LEARNING CURVES UNDER DIFFERENT NOISE RATES
We show the learning curves of PGD-AT and TRADES under varying levels of label noise in Fig. A.6 and Fig. A.7, respectively. In this experiment, we adopt the weight decay and data augmen- tation for regularizations. We can see that the network achieves maximum accuracy on the test set before fitting the noisy training set. Thus the model learns easy and simple patterns first before fit- ting the noise, similar to the finding in ST (Arpit et al., 2017). ). can also observed that under 80% noise rate, PGD-AT fails to converge. Note that when the noise rate is 0%, the network is trained on true labels, but the robust test accuracy also decreases after a certain epoch. This phenomenon is called robust overfitting (Rice et al., 2020), which is studied in Sec. 4.
17

---

Published as a conference paper at ICLR 2022
1.0
1.0
0.8
Training Accuracy
Training Accuracy
0.6
true labels (natural acc.)
0.4
true labels (robust acc.)
trée labels (natural acc.)
random labels (natural acc.)
frue labels (robust aec.)
random labels (robust acc.)
random labels (natural acc.)
random labels (robust acc.)
0.0
125
75
100
125
150
Epochs
Epochs
(a) PGD-AT
(b) TRADES
Figure A.5: The natural and robust training accuracies of PGD-AT and TRADES on CIFAR-10 with 16/255 when trained true random labels.
Nelog Bate 0.0
Neioe Baac 0.2
Naiee Basz
Naiee Rasc 68
Navac Rasc
G4
G4-
est trailing..
tesuni aoc.
Specks
Specks
Spacis
Oppicle
Figure A.6: Accuracy curves of PGD-AT under different noise rates on CIFAR-10.
1.6
0.2
0.2
0.3
160
159
300
16
150
150
150
100
150
330
100 Specks
150
1001
150
Spects
Spedks
Specks
Specckr
Figure A.7: Accuracy curves of TRADES under different noise rates CIFAR-10.
1.3 EXPLICIT REGULARIZATION
We consider three common regularizers, including data augmentation, weight decay, and dropout (Srivastava et al., 2014). We train the models based TRADES on true and random labels with several combinations of explicit regularizers. As shown in Table A, , the explicit regularizations do not significantly affect the model's 's ability to memorize adversarial examples with random labels.
A.2 MORE RESULTS ON THE CONVERGENCE OF AT
A.2.1 TRAINING CONFIGURATIONS
We study different training configurations on PGD-AT with random labels. We consider various factors follows. These experiments conducted on CIFAR-10.
Model capacity. Recent work suggests that model size is a critical factor to obtain better robustness (Madry et al., 2018; Xie & Yuille, 2020). A possible reason why PGD-AT fails to converge with random labels may also be the insufficient model capacity. Therefore, we try to larger models, including WRN-34-20 (which is used in Rice et al. (2020)) and WRN-70-16 (which is used in Gowal et al. (2020)). However, using larger models under this setting cannot solve the convergence problem.
18

---

Published as a conference paper at ICLR 2022
Table A.I: 1: The training accuracy. accuracy, and generalization gap (%) TRADES when trained true random labels, with without explicit regularizations, including data augmentation (random crop and flip). weight decay (0.0002), and dropout (0.2).
Data Augmentation
Weight Decay
Training Accuracy Robust
Test Accuracy Robust
Generalization Gap
Labels
Dropout
Natural
Natural
Natural
Robust
X
X
99.73
99.65
77.53
37.47
22.20
62.18
true
X
99.57
97.03
82.91
45.37
16.93
51.66
true
X
/
X
99.59
99.53
77.31
38.94
22.28
60.59
true
X
X
v
99.65
99.40
79.96
39.86
19.69
59.54
V
X
99.50
97.28
84.26
49.16
15.24
48.12
true
V
99.41
99.20
80.28
41.64
19.13
57.56
random
X
x 99.80
99.55
9.79
0.15
90.01
99.40
random
X
99.36
86.10
9.71
0.24
89.65
85.86
X
X
random
99.84
99.53
10.13
0.23
89.71
99.30
random
99.15
92.23
9,04
0.17
90.11
92.06
random random
X
99.25
69.62
9.67
0.24
89,58
69.38
99.38
81.57
9.54
0.19
89.84
81.38
Attack steps. We adopt the weaker FGSM adversary (Goodfellow al., 2015) for training. We also adopt the random initialization trick argued in Wong al. (2020) and adjust the step size as a = 255, yielding the fast adversarial training method (Wong et al., 2020). However, fast AT still cannot converge.
Optimizer. We try to use various optimizers, including the SGD momentum optimizer, the Adam optimizer (Kingma & Ba, 2015), and the nesterov optimizer (Nesterov, 1983); different learning rate schedules, including the piecewise decay and cosine schedules, and different learning rates (0.1 and 0.01), but none of these attempts make PGD-AT converge.
Perturbation budget. The perturbation budget is important factor to affect the conver- gence PGD-AT. When approaches 0, PGD-AT would degenerate into standard training, which can easily converge (Zhang al., 2017). Hence try different values of c, and find that PGD-AT can converge with a smaller (e.g., 1/255) but cannot converge when 2/255.
A.2.2 GRADIENT STABILITY UNDER COSINE SIMILARITY
In Fig. 4(a), we show the gradient change of PGD-AT, TRADES, and the clean CE loss under the & distance. We further show the cosine similarity between the gradients at 0 and 0 + Xd in Fig. A.8. The cosine similarity is also averaged over all data samples. The results based on cosine similarity consistent with the results based on the Eg distance, showing that the gradient of PGD-AT changes more abruptly.
A.2.3 THE FAILURES OF AT UNDER REALISTIC SETTINGS
We find that some AT methods (e.g., PGD-AT) suffer from a gradient instability issue, which re- sults the convergence problem when trained random labels. Under other realistic setting, our analysis may also be valuable.
First, PGD-AT fails converge under 80% noise rate. We think that the unstable gradients can overwhelm the useful gradients given by clean examples. To prove it, we train the models under 80% uniform label noise, by either PGD-AT or standard training (ST) on natural examples. We then select 100 training images with wrong labels and another 100 training images with true labels for evaluation. Similarly, we calculate the gradient norm of the cross-entropy loss w.r.t. model parameters of each method. We show the results in Fig. A.9. For AT, the gradient norm of clean examples is larger than that of noisy examples at beginning, which makes the model learn to classify. However, for PGD-AT, the gradient norm of clean examples is almost the same as that of noisy examples (the two curves overlap together). And the unstable gradients provided by noisy examples would overwhelm the useful gradients given by clean examples, making the network fail to converge.
19

---

Published as a conference paper at ICLR 2022
10a
1.0
PGD-AT
PGD (clean)
TRADES
PGD AT (noisy)
Cosine Similarity
0.9
CE loss
10a
Standard (clean)
Gradient Norm
Standard (noisy)
0.8
10
0.7
www
108
0.6
10a
-0.050
-0.025
0.000
0.025
0.050
0
25
50
75 Epochs
100
125
150
Figure A.8: The cosine similarity between the gradi- ents 0 and 0 + Ad of different losses, where O are initialized, À [-0.05, 0.05].
Figure A.9: The gradient norm of PGD-AT
ST given clean examples or noisy examples, when trained on 80% uniform label noise.
Second, when the perturbation budget is large (e.g., 6 625),, PGD-AT cannot converge with true labels, while TRADES can achieve about 50% training accuracies. This can also be explained by our convergence analysis that the gradient is very unstable PGD-AT with a larger perturbation budget, making it fail to converge.
A.3 MORE DISCUSSIONS THE GENERALIZATION AT
As shown in Table A. when trained on true labels, although the regularizers can help to reduce the generalization gap, the model without any regularization can still generalize non- non-trivially. The three explicit regularizations do not significantly affect the model's ability to memorize adversarial exam- ples with random labels. In consequence, the explicit regularizers are not the adequate explanation of generalization. By inspecting the learning dynamics of AT under different noise rates in Fig. A.6 and Fig. A.7, the network achieves maximum accuracy on the test set before fitting the noisy train- ing set, meaning that the model learns simple patterns (i.e., clean data) before memorizing the hard examples with wrong labels, similar to the observation in standard training (Arpit et al., 2017). The results suggest that optimization by itself serves implicit regularizer find model with good generalization performance.
rand init; natural acc. robust adv init; natural acc. init; robust acc.
0.8
hu
Accuracy
0.6
Accuracy
0.4
natural
0.2
rand init; robust acc.
adv init; natural acc.
adv init; robust
0.0
0.0
50
100
150
200
0
100
200
Lipochs
Epochs
(a) Vanilla SGD
(b) SOTA SGD
Figure A.10: The natural and robust testing accuracies of TRADES on CIFAR-10 with different initialization strategies and training methods.
recent work (Liu et al., 2020b) points out that in standard training, pre-training on random labels can lead to substantial performance degeneration of subsequent SGD training true labels, while adding regularizations can overcome the bad initialization caused by pre-training with random la- bels. In this paper, further investigate whether this finding can generalize adversarial training.
20

---

Published as a conference paper at ICLR 2022
As PGD-AT cannot converge with random labels, adopt TRADES conduct experiments. Fol- lowing Liu et al. (2020b), we consider two initialization strategies - random initialization and adversarial initialization generated by training random labeling of the training data. We also sider two training methods vanilla SGD training and SOTA SGD training with data augmentation (random crops and flips), weight decay, and momentum. The results be seen that with vanilla SGD, the adversarial initialization can lead to worse performance than the random initialization. But with the regularization techniques, the models with different initializa- tions converge to nearly the same test accuracy. The results are consistent with the findings in Liu al. (2020b).
shown in Fig. A.10. It can
B PROOF OF THEOREM
Proof. Recall that [x. y. 0) [sx) ((f((x)),))))))))))))))))()))))) y) the adversarial loss of PGD-AT. First, we have
V y, 01 VaJ(x, 02)l12
VoJ y, 01) (x),,
Vx + (x),
VoL(Jo, (x), (x)
(B.1)
VoJ y, (..0) - (x),
VoJ y, 020
[oo((o, (x),
VeL(Jo, (x), y)ll.
From the assumption, for any E R2d and x' E S(x), have
[Vo[(fo(x') (o((fo(x), 8)1l K||x - x11p S cK,
due to the definition of S(x). We also note that J(x, y, 0) is the maximal cross-entropy loss L. within S(x), such that we have
[xo v. VoL(fo(x) cK.
(B.2)
Combining Eq. (B.1) and Eq. (B.2), we can obtain Eq. (5).
Note that the bound is tight since the all the equalities can be reached.
Remark 1. We note that recent work (Liu et al., 2020a) gives similar result gradient stability. The difference is that they assume the loss function satisfies an additional Lipschitzian smoothness condition
[Ve[(fe, (x), v) (e](fe((x). 8)ll2 I Koll0 -021l2.
where Kg is another constant. Then they prove that
[V, y. VoJ y, 02)l12 S Ko 01 02112 + 2cK.
can be noted that with this new assumption, can simply obtain this result by Theorem 1. Therefore, Theorem is a more general result of the previous one.
B. THEORETICAL ANALYSIS FOR TRADES
Note that TRADES adopts the KL divergence in adversarial loss. The KL divergence is defined on two predicted probability distributions over all classes, as
fo(x)y log fo(x), fe (x())
D(fo(x)llfo(()) = Σ
yEfI....C)
However, based on the local Lipschitz continuity assumption of the clean cross-entropy loss (which only concerned with the predicted probability of the true class) in Eq. (4), we cannot derive similar theoretical bound on the gradient stability TRADES as in Eq. (5). Therefore, we need to make different assumption the KL divergence. For example, suppose the gradient of the KL divergence satisfies
D(oo
K'| x||p>
21

---

Published as a conference paper at ICLR 2022
for any E Rd x' S(x), and any 0, where K' is another constant. We denote the adversarial loss of TRADES as J'(x, (x, y, 0), then we have
IooJ v, (x 01)1lz
0 (fo, (x), VoL(fo, (x), v)1l2+
V
max xES(x)
Difo, (x) (x2)
max B
Dife, (x)e,x)]lfo, (x)))l2
xES(X)
VeL(o (x), - V (x](fo(()) w)ll2 + 23eK'.
Although we can derive similar bound on gradient stability of TRADES, this bound is not directly comparable to Eq. (5) since we cannot find the relationship between K and K" However, our empirical analysis gradient magnitude in Sec. 3.2 has shown that TRADES is dominated by the clean cross-entropy loss at the initial training epochs, thus the gradient stability the TRADES loss will be similar to that of the clean cross-entropy loss, also revealed in Fig. 4(a). Therefore, the gradient of TRADES would be relatively stable.
C FULL EXPERIMENTS ON ROBUST OVERFITTING
C.1 ADDITIONAL EXPERIMENTS ON EXPLAINING ROBUST OVERATTING
1.0
4.0
First Model
Second Model
Robust Test Accuracy
Adversurial Loss
3.0
0.6
2.0
1/255
2/255
1.0
0.2
4/255
8/255
0.0
0.0
100
150
200
100
200
300
400
500
Epochs
Sample
Figure C.1: The robust test accuracy of TRADES der various perturbation budgets e.
Figure C.2: The adversarial loss of WRN-28 28-228-10 10 and ResNet-18 trained by PGD-AT 500 samples sorted by the the first model (i.e., WRN-28-10).
First, we show the robust test accuracy curves TRADES under various perturbation budgets in Fig. C.I. It can also be observed that when the perturbation budget is small, robust overfitting does occur.
Second, we show that the "hard" training examples with higher adversarial loss values are consistent across different model architectures. We train one WRN-28-11 10 model and one ResNet- 18 model based PGD-AT. We then calculate the adversarial loss for each training sample for these two models. We show the adversarial loss on 500 samples sorted by the loss of the first model (i.e., WRN- 28- 10) in Fig. C.2. It can be seen that the samples with lower adversarial losses of the first model also have relatively lower losses of the second one and vice versa. The Kendall's rank coefficient of the adversarial loss between the two models is 0.78 in this case.
Third, we visualize the hard training examples with high adversarial loss values in Fig. C.3. It can be seen that these examples difficult to recognize and their labels may be wrong. Therefore, the one-hot labels for these hard training examples can be noisy for AT, leading to the robust overfitting problem.
22

---

Published as a conference paper at ICLR 2022
Bird
Bird
Truck
Deer
Horse
Cat
Deer
Natural Examples
Adversarial
Examples
Figure C.3: The hard training examples with high adversarial loss values.
Table C.I: Test accuracy (%) of several methods using different model architectures and threat models. We
choose the best checkpoint according the highest robust accuracy on the test set under PGD-I0.
Natural Accuracy Final
PGD-10
Methods
Networks
Norms
Best
Diff
Best
Final
Diff'
PGD-AT
WRN-34-10
86.58
86.83
-0.25
55.83
49.52
6.31
PGD-AT+TE
WRN-34-10
85.43
85.10
0.33
59.30
56.63
2.67
Eoo (c 8/ 8/255)
PGD-AT
VGG-16
79.60
81.26
48.52
43.02
5.50
PGD-AT+TE
VGG-16
78.19
79.13
-0.94
52.06
51.29
0.77
PGD-AT
88.82
88.96
-0.14
69.05
65.96
3.09
PGD-AT+TE
87.95
88.20
-0.25
72.58
71.93
0.65
R8
(e 128/255)
TRADES
86.50
86.57
-0.07
70.22
66.07
4.15
TRADES+TE
88.42
88.60
-0.18
72.72
72.43
0.29
C.2 ADDITIONAL EXPERIMENTS ON MITIGATING ROBUST OVERFITTING
We show the results of our proposed methods on other network architectures (including WRN- 34-10 and VGG-16) and threat models (including & norm) in Table C.I. The results consistently demonstrate the effectiveness the proposed method.
We further show the results of PGD-AT, PGD-AT+TE, TRADES, and TRADES+TE on CIFAR-10 over runs in Table C.2.
23

---

Published as a conference paper at ICLR 2022
Table C.2: Test accuracy (%) of several methods on CIFAR-I0 under the too norm with 8/255 based on the ResNet- architecture. We show the mean/std of the results over 3 runs.
Natural Accuracy Final
Method
Best
Diff'
PGD-AT
83.76 0.02
84.93 = 0.26
-1.17 0.29 -0.33 0.31
PGD-AT+TE
82.36 0.18
82.69 = 0.14
TRADES
81.34 0.15
82.70 = 0.21
-1.36 0.36
TRADES+TE
83.66 + 0.19
83.89 = 0.09
-0.23 0.21
PGD-10 10
Method
Best
Final
Diff
PGD-AT
$2.62 0.10
44.91 = 0.01
7.71 + 0.11
PGD-AT+TE
55.74 + 0.17
54.82 = 0.23
0.92 = 0.07
TRADES
50.48 0.23
2.77 0.16
TRADES+TE
54.93 + 0.16
54.04 0.19
0.89 0.13
PGD-1000
Method
Best
Final
Diff'
PGD-AT
51.26
0.03
42.72
0.06
8.54 - 0.06
PGD-AT+TE
54.54 0.27
53.01
0.34
1.53
0.24
TRADES
52.24
0.20
48.74
0.17
3.50 - 0.13
TRADES+TE
53.55 1 0.16
0.07 52.93
0.62 = 0.11
C&W-1000
Method
Best
Final
Diff
PGD-AT
50.24
0.12
43.59 = 0.07
6.65
0.19
PGD-AT+TE
52.31 + 0.01
51.67 0.12
0.64
0.11
TRADES
49.83 0.05
48.11 0.04
1.72
0.02
TRADES+TE
50.80 0.02
50.61 0.07
0.08
AutoAttack
Method
Best
Final
Diff
PGD-AT
47.85
0.17
41.62 0.16
6.23 0.26
PGD-AT+TE
50.37 0.22
49.36 = 0.24
1.01 = 0.03
TRADES
48.86 + 0.18
46.73 = 0.07
2.13 = 0.11
TRADES+TE
49.40 1 0.27
48.77
0.21
0.63 - 0.05
24


Reviews:
[{"review_id": "Eh3OwFJAbQL", "reviewer": "Reviewer_x9Ej", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper presents several interesting insights about the behavior of adversarial training algorithms. And the proposed methods are shown to out-perform PGD-AT and TRADES. However, the findings in the paper are disconnected and some of them lack explanations. Therefore, it is hard to fully understand the importance of these findings. Moreover, the proposed method, the Temporal Ensembling (TE) approach, does not have significant novelty compared to previous methods. Next, we detail our concerns.\n\n1. In Section 3.1 and Section 3.2, the authors make great efforts to compare the difference between algorithmic stability of PGD-Train and TRADES, when 100% training labels are flipped. However, the stability difference will not result in any impact to the models, under the cases if fewer labels (0-80%) are flipped (Figure 2). Therefore, it is hard to justify the contribution of this analysis, if this difference is always oblivious under general scenarios.\n2. In Section 3.3, the authors also mention that currently existing complexity measures cannot adequately explain the robust generalization performance of AT. What is the reason for the gap between theories and the practices?\n3. Moreover, in Section 4.1, the authors deploy experiments to demonstrate that there are “hard” training samples which are consistently hard to be classified. However, why the existence of hard training examples can support the claim they can result in overfitting issues? What if removing these “hard” / “noisy” samples from the training set, will this resolve the overfitting problem?\n4. For the proposed method, the Temporal Ensembling (TE) approach, it is not very different from an existing work [1]. Although the Equ.(7) in this paper and Algorithm 1 in [1] only have minor differences, both the papers are designed to (adversarially) train one model, with a self-adapted label instead of the one-hot labels. Furthermore, these two papers have the same way to generate the self-adapted labels. The only difference is the training objective. However, they also have high similarity in terms of definitions and experimental effects.\n \n**References**:\n[1] Huang, Lang, Chao Zhang, and Hongyang Zhang. \"Self-adaptive training: beyond empirical risk minimization.\" Advances in Neural Information Processing Systems 33 (2020).\n", "summary_of_the_paper": "This paper provides comprehensive studies on several facts about the memorization of adversarial training algorithms (AT). Specifically, the authors uncover two findings: (1) Given a sufficient model capacity, PGD Adversarial Training cannot fit the whole training set (with 100% labels flipped). While, TRADES can fit the whole random flipped set. It may be because the objective function of TRADES has a clean loss, which can improve the stability of training. (2) Memorization of one-hot labels in AT methods can be one important factor to the robust overfitting issues. Based on these findings, the authors propose a method Temporal Ensembling (TE) approach to avoid fitting all adversarial examples with one-hot labels. ", "summary_of_the_review": "Overall, the findings of this paper lack some clarifications about their significance. Moreover, the proposed adversarial training method does not have enough novelty beyond currently existing methods. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5CCYY8saIr0", "reviewer": "Reviewer_c7wV", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n1. **Clean methodology and in-depth analysis**: I admire the effort made by the authors to provide a very in-depth analysis of their results, thoroughly checking different metrics, datatasets, and training setups, and trying to identify the root cause of all the described phenomena.\n2. **Relevance of the results**: The fact that PGD-AT cannot memorize random labels due to its gradient instability, while TRADES or a warmed-up version of PGD-AT can, is quite significant. It points towards the importance of the early learning phase for adversarial training and it can spark new research avenues to understand the dynamics of adversarial training. Similarly, the fact that different complexity measures proposed in the literature to explain generalization do not correlate, and sometimes anti-correlate, with actual performance, might have been extended from prior results on standard training, but it is still of great relevance.\n3. **Connection between robust overfitting and label noise**: The suggested cause for robust overfitting, i.e., introduction of label noise by the adversarial attacks, is bold, and the proposed method to overcome it seems quite effective.\n\n## Weaknesses\n1. **Theorem 1 does not say anything about stability of TRADES**: Although Theorem 1 strengthens the claim made by the authors explaining why PGD-AT has a greater gradient instability than standard training, it does not say anything on how TRADES can overcome it. As far as I understand it, a similar result might be true for the TRADES regularization term. This slightly weakens the theoretical arguments of the authors.\n2. **Connection between robust overfitting and label noise could be more compelling**: As I said, I appreciate the bold claim made by the authors to explain the phenomenon of robust overfitting, and its connections to memorization in AT. However, I still believe some of their arguments are a bit speculative and could be better supported by the data. In particular, the fact that models share the ranking of difficulty of different examples is, in my opinion, a rather indirect way to measure the presence of label noise in a dataset. Something that would make the claim of the authors more convincing is to show that the \"hard\" instances they identified in the dataset do correspond to incorrectly classified or very noisy samples.\n3. **Too many details off-loaded to appendix**: In my opinion, the authors postpone too many important discussions to the appendix which hurts the readability of the paper. Specifically, I believe that Fig. A8 and Appendix A2 should belong to the main body of text.\n\n## Other comments\n1. **Fine-tuning experiments**: I am of the opinion that the paper, in its current form, deserves to be accepted to the conference. However, I still believe there are certain experiments which, if included in the paper, would greatly strengthen the relevance of this work, and could lead me to increase my score. Specifically, there are two important works in the literature which study the effect of pre-training with random labels in performance (Liu et al. 2020, Maennel et al. 2020) and whose transferrability to the adversarial setting is currently unknown. Considering the fact that some of the experiments in this work point towards the importance of the early stages in training in the memorization behavior of AT, I wonder how different pre-training schemes with clean and random labels affect the findings of the authors. Namely, does temporal ensembling work better when the network is standardly trained on clean labels? Do neural networks robustly generalize when adversarially pretrained on random labels? Do they learn meaningful representations? I truly believe that answering any of these questions would be of great significance to the community, and therefore would make this work much more relevant.\n   - Shengchao Liu, Dimitris Papailiopoulos, Dimitris Achlioptas. \"Bad Global Minima Exist and SGD Can Reach Them.\" NeurIPS 2020\n   - Hartmut Maennel, Ibrahim Alabdulmohsin, Ilya Tolstikhin, Robert J. N. Baldock, Olivier Bousquet, Sylvain Gelly, Daniel Keysers. \"What Do Neural Networks Learn When Trained With Random Labels?\". NeurIPS 2020\n2. **Inconsistent x-axis in Fig. 4**: Fig.4a and Fig. 4b are technically showing complementary views of the instability of gradients in the PGD loss landscape. However, the x-axis in this plot is not consistent. One of these plots shows the change of gradient around a specific weight location, and the other the change of gradient during training. This looks slightly suspicious (although I feel it is minor) and I would encourage the authors to either use the same metric in both plots, or show the analogous result for the other x-axis in the appendix.\n\n", "summary_of_the_paper": "This paper presents a thorough investigation of the dynamics of adversarial training (AT) when a network is trained on a dataset with random labels. This methodology was very successful at identifying theoretical gaps in our understanding of neural networks for standard training, so here the authors propose to replicate those experiments in the context of adversarial robustness. Surprisingly, the authors find that PGD-AT is incapable of fitting random labels when trained from scratch, while TRADES or a warmed-start version of PGD-AT can. They also suggest plausible explanations for this behavior. Finally, the authors use their novel findings to explore the reasons behind robust overfitting, and propose a new method to prevent it.\n", "summary_of_the_review": "This paper presents a very rigorous empirical investigation of the dynamics of memorization in adversarial training. The provided insights are novel, properly isolated and clearly explained. Besides, the results are not only insightful from a theoretical perspective, but they also provide actionable insights. The inclusion of a few additional experiments could strengthen more the paper, but in the current form, I believe the paper is good enough for acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "3pGzHbGU45-", "reviewer": "Reviewer_erPY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\nThis paper is easy to follow and well written and the hypothesis regarding the difference between PGD and TRADES is well analyzed by separating the gradient of each loss to provide an easy-to-understand analysis. To validate its observation, the authors designed a set of comprehensive experiments for evaluation and comparison purposes. This paper also proposes a new hypothesis about the cause of robust overfitting, which can be intuitively understood and supported by comprehensive experiments.  Also in section 5, It is experimentally well designed and confirmed that the proposed method by the regularization term using the temporal ensembling approach alleviates robust overfitting, and showed better robust accuracy than the existing methods.\n\nWeakness\n1) Complexity measures : In Figure 5, the authors show that the previous complexity measures cannot adequately explain and ensure the robust generalization performance in AT. However, since there are few model plots shown in the graph, there seems to be insufficient justification that the previous approaches are not suitable. The spectral norm and flatness-based measure in Fig. 5 may be able to explain the generalization gap if the scale is different and more model plots are expressed. It would be nice to alleviate the expression that all these measures are inadequate. It would be better if more abundant model settings and experimental results were presented. Moreover, there is a lack of explanation of why previous measures are not adequate in this setting and it would be better if the authors could clearly suggest a new alternative.\n2) Robust overfitting : Although it has been shown empirically that overfitting is improved through the Temporal Ensembling (TE) approach, there is a lack of novelty for the method itself. The proof of hypothesis by Fig 6 leaves little doubt, since the phenomenon in Fig 6(b) is natural because robust overfitting will not occur if we do not use AT. Also in Fig 6(c), the phenomenon can be due to adversarial transferability[1].\n3) Overall, It would be nice if all experiments were configured in mean/std format in various initial parameters (e.g., random seed) of the model.\n4) In section 4.2, authors claimed that most of the approaches for noisy labels are not suitable for AT, since excluding the noisy label leads to the reduction of the training data. However, these approaches already showed good performance with small datasets like MNIST and CIFAR. It is hard to understand why this suddenly became a problem, even all natural examples can be treated as ‘clean examples’ in AT. If the lack of training data is a main problem, will it be automatically relieved when we use a bigger dataset, like ImageNet? \n\n5)  An adversarial example is a variable of model parameters, and it is continuously changed as training proceeds. Is it appropriate to apply a Temporal Ensemble to this problem, which assumes that the training data is unchanged and takes a moving average of prediction for several epochs? \n\n6) In Theorem 1, the authors claimed that the upper bound of the inequality is tight, because the equality can be reached. However, in order for the inequality to be equal, four gradients - adversarial example with theta 1, clean example with theta 1, clean example with theta 2, adversarial example with theta 2 - must exist in a straight line within a millions of dimensions of space. And the equality of Lipschitz constraints should be reached for both points. It is hard to believe that this can be occured in a real experiment. Furthermore, even assuming that the equality can be reached, it does not provide any information about the magnitude of the expected gap in the inequality. Experimental evidence is needed to continue to assert this. \n\n7) Apart from that, the authors claimed that the right side of the inequality is large due to the Lipschitz constant K. But the Lipschitz continuity of the norm of the gradient is not a widely used concept, and the typical value of Lipschitz constant K in deep neural networks may not be common sense. It would be better if there is a justification about why 2*epsilon*K is (obviously) bigger than the difference between the gradients.   \n\n[1] Adversarial Examples Are Not Bugs, They Are Features (https://arxiv.org/abs/1905.02175)\n", "summary_of_the_paper": "This paper presents the memorization effect/behavior in adversarial training in perspective of model capacity, convergence and generalization. They first investigate the difference of memorization behavior between two adversarial training methods, PGD-AT and TRADES, with random labels and provide the proof of gradient instability for PGD-AT in terms of Lipschitz constant. Also they suggest that previous complexity measures are inadequate to explain robust generalization. Lastly, they investigate the cause of robust overfitting in AT and propose a mitigation algorithm by a regularization term for avoiding the excessive memorization of adversarial examples.", "summary_of_the_review": "This paper is easy to follow and well written and the hypothesis regarding the difference between PGD and TRADES is well analyzed. They present the problem of memorization in AT by robust overfitting and propose the regularization term by temporal ensembling approach with their hypothesis, also show the effectiveness of the proposed method in systematic experiments.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0ntQT9Q-jrS", "reviewer": "Reviewer_g5ke", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Abstract and introduction:**\n\nThe abstract and introduction are very well written and clear. This is a big strength. Figure 1 is also very clear.\n\n**Section 3.1:**\n\nThe fact that PGD-AT fails to converge and differs so much from TRADES is not intuitive and a great contribution. The fact that this happens over several datasets, architectures, and threat models does indeed imply that the adversarial training algorithm has a significant impact on the convergence and not the capacity of the network.\n\n**Section 3.2:**\n\nThis section is very intuitive and it makes sense that the authors consider the gradient norm and stability. It makes sense that the instability of the gradient would cause PGD-AT to fail to converge, However, even though the upper bound in Theorem 1 is tight, it can still be far off for specific values of the inputs. So the theoretic justification that the gradient of the PGD-AT loss will vary drastically based on this upper bound is not very strong.\n\n**Section 4.1:**\n\nThe author’s hypothesis that memorization of one-hot labels causes robust overfitting is convincing and they provide evidence of this. This is a strength. \n\n**Experiments:** \n\nOverall one of the biggest strengths of this paper is the thoroughness of the experiments. The authors did a great job of providing a significant amount of evidence for their claims. \n", "summary_of_the_paper": "Memorization has been investigated in deep neural network classifiers but not that much in adversarial training. The authors investigate the ability of adversarial training to memorize random datasets. They use PGD-AT and TRADES to do the adversarial training and see different convergence properties, which differs significantly from training on true labels. They also discover that robust overfitting can occur in adversarial training with memorization and use temporal ensembling to mitigate it.", "summary_of_the_review": "Overall, the paper is very well written, what the authors do is clear, makes sense, and is supported by a significant amount of data. There are really few weaknesses with the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper demonstrates that deep networks can memorize adversarial examples of training data with completely random labels, which motivates some analyses on the convergence and generalization of adversarial training (AT). The authors identify a significant drawback of memorization in AT that could result in robust overfitting and propose a new algorithm to mitigate this drawback. Experiments on benchmark datasets validate the effectiveness of the proposed algorithm. One of the reviewers is concerned about (1) the validity of stability analysis where 80% of the data labels are noisy, and the perturbation (64/255) is large, (2) the gap between theory and practice, and (3) novelty. The authors have made a great effort to address these concerns. Although there is still no consensus after the author's response, the majority of the reviewers are in strong support. I, therefore, recommend acceptance.
Decision: Accept (Poster)