Towards the Memorization Effect of Neural Networks in Adversarial Training
Authors: Han Xu
Xiaorui Liu
Wentao Wang
Wenbiao Ding
Zhongqin Wu
Zitao Liu
Anil Jain
Jiliang Tang
Venue: ICLR 2022 Submitted (2022)
Keywords: Adversarial training
Robustness
Overfitting
Neural networks

Abstract: Recent studies suggest that "memorization" is one important factor for overparameterized deep neural networks (DNNs) to achieve optimal performance. Specifically, the perfectly fitted DNNs can memorize the labels of many atypical samples, generalize their memorization to correctly classify test atypical samples and enjoy better test performance. While, DNNs which are optimized via adversarial training algorithms can also achieve perfect training performance by memorizing the labels of atypical samples, as well as the adversarially perturbed atypical samples. However, adversarially trained models always suffer from poor generalization, with both relatively low clean accuracy and robustness on the test set. In this work, we study the effect of memorization in adversarial trained DNNs and disclose two important findings: (a) Memorizing atypical samples is only effective to improve DNN's accuracy on clean atypical samples, but hardly improve their adversarial robustness and (b) Memorizing certain atypical samples will even hurt the DNN's performance on typical samples. Based on these two findings, we propose Benign Adversarial Training (BAT) which can facilitate adversarial training to avoid fitting "harmful" atypical samples and fit as more "benign" atypical samples as possible. In our experiments, we validate the effectiveness of BAT, and show it can achieve better clean accuracy vs. robustness trade-off than baseline methods, in benchmark datasets such as CIFAR100 and Tiny ImageNet.

Under review as a conference paper at ICLR 2022
TOWARDS THE MEMORIZATION EFFECT OF NEURAL NETWORKS IN ADVERSARIAL TRAINING
Anonymous authors Paper under double-blind review
ABSTRACT
Recent studies suggest that "memorization" is one necessary factor for overparame- terized deep neural networks (DNNs) to achieve optimal performance. Specifically, the perfectly fitted DNNs can memorize the labels of many atypical samples, gen- eralize their memorization to correctly classify test atypical samples and enjoy better test performance. While, DNNs which optimized via adversarial training algorithms can also achieve perfect training performance by memorizing the labels of atypical samples, well as the adversarially perturbed atypical samples. How- ever, adversarially trained models always suffer from poor generalization, with both relatively low clean accuracy and robustness on the test set. In this work, we study the effect of memorization in adversarial trained DNNs and disclose two important findings: (a) Memorizing atypical samples is only effective to improve DNN's accuracy on clean atypical samples, but hardly improve their adversarial robustness and (b) Memorizing certain atypical samples will even hurt the DNN's performance on typical samples. Based on these two findings, we propose Benign Adversarial Training (BAT) which can facilitate adversarial training to avoid fitting "harmful" atypical samples and fit as more "benign" atypical samples as possible. our experiments, we validate the effectiveness of BAT, and show that it can achieve better clean accuracy vs. robustness trade-off than baseline methods, in benchmark datasets such as CIFARI00 and Tiny ImageNet.
INTRODUCTION
is evident from recent studies that the memorization effect (or benign overfitting) (Feldman 2020
Feldman & Zhang 2020: Bartlett et al., 2020 Chatterji & Long 2020 Muthukumar et al. 2020
is one necessary factor for overparametrized deep neural networks (DNNs) to achieve the close-to- optimal generalization error. From the empirical perspective, the works (Feldman 2020, Feldman & Zhang 2020) suggest that modern benchmark datasets, such as CIFAR 10, CIFAR R00 (Krizhevsky al. 2009) and ImageNet (Krizhevsky et al. (012). always have very diverse data distributions, especially containing a large fraction of "atypical" samples. These atypical samples are both visually and statistically very different from other samples their labeled class. For example, the images in the class "bird" may have a variety of sub- sub-populations or species, with many (typical) samples in the main sub- sub-popula coob-ooulttiin and other (atypical) samples in Jess-frequent and distinct sub- -obpouuattiios. Since these atypical samples are deviated from the main sub-population, DNNs can only fit these atypical samples by "memorizing" their labels. While, memorizing/ fitting these atypical samples will not hurt the model performance on typical samples, but can boost DNNs' accuracy by correctly classifying the atypical samples appearing in the test set.
Similar to the classification models which are trained via empirical risk minimization (ERM) algo- rithms, adversarial training methods (Madry et al.] [017]] Kurakin et al., 2016) are also devised to fit the whole training dataset. Specifically, ad veersariaal training minimizes the model's error against adversarial perturbations (Goodfellow et al.) 2014, Szegedy et al.) 2013) by fitting the model on manually generated adversarial examples of all training data. Although adversarial training and memorize all training data as well as their adversarially perturbed counterparts, they always suffer from both poor clean accuracy and adversarial accuracy (or robustness on the test set (sippaa et al.
fit.
!
'Clean Accuracy: Model's accuracy on unperturbed samples. Adversarial accuracy (robustness): Model's accuracy on adversarially perturbed samples. Without loss of generality, this paper discusses the adversarial accuracy under -8/25/ PGD attack (Madry et al. 2017)

---

Under review as a conference paper at ICLR 2022
2018 Schmidt et al. 2018). Recent study (Rice al. 2020 indicates that, during the adversarial training process, one model's test adversarial accuracy will even keep decreasing as it hits higher training adversarial accuracy (during the fine-tuning epochs). Thus, it is natural to ask question: What is the effect of memorization in adversarial training? In particular, can memorizing the atypical samples and their adversarial counterparts benefit the model's 's accuracy and robustness?
To answer this question, we first conduct preliminary studies to explore whether memorizing atypical samples in adversarial training can benefit the DNNs' test performance, especially on those test atypical samples. In Section 3.1 we implement PGD adversarial training (Madry et al. (017)) on CIFARI00 under WideResNNtt models (He et al. 2016) and fine-tune them until achieving the optimal training performance. From the results in Section 3.I we observe that the memorization in adversarial training can only benefit the clean accuracy of test atypical samples. When the DNNs gradually fit/memorize more atypical samples, they can finally achieve fair clean accuracy close to 40% on the test atypical set. However, the adversarial accuracy on test atypical set is constantly low (~ 10%) during the whole training process, even though the models can fit almost all atypical (adversarial) training samples. Based on the theoretical study (Schmidt et al. 2018), the adversarial robustmess is hard to generalize especially when the training data size is limited. Since every single atypical sample is distinct from the main aob-population and rarely appears in the training set, the data complexity for each specific atypical sample is very low. Thus, its adversarial robustness can be extremely hard to generalize. Notably, for datasets such as CIFAR100 the entire atypical set covers least 40% samples of the whole dataset. Therefore, completely failing on atypical samples could be one of the key reasons leading to the poor robustness generalization of DNNs.
Furthermore, we find that in adversarial training, fitting atypical samples will even hurt DNNs' performance on those "fypical" samples (the samples in the main sub-population). In the Section 3.2 we again implement PGD adversarial training (Madry et al., (217)] on CIFAR100 for several trails. which are trained with different amount of atypical samples. Based on the results from Section[.22 an adversarially trained model on the training set without any atypical samples has 95% clean accuracy and 55% adversarial accuracy on the test typical set. While, the model trained with 100% atypical samples only has 90.2%/50.44% clcan/adversarial accuracy respectively. In other words, atypical samples act more like "poisoming data" " (Biggio al. 2012) to deteriorate model performance on typical samples. Furthermore, our study in Section 3.2 also demonstrates that this poisoning effect is absent in traditional ERM, where fitting atypical samples will not reduce the model accuracy. To deepen our understanding on this finding, we build a theoretical analysis based on Gaussian mixture models. We prove that, given certain atypical samples, any model which fits its adversarial counterpart (adv. training) must have a poor accuracy on typical samples. While, under the same setting, the models only fitting the clean version of this atypical sample (traditional ERM) can achieve optimal accuracy (i.c. > 99%). Our empirical and theoretical results highlight the key difference between the effect of memorization in adversarial training and traditional ERM.
Motivated by our findings, we propose novel algorithm called Benign Adversarial Training (BAT), which can eliminate the negative influence from memorizing those "poisoning" atypical samples, meanwhile preserving the model's ability to memorize those "benign useful" atypical samples. It worth mentioning that, by fitting those "benign" atypical samples, the BAT method can achieve good clean accuracy the atypical set; by eliminating those poisoning atypical samples, the BAT method can improve the clean & adv. accuracy on the typical set. Compared with PGD adversarial training (Madry et al. 2017) when it achieves the highest adversarial robustness, BAT has higher clean accuracy as well as adversarial accuracy. Compared to other popular variants of adversarial training such as (Zhang et al. 2019 Wang et al., 2019 Zhang et al.] 2020), BAT is the only one obtaining both better (or comparable) clean and adversarial accuracy than (Madry et al. 2017). on complex datasets such as CIFAR 100 and Tiny Imagenet (Le & Yang 2015).
2
DEFINITION AND NOTATION
2.1 ATYPICAL SAMPLES AND MEMORIZATION
In this section, we start by introducing necessary concepts and definitions about the memorization effects. As well known, the practice of training deep neural networks (DNNS) cannot be well ex- plained by standard theories about model generalization (Evgeniou et al.) 2000; Bartlett & Mendelson 2002). At high level, the standard theories underline the importance of regularization the model
2

---

Under review as a conference paper at ICLR 2022
complexity, avoid "overfitting" the outliers and nonuseful samples. While, for DNNs, we always tune the model to hit almost perfect training accuracy and enjoy good test performance.
Fortunately, recent works (Feldman 2020 Feldman & Zhang 2020 Bartlett et al.) 2020: Muthuku mar et al. 2020) make significant progress to close this gap from both theoretical and empirical perspectives. They suggest that the memorization is one key property for DNNs to achieve optimal generalization performance. In detail, the works (Feldman 2020; Feldman & Zhang 2020) point that the common benchmark datasets, such as CIFARIO, I0, CIFARTOO, ImageNet, contain large portion of atypical samples (or namely, rare samples, sub-populations, etc.). These atypical samples look very different from the other samples in the main distribution of its labeled class, and are statistically indistinguished from outliers or mislabeled samples. DNNs can only fit these samples by memorizing their labels. Moreover, without memorizing these atypical samples during training, the DNNs can totally fail to predict the atypical samples in the test set (Feldman & 'Zhang 2020).
Identify Atypical Samples. To identify such atypical samples in common datasets in practice, one representative strategy (Feldman & Zhang, (2200) proposes to examine which training samples can only be fitted by memorization, and measure each training sample's "memorization value". Formally, for training algorithm A (i.e., ERM), the memorization value "mem(A, AA, D, xx,"" of training sample (x., y4) E D in training set D is defined as:
mem (A, 2.)
Pr. (F(x,) = y:) F+A(D)
(F(z;) y1), F+A(D()))
Pr.
(I)
which calculates the difference between the model Fss accuracy on o; with and without x, removed from the training set D. In practice, (Feldman & Zhang, 2020) trains DNN model using ERM method for 1,000 trials, with each trial preserves 70% samples of the whole training dataset. Based on this metric. one can identify all common datasets having a large fraction of atypical samples. For example, CIFAR1O, CIFARION 100 and Tiny Image have more than 11% 40% and 49% samples with a large memorization value > 0.15. The similar strategy can also facilitate to find atypical samples in the test set, which are the samples that are strongly influenced by atypical training samples. In detail, by removing an atypical training sample (1,,,:)), we calculate its "influence value" " on each test sample 2/) E DD' in test set D':
Pr. (F(x)) v/)) infi (n, 2() F-A(D)
Pr. F-A(D(a))
(F(x)) =11)-
If the sample pair (I;, 2²)) has high influence value, removing the atypical sample v; will drastically decrease the model's accuracy on xzj. In Appendix A we provide more detailed discussions on the memorization effect and atypical samples, containing: (a) the distribution and frequency of atypical samples in common datasets and (b) alternative (more efficient) metrics to identify atypical samples, such as confidence-basse method and ensemble disagreement (Carlini et al. 2019).
2.2 ADVERSARIAL TRAINING
Similar to classification models trained via empirical risk minimization (ERM) algorithms, adversarial training methods (Madry et al., 2017: Kurakin et al., (2066) are also devised to fit the whole dataset by training the model on manually generated adversarial examples. Formally, they are optimized to have the minimum adversarial loss:
min E F
max L(F(x ((((x(+))))) 5), [18
(3)
which is the model F's average loss on the data (I,, y) that perturbed by adversarial noise 5. These adversarial training methods have been shown to be one of the most effective approaches to improve the model robustness against adversarial attacks. Note that similar to traditional ERM, adversarially trained models can also achieve very high training performance. For example, under NeteResNet28- 10 (WRN28), PGD adversarial training (He et al.) (016) can achieve 100% clean accuracy and 99% adversarial accuracy. It suggests these DNN models have sufficient capacity to memorize the labels of these atypical samples and their adversarial counterparts. However, different from ERM, adversarially trained models usually suffer from bad generalization performance during test. For example, the WRN28 model can only have 59% and 24% test clean accuracy and adversarial accuracy. Moreover, the study (Rice et al. (020)) suggests that during the adversarial training process, the model's 's test adversarial accuracy keeps dropping as more training data is fitted (after the first time learning rate
3

---

Under review as a conference paper at ICLR 2022
decay). Thus, these facts indicate that the memorization in adversarial training is probably not always beneficial to the test performance and requires deep understanding. In the following section, we will draw key connections between these properties with the memorization effect.
3 ATYPICAL SAMPLES IN ADVERSARIAL TRAINING
In this section, we attempt to understand adversarial training's behavior by studying its relation with the memorization effect. The discussions are based on PGDD-adversaria training (Madry et al. 2017) on CIFAR 100. Implementation details and results on more datasets, i.e., CIFAR 10 and Tiny ImageNet (Le & Yang 2015) are shown in Appendix where we make similar observations.
3.1 ADVERSARIAL ROBUSTNESS OF ATYPICAL SAMPLES CAN HARDLY GENERALIZE
In this subsection, we first check whether fitting atypical samples in ad- versarial training can effectively help the model correctly and robustly pre- dict the atypical samples in the test set. We apply PGD adversarial train- ing (Madry et al. (017)) original CIFARI00 dataset for 200 epochs and evaluate the model's clean accuracy and adversarial accuracy on training atypical set Dayp = fo, D mem(x;) > 0.15] and its correspond- ing test atypical set Panyp (x', E
1.0
1.0
Tost
Tost
0.8
Traning
0.8
Traning
6.6
25
50
75
25
75
Epochs
Tpoche
Figure 1: Clean Acc. (left) & Adv. Acc (right) on Atypical Set of CIFARIO0 under WRN28
D' infl xz) ) > 0.15, for Vx, Payp]. In Figure. we report the algorithm's performance (clean & adv. acc.) on these atypical sets along with the training process. From the results, we observe that the WRN28 model capable to memorize almost all atypical samples and their adversarial counterparts, since it can achieve = 100% clean & adversarial accuracy on the training atypical set. As the training goes, the models' clean accuracy on the test atypical set gradually improves and finally approaches 41 41%. However, its adversarial robustness keeps constant around 10% from the beginning epochs to the last ones, no matter how high the training performance is. These results suggest that memorizing atypical samples in adversarial training may only improve their test clean accuracy, but hardly help their adversarial robustness. Recall that in CIFAR 100, atypical set Dayp (with memorization value 0.15) covers 40% samples of the whole dataset. Completely failing on the adversarial robustness of atypical samples could be one key reason that contributes to the poor robustness generalization of DNNs (Rice al., 2020).
As the previous theoretical study (Schmidt et al. 2018) stated, for model to achieve good robustness generalization performance, it always demands training set with much more samples, than a model to have good clean accuracy generalization. In our case, the sub- sub-population of each particular atypical sample has a very low frequency to appear in the training set, and it is always deviated from the main sub-po popullatul.... Thus, in the sub-population of this atypical sample, it is equivalent to classification task based on an extremely small dataset, with one or a few training samples given. Thus, the robustness of atypical samples can be extremely hard to generalize.
3.2 MEMORIZING ATYPICAL SAMPLES HURTS TYPICAL SAMPLES' PERFORMANCE
In this subsection, we find that fitting atypical samples will even bring negative effects on "typical" samples. Here, we define "typical" samples as the subset of training set DD which has low memorization values: Dryp = { D mem(x;) < 0.02]. For the test typical set D'y we exclude all test samples which have high influence values from any atypical training samples, and also exclude the samples that have low success rate to predict using ERM algorithm A (the samples which cannot be learned from D): D' D' (2 infi xz, 21,) > 0.02, for Vz, Dayp] U {a/s Pr.re (FD)(((())) 0i) 0.8).
We conduct PGD adversarial training (Madry et al. 2017) for several trials on resampled CIFAR datasets: each dataset is constructed with the whole training typical set Doyp and part of the training atypical set D Dayp (randomly sample 0%, 20% and 100% in Dayp)). ). In Fig, 2 we report the adversarially trained model's clean and adversarial accuracy on the test typical set D and check the
4

---

Under review as a conference paper at ICLR 2022
0.90
0.05
Appriul 207 App oul 100%
Ropeud-t00 2000 Ropica--109
ooverser ACCUUITT
0.95
0.90
0.70
0.40
Ecochs
Epochs
Eoochs
(a) Adv. Training (Clean Acc.)
(b) Adv. Training (Adv. Acc.)
(c) Traditional ERM (Clean Acc.)
Figure Adversarial Training vs. Traditional ERM on Typical Set of CIFARIO0 under WRN28
impact of atypical samples on the typical samples. From the results, we find that the existence of atypical samples makes a significant influence on the typical samples. For example, an adversarially trained model without atypical samples has 94. 7% clean accuracy and 55.0% adversarial accuracy on the test typical samples (on the last epochs). While, the model trained with all atypical samples only has 90.2% and 50.4% clean & adv. accuracy, respectively. These results suggest: the more atypical samples exist the training set, the poorer performance the model will have on D In other words, these atypical samples act like "poisoning" samples (Biggio al. 2012 Xu et al. 2019) which can deteriorate the model's performance on typical samples. It also worth mentioning that this poisoning phenomenon does not appear in the traditional ERM algorithm. From Figure 2c can see that the models have equal accuracy on the typical set, when different numbers of atypical samples appear in the training set.
Alypical
"Apple"
"Lamp"
Typical "Cup"
Figure 3: Poisoning Atypical Samples
Poisoning Atypical Samples A natural question is what kinds of atypical samples are likely to "poison" model robustness and why? Different from previous literature about poisoning samples. which assume that poisoning samples are most mis-labeled samples (Li et al., 2020), CIFAR100 is clean dataset with no or very few wrong labels. However, we hypothesize that the atypical samples which poison the model performance might pertain some features of a "wrong" class. Recall that atypical samples are always distinct from the main data distribution in their labeled class, likely that they closer to the distribution of "wrong" class. As shown in Fig. 3g in CIFAR100 dataset, we identify an atypical "plate" which visually resembles images in "apple" (the examples are found via the method in Section 4.1.. If the model memorizes this atypical "plate" and predicts any samples with similar features to be "plate", , the model cannot distinguish between "apple" and "plate".
Why does this poisoning effect only happen during adversarial training, instead of traditional ERM? To deepen our understanding of this question, we theoretically analyze the consequence of fitting "poisoning" atypical samples for both algorithms. In particular, we formally state: the poisoning effect during adversarial training can be significantly stronger than traditional ERM. Our analysis is based on binary classification task with two-dimensional Gaussian Mixture distribution:
~ (-1,+1}, 0 (1 IN {
N o0) (-
if +1
(4)
if -1,
and we consider linear models == sign(wnTr x b), where wo and are the model weight and bias. Moreover, assume there is atypical training sample xo IR2 labeled as "-1", " with Fuclidean distance to the center each class ||:0 (-0)ll2 d_1, 1| - Ø|1 = d41- Intuitively, this setting if d=1 is relatively small, since has short distance the x n for 2o to preserve certain similarity to the class "-1". The following theorem shows that there always exists model which fits the sample ro, and achieve optimal accuracy (i.e. > 99%) on both classes, although d41 can be small.
of zo resembles the cases in Figure 3
wrong class. We also constrain d_1
5

---

Under review as a conference paper at ICLR 2022
Theorem (Fitting Atypical Samples in traditional ERM) For a data distribution in Eq. 4 given that n/o a is large enough, we define I = /2. 12.4-11100 (0.99) aa (1 + Φ 10:0==1100000 (0.99) ( For point zo with d=1 1 S 2n, /+1 2 I, then there exist model F filting vo: F(xo) - -1 and has classification error at most 1% on both classes.
The detailed proof can be found in Appendix C Remarkably, for classification tasks if the two classes are well separated, i.e. o/n is sufficiently small, there exist models that can achieve high performance even when the atypical sample Ho is only around 112---1(000 of class "+1", which much smaller than the center distance of the two class 2n. This suggests that for traditional ERM, the poisoning effect of fitting the atypical sample zo is negligible, even if zo is close to the wrong class "+1"". Next, we show that under the same setting, any model which fits the adversarial counterpart of zu, must have a poor accuracy on the class "+1".
o) distant away from the center
Theorem 2 (Fitting Atypical Samples in Adversarial Training) For the same data distribution in Eq. 4 and same definition I = V2-4 (0.99)- o. (1+---1----00
3)-1/² We assume I S d+1 < k-1, where 1 S k << n/o. Then, any model F will have classification error least 50% on class "+1" 1, when F fits the adversarial example of zu! F + 6) = -1, -1,1116112 S c, and
e (k v2 4 (0.99)) (1 + + (0.999 )) 1/2 )-1//
(5)
7)
Notably, when the two classes are well separated, i.c. o/n small, the adversarial training bound is also much smaller than тj. The analysis suggests that when To is close to the class "+1", all linear models which fit the adversarial example of 2g will have a large error on class "+1", even the adversarial training bound c is small. Thus, in adversarial training, the poisoning effect of fitting atypical samples is significantly greater than that in traditional ERM. These theorems highlight the key difference between adversarial training and traditional ERM when fitting atypical samples, which motivate us to devise strategies to eliminate this poisoning effect during adversarial training.
4 BENIGN ADVERSARIAL TRAINING
Based on previous discussions, the effect of atypical samples in adversarial training can be briefly summarized as: (a) They can benefit the models' clean accuracy (especially on atypical samples), but hardly improve their robustness. (b) They hurt the performance of typical samples. In this section, we ask the question: Can we eliminate the negative influence from fitting atypical samples during adversarial training? Admittedly, strategies for adversarial training to stop at early epoch (Rice et al.) 2020) is potential to achieve this goal, since the atypical samples may be fitted the later stages of training. However, it is still unclear how to properly identify such epoch in general scenarios. Moreover, the early-stop mechanism tends to ignore all atypical samples and significantly limits the model's ability to handle atypical samples, especially complex datasets such CIFAR 100 and ImageNet with large fractions of atypical samples.
In this work, we propose novel algorithm Benign Adversarial Training (BAT). It is composed of two major components: (i) Reweighting to mitigate the impact from poisoning atypical samples and learn benign atypical samples; (ii) Discrimination Loss to further protect the discrimination ability among different classes. Combining these two components, BAT can improve the performance of typical samples, but still preserve the ability to fit those "useful" atypical samples. Compared to PGD adversarial training (Madry et al. 2017). BAT enjoys better clean vs. adversarial accuracy trade-off. More results can be found in Section Next, we detail these two components of BAT.
4.1 REWEIGHTING POISONING SCORE
Following PGD adversarial training (Madry et al. adversarial examples using Projected Gradient Descent (PGD). During the training process, in order to identify which samples can poison or degrade the model's performance, define the "poisoning score" for each training (adversarial) sample 2: as:
2017). BAT starts by fitting manually generated
q(a? = max Fe(a..hv)
(6)
which is the model F's largest prediction score (after softmax) to a wrong class other than the true class y. A high poisoning score suggests that the current model predicts the sample 2y to be wrong

---

Under review as a conference paper at ICLR 2022
class with high confidence. Note that the atypical samples are usually fitted into the model later than typical samples. Under a model with a few atypical samples fitted, if there is an atypical sample 27th with high q(a)d)) a:ddv very likely to be close to the distribution class. In Figure. we present several atypical (adversarial) samples with poisoning scores larger than 0.8. These samples present semantic features which are very similar to the wrong class that the model predicts. Therefore, fitting these atypical samples could cause the model to learn mixed concepts of features and degrades the model performance.
wrong class instead of its true
During the training process, desire that the model should mitigate the influence from the atypical samples with large poisoning scores, but still learn other "useful /benign" atypical samples. Thus, design cost-sensitive reweighting strategy which downweights the cost of atypical samples with large poisoning scores. In particular, we specify the weight value w; for each training sample ar as:
exp(p< ((x))))
if mem(z;) > and arg max, Fi (x Otherwise.
W/
where E R and control the size of the reweighted atypical set. Since the function exp(-a(-)) is decreasing and ranges from to O, the samples with large poisoning scores will be assigned with small weights close to O. Thus, in the reweighting algorithm, we train the model to find optimal model Fiw by assigning the weight vector w:
Fi arg min
E; w. Σ [c C(F(at")) yr)
(8)
In the training process, those (adversarial) training samples with large poisoning scores are assigned with small weights and correspondingly their influence will be largely mitigated, Note that in Eq. 7 estimating mem(x;) for each 2g via the leave-out retraining method described in Eq. I is potential to be computational costly. However, alternative efficient strategies have been discussed to estimate how likely one sample is atypical, including confidence- disagreement (Carlini al. 2019). These methods are shown to have good alignment with the method in Eq. [ thus these strategies can be leveraged to improve the efficiency of Eq. 7 In Appendix A we provide detailed discussions about these methods.
strategies and ensemble
4.2 DISCRIMINATION LOSS.
In addition, we introduce Discrimination Loss to further eliminate the poisoning effect of fitting atypical samples. Based on the theoretical analysis Section 3.2 imagine that the atypical sample xo has a fixed distance d_1 to the center of its labeled class, strategy to mitigate the poisoning effect of fitting Zn is to increase the two class's 's center distance 21 and hence increase the distance d4 1 of the poisoning example the poisoned class "+1". Motivated by this point, introduce LDt to increase the distances of typical samples among different classes in the representation space.
(D)(())
E (ZODAMREE)R))
log
(9)
((
where y: === Msi Uh # vk= = 1,2, K; mem(z.), mem(z,), mem(zx) <
Specifically, h(.)) is the model F's pepp-enttiigg ultimatte layer's output representation: E R+ is the temper- ature value and K E Z+ is a fixed number of the samples that are randomly chosen with different labels with y Intuitively, constraining the Discrimination Losy imposes the model to output similar representations for the typical sample pairs (z .z.ill) in the same class, and distinct representations for (a::dy ....fh)) from different classes. Similar approaches have been widely used in representation learning, such as Triplet Loss (Schroff et al.) 2015) and contrastive learning methods (Chen et al.) 2020), which stress the good property of DNNs' representation space. With the Discrimination Loss incorporated, the final objective of BAT is formulated as:
Fbat = arg min
(E
Σ ((F(a.)))))))))))))) v.y:]] + Lmn(FF)
(10)
where B > O that controls the intensity of Discrimination Loss. The detailed training scheme of Benign Adversarial Training is presented in Appendix D
In practice, under training algorithms using mini-hatches, the set {Ix K can be replaced by all (typical)
samples in the mini-hatch, with different labels from Vi-
7

---

Under review as a conference paper at ICLR 2022
5 EXPERIMENT
In this section, we present the experimental results to validate the effectiveness of the proposed BAT al- gorithm on benchmark datasets and compare with state-of-the-art baseline methods. The implemen- tation can be found via wttp::tps:///mm http:///nnonoouuuooo 4
5.1 EXPERIMENTAL SETUP
In this work, in order to demonstrate the merit of BAT, we conduct the experiments mainly on benchmark datasets CIFAR100 (Krizhevsky et al., 2009) and Tiny ImageNet (Le & Yang (0135). which are relatively complex datasets (i.e., containing larger fractions of atypical samples). For both datasets, we study the algorithms under the model architectures 8 and WideResNet (WRN) (He et al. 2016). In this section, we only present the results of WRN models and leave the results R8 8 in Appendix E As a fair comparison with BAT, we implement the baseline algorithms including PGD adversarial training (Madry et al. (017)) as well as its most popular variant TRADES (Zhang et al. (019).. In addition, we include several recent algorithms: MART (Wang) et al. 2019) and GAIRAT (Zhang et (222), which also incorporate reweighting strategies into adversarial training. For BAT and all baseline methods,
run the algorithms using SGD (Bottou 2010) for 160 epochs where the learning rate is from 0.1 and decays by 0.1 after the epoch 80 and 120. More implementation details are listed in Appendix B
Performance on CIFAR100. For a comprehensive comparison between different methods on CI- FARIO0, in Table we report the models' clean accuracy and adversarial accuracy against 48 adversarial examples, generated by various attacking algorithms including FGSM attack (Goodfellow al. 2014), PGD attack (Madry et al. 2017). CW (Carlini & Wagner 2017) and Auto-Attack (Croce & Hein 2020)). For BAT, we report its performance when choosing its optimal hyperparameter: a' -, 3 0.2 and a = 2, 3 = 0.2. In Section 5.2 we will discuss the impact of the selection of a: and 3 on BAT and argue that both components are necessary for BAT to achieve the optimal performance. For baseline methods, the settings follow the original papers' suggestions and their results reported the checkpoints with the best PGD adversarial accuracy.
Table I: Performance of BAT vs. Baselines on CIFARI00 Under WRN28
Method
Clean Acc.
FGSM
PGD
CW
AA.
PGD Adv. Training
59.7 (-2.3)
(-3.7)
24.7 (-3.8)
24.2 (-2.3)
22.5 (-2.3)
TRADES (1/A=
57.3 (-4.7)
34.5 (-4.1)
24.9 (-3.6)
(-1.9)
22.9 (-1.9)
MART (Wang et al. 2019
56.5 (-5.5)
36.1 (-2.5)
26.8 (-1.7)
25.3 (-1.2)
23.8 (-1.0)
AT iZhang et al.
2020)
60.2 (-1.8)
34.8 (-3.8)
24.4 (4.1)
24.8 (-1.5)
22.9 (-1.9)
BAT (a- = 1 0.2)
62.0 (+0.0)
38.6 (40.0)
28.5 (40.0)
26.5 (40.0)
24.8 (40.0)
BAT (a 2, 8 0.2)
61.4 (-0.6)
38.9 (+0.3)
28.2
(-0.3)
26.3 (-0.2)
24.8 (-0.0)
In Table [ we also report the performance difference between baseline methods vs. BAT (a = 1, 3 = 0.2) parentheses. From the results, we can find that BAT' enjoy good clean & adversarial accuracy trade-off' at the same time. For example, BAT methods can achieve the highest clean accuracy as well adversarial accuracy among all baseline methods. Compared to PGD adversarial training Madry trainingMMadry et al. (2017), BAT methods have around 2 to 3% improvement on both clean accuracy and adversarial accuracy on strong attacks such as PGD, CW and AA. Compared to other baseline methods such as MART Wang et al. 2019) it has more than 1% robustness improvement evaluated by strong adversarial attacks, but much higher clean accuracy (> 5%).
Performance on Tiny ImageNet. Tiny ImageNet (Le & Yang 2015) contains 200 classes of the images in the original ImageNet (Krizhevsky et al. 2012) dataset, with 500 training images for each class, and image size 64 64. In our experiments, we only choose the first 50 classes in Tiny ImageNet for training and prediction. Since the image size is 64 x 64, for both training and robustness evaluation, we consider the adversarial attacks are bounded by 4oo normm-4/255 In Table 2 we report the performance of BAT and baseline methods. Similar to the conclusions we can make from CIFAR100 BATs can achieve consistently better clean accuracy as well adversarial accuracy compared to baseline methods.
5.2 ABLATION STUDY
In this subsection, we study the potential impact of the hyperparameters chosen in BAT, which is that controls the Reweighting process and /3 that controls the Discrimination Loss. In Fig. 4 we
8

---

Under review as a conference paper at ICLR 2022
Table 2: Performance of BAT vs. Baselines on Tiny ImageNet Under WRN28
Method
Clean Acc.
FGSM
PGD
CW
AA.
PGD Adv. Training
58.9 (-3.4)
(-5.9)
31.7 (-4.1)
(-.07
30.0 (-4.3)
TRADES (1/A-
59.7 (-2.6)
37.4 (-4.2)
32.0
(-1.)
32.0 (-2.3)
MART (Wang et al. 2019)
58.2 (-4.1)
(-0.6)
(-0.2)
33.9 (40.2)
34.1 (-0.2)
GAIRAT (Zhang et al 2020)
60.5 (-1.8)
39.1 (-2.5)
(-2.7)
(-.8)
32.3 (-2.0)
BAT (a = 1, 0.2)
62.3 (+0.0)
41.6 (+0.0)
35.8 (+0.0)
33.7 (+0.0)
34.3 (+0.0)
BAT (o 2, 0.2)
62.4 (+0.1)
43.1 (+1.5)
(71.6)
35.3 (+1.6)
35.6 (+0.7)
conduct the experiments on CIFARIOO with BAT when is chosen in [0,1,2.3] and / is in [0, 0.1, 0.2. 0.3]. In Fig. 4 we show each model's overall clean accuracy (left) & adversarial accuracy on PGD attack (right) along the Z-axis, and X-axis / Y-axis indicate the models' corresponding variables of (at, 3). Note that when a = 3 = O. the BAT method regresses to original PGD adversarial training. From the result, can see that a positive pair of (a, (3) can benefit both model clean and adversarial accuracy (on PGD attack). Therefore, both components (i.e., Reweighting and LDI.) of BAT are helpful and necessary. However, when a 3 is too large, it will hurt the BAT's clean accuracy. Hence, in CIFAR100, when a = 1 or 2 and 3 = 0.2, BAT can achieve the optimal performance.
RELATED WORKS
Memorization and atypical sam- ples. The memorization effect of overparameterized DNNs have been extensively studied both empir- ically (Zhang et al. 2016)] Nakkiran eet al.] 2019) and theoretically (Bartlett & Mendelson 2002). From traditional views, the memorization can be harm- ful to the model generalization, be- cause it makes DNN models easily fit those outliers and noisy labels. How- ever, recent studies point out the concept of "benign overfitring" (Bartlett et al. 2020 Feldman 2020, Feldman & Zhang 2020), which suggests the memorization effect necessary for DNNs to have extraordinary performance on modern machine learning tasks. Especially, the recent work (Feld- man & Zhang 2020) empirically figures out those atypical/rare samples in benchmark datasets and show the contribution from memorizing atypical samples to the DNN's performance. Besides the work (Feldman & Zhang, 2020), there also other strategies (Carlini al.) 2019) to find atypical samples in training dataset. Notably, our work is not the first effort to study the influence of memorization DNN's adversarial robustness. A previous study (Sanyal et al. (020)) illustrates that memorizing the mis-labeled samples might be a reason to cause the DNNs ad adversarial vulnerability. In our paper, we focus on atypical samples, which appear much more frequently in common datasets, and we study their impacts especially on adversarial training algorithms (Madry et al.
030
2
apiaa
0.2
0.1
0.1
0
Figure 4: Clean Acc.(left) & Adv. (right)) of BAT
2017 Zhang
al. 2019
Chatterji & Long 2020 Muthukumar et al.) 2020).
Adversarial robustness.
Adversarial training methods Maddy et al. 2017 Zhang et al. 2019
Wang et al., 2019) Zhang al. 2016 Rice al., (2200) are considered as one of the most reliable and effective methods to protect DNN models against adversarial attacks (Goodfellow et al. 2014 Xu et al. 2019). However, there are several intrinsic properties of adversarial training different from traditional ERM which require deeper understandings. For example, they always suffer from poor robustness generalization (Schmidi et al. 2018:) Rice et al. 2020); they always present strong trade-off relation between clean accuracy they are likely to have biased accuracy & robustness among different subgroups of data (Xu et al. 2021). Our work aims to study the robustness generalization issues from the data perspective and demonstrate the significant connection of the memorization effect with these properties.
robustness (Tssppras et al. 2018 Zhang et al. 2019);
7 CONCLUSION
In this paper, we draw significant connections of the memorization effect of deep neural networks with the behaviors of adversarial training algorithms. Based on the findings, devise novel algorithm BAT to enhance the performance of adversarial training. The findings of the paper can motivate the futures studies in building robust DNNs with more attention on the data perspective.
9

---

Under review as a conference paper at ICLR 2022
REFERENCES
Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities; Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov):463-482, 2002.
Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117(48):30033-3007) 2020.
Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning attacks against support vector machines. arXiv preprint arXiv: ww.. 6389, 2012.
Léon Bottou. Large-scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT"2010, pp. 177-186. Springer, 2010.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp), pp. 39-57. IEEE, 2017.
Nicholas Carlini, Ulfar Erlingsson, and Nicolas Papernot. Distribution density, tails, and outliers in machine learning: Metries and applications. arXiv preprint arXiv: 1910.11910. 2019.
Niladri S Chatterji and Philip Long. Finite-sample analysis of interpolating linear classifiers overparameterized regime. arXiv preprini arXiv:2004. 12019, 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks, In International Conference on Machine Learning, pp. 2206-2216. PMLR, 2020.
Theodoros Evgeniou, Massimiliano Pontil, and Tomaso Poggio. Regularization networks and support vector machines. Advances in computational mathematics, 13(1):1-50, 2000.
Daniel Fagnant and Kara Kockelman. Preparing nation for autonomous vehicles: opportunities, barriers and policy recommendations. Transportation Research Part A: Policy and Practice, 77: 167-181, 2015.
Vitaly Feldman. Does learning require memorization? a short tale about long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pp. 954-959, 2020.
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. arXiv preprint arXiv:2008.00 037003 2020.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572. 2014.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Fei Jiang, Yong Jiang, Hui Zhi, Yi Dong, Hao Li, Sufeng Ma, Yilong Wang, Qiang Dong, Haipeng Shen, and Yongjun Wang. Artificial intelligence healthcare: past, present and future. Stroke and vascular neurology, 2(4), 2017.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep con- volutional neural networks. Advances in neural information processing systems, 15:1097-1105, 2012.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning at scale. arXiv preprint arXiv: 1.0:://311.00666 2016.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7:7, 2015.
10

---

Under review as a conference paper at ICLR 2022
Mingchen Li, Mahdi Soltanolkotabi, and Samet Oymak. Gradient descent with early stopping is provably robust to label noise for overparameterized neural networks. In International Conference on Artificial Intelligence and Statistics, pp. 4313-4324, PMLR, 2020.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 2017.
Abdul Adeel Mohammed, Rashid Minhas, QM Jonathan Wu, and Maher Sid-Ahmed. Human face recognition based on multidimensional pea and extreme learning machine. Pattern Recognition, 44(10-11):2588 2597, 2011.
Vidya Muthukumar, Kailas Vodrahalli, Vignesh Subramanian, and Anant Sahai. Harmless interpo- lation of noisy data in regression. IEEE Journal on Selected Areas in Information Theory, 1(1): 67-83, 2020.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever, Deep double descent: Where bigger models and more data hurt. arXiv preprint arXiv:19/2.02292. 2019.
Leslie Rice, Eric Wong, and Zico Kolter. Overfitting in adversarially robust deep learning. In International Conference Machine Learning, pp. 5093-8104. PMLR, 2020.
Amartya Sanyal, Puneet K Dokania, Varun Kanade, and Philip HS Torr. How benign is benign overfitting? arXiv preprint arXiv:2007.0444 044028, 2020.
Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Mądry. Adver- sarially robust generalization requires more data, arXiv preprint arXiv: 11285, 2018.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 815-823, 2015.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.
Dimitris Tsipaa. Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. arXiv preprint 10Xiv 2018.
Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving adversarial robustness requires revisiting misclassified examples. In International Conference on Learning Representationss, 2019.
Han Xu, Yao Ma, Haochen Liu, Debayan Deb, Hui Liu, Jiliang Tang, and Anil Jain. Adversarial 2019.
attacks and defenses in images, graphs and text: A review. arXiv preprint
Han Xu, Xiaorui Liu, Yaxin Li, Anil Jain, and Jiliang Tang. To be robust or to be fair: Towards fairness in adversarial training. In International Conference on Machine Learning, pp. 11492- 11501. PMLR, 2021.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint 1eXiv:16111.335 2016.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P Xing, Laurent El Ghaoui, and Michael I
Jordan. Theoretically principled trade-off between robustness and accuracy. arXiv preprint 10Xiv:1901.08573 2019.
Jingfeng Zhang, Jianing Zhu, Gang Niu, Bo Han, Masashi Sugiyama, and Mohan Kankanhalli. Geometry-aware instance-reweighted adversarial training. arXiv preprint urXiv20010.01336, 2020.
11

---

Under review as a conference paper at ICLR 2022
Appendix
In the supplementary materials, provide details about atypical samples and the proposed algorithm, as well as the full experimental results the preliminary study and the proposed method.
Additional Introduction of Atypical Samples
Appendix A
Additional Results of Preliminary Study (in Section 3.1
Appendix B.]
Traditional ERM Adversarial Training
Additional Results of Preliminary Study (in Section
Appendix B.2
Traditional ERM Adversarial Training
Proof of Theories
Appendix
Full Training Scheme of BAT
Appendix
Additional Results of BAT
Appendix
Boarder Impacts of this Paper
Appendix
A ADDITIONAL INTRODUCTION OF ATYPICAL SAMPLES
In this section, we provide additional introductions about the memorization effect and atypical samples in image classification tasks. We first introduce several strategies to identify atypical samples common datasets. Then, we show several examples illustrate the distribution & frequency in common datasets.
A.1 How TO ESTIMATE WHETHHER ONE SAMPLE IS ATYPICAL?
Leave-out Method The work (Feldman & Zhang 2020) proposes to examine which training samples can only be fitted by memorization, and measure each training sample's "memorization value" Formally, for a training algorithm A (i.e., ERM), the memorization value (iemmA,,,xz, of training sample (z,, ys) in training set D is defined as:
mem D,
(F(x:) = u:)) Pr. (F(x:) = v1) F-A(D)a)) F-A(D)
Pr.
which calculates the difference between the model F's accuracy on æ; with and without æ; removed from the training set D of algorithm A. Note that for each sample 24, if its memorization value high, it means that removing 2, from training data will cause the model with high possibility to wrongly classify itself, so 2; is very likely to be fitted only by memorization and be atypical. Although promising, the computational efficiency of Leave-out method is potential to be low. For example, the Leave-Our Out method requires repeatedly training DNN model for 1,000 trials accurately estimate the memorization values.
Ensemble Disagreement The work (Carlini et al. 2019) provides an alternative way to efficiently figure out the examples which are poorly represented by other samples from the training dataset (atypical samples). They assume the samples which are well-represented in the distribution should be relatively easy for many types of DNN models to learn. Concretely, the ensemble agreement trains series of models F = b with different architectures and initializations. Then, for each sample Iz, it computes the disagreement of all models:
ed(x;) = m
(DDiverrennc((((t)).F... j=1 p=1
Intuitively, if cd(x;) is large, different models will give different predictions to the sample z;- It suggest that the sample z; cannot be easily learned by most models in F.. Therefore, the sample x; is likely to be atypical. In practice, (Carlini & Wagner 2017) trains multiple DNN models for 10 random initializations, or get 10 pretrained from public resources, which significantly improve the efficiency to find atypical samples.
Confidence- Based Method Similar to the ensemble disagreement method, the work Carlini et al. (2019) assumes that the models in F (F) should be confident examples that
12

---

Under review as a conference paper at ICLR 2022
well-represented by the whole training dataset. Thus, the propose to measure the average confidence value these models on sample z;;
conf(
max(FF(za))oo)))
The smaller the confidence value a sample Zi. it is more likely to be atypical. Remarkably, the work Carlini et al. (2019) empirically demonstrates that the three metrics decribed above have good alignment for datasets such as MNIST, CIFARIO, 1O, CIFAR 100 (Krizhevsky et al., 2009) and ImageNet (Krizhevsyy al. 2012). Therefore, for practical application of proposed method BAT, we can also substitute the leave-out method to ensemble disagreement and confidence- baafideeneebased methods, in order enhance the efficiency of BAT.
A.2 DISTRIBUTION OF ATYPICAL SAMPLES
In Fig. 5 we provide several examples of images from CIFARIO, CIFAR100 (Krizhevsky et al. 2009) and Tiny ImageNet (Le & Yang 2015) respectively, with different memorization value (as defined in Section 2. around 0.0, 0.5,, 1.0. These examples suggest that if the memorization value of an image is large, this image is very likely to be "atypical", as presents very distinct semantic features with the images in the main distribution (with memorization value 0.0).
0.0
0.0
0.5
0.5
1.0
1.0
"Dog" CIFAR10
"Plate" CIFAR100
0.0
0.5
1.0
'Mountain Lion" in Tiny ImageNet
Figure 5: Examples of Images with Different Memorization Values
13

---

Under review as a conference paper at ICLR 2022
In Fig. 6 provide histograms to show the distribution of the estimated memorization values of all training samples from CIFARIO, 10, CIFAR 100 and Tiny ImageNet. From Fig, 6 we can observe that atypical samples (with high memorization value 0.15) consist of significant fraction (over 40% & 50% respectively) in CIFARION and Tiny ImageNet. CIFAR10, they also consist of a non-ignorable fraction which is 10%.
53.0%
24.0%
60.0%
23.0%
10.%%
26.0%
0.%%
80%
0.
02
244
0.6
0.5
20
0.8
60
02
04
2.6
0.5
1.0
0.0%
0.0
0.2
64
0.5
10
(a) CIFARI00.
(b) CIFARIO.
(c) Tiny ImageNet.
Figure 6: Frequencies of Training samples with Different Memorization Values in Various Datasets
In Fig. 7 we provide several pairs of images with high influence value (as defined Section 2. which over 0.15. In each pair, the training sample also has high memorization value over 0.15. These examples suggest that there exist atypical samples in both training & test sets of CIFAR10 CIFARI00 and Tiny ImageNet. A pair of atypical samples (in the training set and test set) with high influence value are visually very similar. Moreover, since they have high influence values, removing the atypical samples in the training set is very likely to cause the model fail on the test atypical samples. Therefore, without memorizing the atypical sample in the training set, the model can hardly predict the atypical samples in the test
CIFAR100
CIFAR10
Tiny ImageNet Train
Train
Test
Train
Test
Test
Figure 7: High Influence Pairs with Influence Value 0.15
B ADDITIONAL RESULTS FOR PRELIMINARY STUDY
In this section, we provide the full results of the preliminary study in Section 3 on CIFARIO, CIFAR100 and Tiny ImageNet, to illustrate the distinct behaviors of the memorization effect between traditional ERMs and adversarial training. In both ERM and adversarial training, we train the models under ResNet18 and WideResNNe22-100 (WRN28) architectures. In the experiments, we train the models for 200 epochs with learning rate 0.1, momentum 0.9, weight decay 5e-4, and decay the learning rate by 0.1 at the epoch 150 and 200. For adversarial training, we conduct experiments using PGD adversarial training Madry al. (2017) 7 by default to defense against loc -8255 adversarial attack, with the exception on Tiny ImageNet, which against Lx -4/255 attack. For robustness evaluation, conduct 20-step PGD attack.
14

---

Under review as a conference paper at ICLR 2022
B.1 ADDITIONAL RESULTS FOR PRELIMINARY STUDY SECTION 3.1
In this subsection, we provide more experimental results to validate the statement in Section 3.1 where state that fitting atypical samples in adversarial training can only improve the clean accuracy of test atypical samples, but hardly help their adversarial robustness. We provide full empirical results to show that: (i) In traditional ERM, fitting atypical samples improves the clean accuracy of test atypical samples. (ii) In adversarial training. fitting (adversarial) atypical samples improves the clean accuracy of test atypical samples but can hardly improve the adversarial robustness of them. The experimental setting follows Section 3.1] where we apply traditional ERM and adversarial training on original CIFARIO I0, CIFARIOU Tiny ImageNet datasets. We evaluate the model's clean accuracy and adversarial accuracy on training atypical set Dayp {I; E D mem(x;) > 0.15) and its corresponding test atypical set D'ays = (2/} E D' infi 22, x3) > 0.15, for Vz, E Datyp)
(i) Additional Results for Preliminary Study Section 3.1 In Traditional ERM
Fig. Fig. and Fig. I0 report the performance (clean accuracy) of traditional ERM, which is evaluated on atypical sets under ResNet18 (left) and WRN28 (right) on CIFARI00, CIFAR 10 and Tiny ImageNet. From the figures, we can obverse that fitting atypical samples during training can effectively help the models to achieve good clean accuracy test atypical samples in all datasets. Note that here we only report clean accuracy as they are not robust against adversarial attacks.
1.0
1.0
0.4
0.0
25
50
15
100
125
150
135
280
04
50
75
100
125
150
175
ResNet18.
(b) WRN28.
Figure 8: Clean Accuracy on Atypical Set of CIFARI00
1.0
Tarnng
Training
0.4
04
0.0
25
20
75
100
125
150
375
200
00
25
50
75
100
375
350
175
(a) ResNet18.
(b) WRN28.
Figure 9: Clean Accuracy on Atypical Set of CIFARIO
Sest Twining
Teat
0.6
0.3
Training
0.4
04
0.0
23
20
15
100
125
150
275
200
06
25
50
75
100
325
150 175
(a) ResNet32.
(b) WRN28.
Figure 10: Clean Accuracy on Atypical Set of Tiny ImageNet
15

---

Under review as a conference paper at ICLR 2022
(ii) Additional Results for Preliminary Study Section 3.1 In Adversarial Training
Fig. 12 and Fig. 13 report the performance of adversarially trained models. We evaluate the clean accuracy and adversarial accuracy the training atypical set Payp and test atypical set 2 Payp From the results, we can observe that although fitting atypical samples can help the model to have modest clean accuracy test atypical samples, the adversarial robustness of them is constantly low and can hardly be improved during the whole training process.
Fig.
Tee Taining
3
3rdt
3-d
Tasto
ning
ming
50
25
70 200 20 150 15 239
50
100
12
330
27
200
08
300 23
50
129
350 379
00
23
30
10
300
12)
330 379
(a) Clean (left) & Adv Acc. (right) under ResNet18.
(b) Clean (left) & Adv Acc. (right) under WRN28.
Figure 11: Clean Accuracy and Adversarial Accuracy on Atypical Set of CIFARI00
1.
1.0
58
Tuining
0.'
08
2.4
24
(a) Clean (left) Adv Acc. (right) under ResNet18.
(b) Clean (left) Adv Ace. (right) under WRN28.
Figure 12: Clean Accuracy and Adversarial Accuracy on Atypical Set of CIFAR10
Ter Taining
THet
3Htt
THer
12n
ning
0.8
24
52
02
02
02
28
30
15
100 105
359
175
200
28
25
75
104
150
175
350
08
25
50
75
100
125
155
04
30
75
166
185
(a) Clean (left) & Adv Ace. (right) under ResNet32.
(b) Clean (left) & Ady Acc. (right) under WRN28.
Figure 13: Clean Accuracy and Adversarial Accuracy on Atypical Set of Tiny ImageNet
B.2 ADDITIONAL RESULTS FOR PRELIMINARY STUDY SECTION 3.2
In this subsection, we provide more experimental results to validate the statement in Section 3.2 where state that fitting atypical samples in adversarial training can hurt the performance (clean adversarial accuracy) of typical samples. We provide full empirical results to show that: (i) In traditional ERM, fitting atypical samples will not hurt the models' performance (clean accuracy) of typical samples. (ii) In adversarial training. fitting atypical samples can degrade the clean & adversarial accuracy of typical samples. (iii) In adversarial training, fitting atypical samples can degrade the quality of learned representations, especially the models' discrimination between classes. The experimental setting follows Section 3.2 where we train the models for several trails on resampled CCIFAR100, CIFARIO. 10, Tiny ImageNet) datasets: each dataset is constructed with the whole training typical set Deyp and part of the training atypical set Dayp (randomly sample 0%, 20% and 100% in Dayp). We evaluate the models" performance on test "typical" set D' which is defined in Section 3.2
16

---

Under review as a conference paper at ICLR 2022
(i) Additional Results for Preliminary Study Section 3.2 In Traditional ERM
Fig. 14 Fig. 13 and Fig. report the performance of traditional ERM, trained on different resampled datasets with different amount of atypical samples existed. The figures report the clean accuracy on test typical set of CIFARI00, CIFARIO and Tiny ImageNet. We also Ieave the robustness performance out here as the models not robust to adversarial attacks. From the results, we can conclude that in traditional ERM, fitting atypical samples will not degrade the models' performance on typical samples. For example, in CIFAR 100 dataset, with 100% atypical samples included (Atypical-100%) the accuracy on the test typical set is even slightly higher than the model trained without atypical samples (At) This conclusion is consistent for all three datasets and model architectures.
1.00
2.00
0.93
0.50
0.90
Mypical 10.7%
0.95
2.94
0.94
2.93
0.2
2.00
0.00
0.50
0.52
008
0.0
100
(a) ResNet18.
(b) WRN28.
Figure 14: Clean Accuracy on Typical Set of CIFARI00
1.00
0.98
3.92
D194
0.54
D92
090
0.00
0.18
0.83
096
0.06
153
300
000
(a) 18
(b) WRN28.
Figure 15: Clean Accuracy on Typical Set of CIFARIO
Dhypinal 2% Shymaal-300 300%
0.44
0/90
0.50
0.6
125
182
375
20 40
50
140 1992
(a) ResNet32.
(b) WRN28.
Figure 16: Clean Accuracy on Typical Set of Tiny ImageNet
(ii) Additional Results for Preliminary Study Section 3.2 In Adversarial Training
Fig. 17 Fig. 18 and Fig. 19 report the performance of adversarial training, on different resampled datasets with different amount of atypical samples existed. The figures report both clean and adversarial accuracy on test atypical sets CIFARI00, CIFARIO and Tiny ImageNet. Based on the experimental results, we find that including more atypical samples can cause the model have worse performance on typical samples in all three datasets. In datasets with a large portion of atypical samples, such as CIFARIO0 1000 the negative effects of atypical samples more obvious. In CIFAR100, training on datasets with 100% atypical samples (Atypical 100%) can cause the clean & adversarial accuracy drop by 7% and 8%, respectively.
17

---

Under review as a conference paper at ICLR 2022
0.55
0.55
Mraica
Serical 201
340
0.00
0.00
Eoodh
(a) Clean (left) Adv Acc. (right) under ResNet18.
(h) Clean (left) Adv Acc. (right) under WRN28.
Figure 17: Clean Accuracy and Adversarial Accuracy Typical Set of CIFARI00
appoarch
Mepicalrss
0.59
8.45
0.45
0.13
25
30
15
180
12
175
0.43
30
200
15
200
179
0.00
23
13
100
155
150
179
0.43
36
100
125
150
(a) Clean (left) & Adv Acc. (right) under ResNet18.
(b) Clean (left) & Adv Acc. (right) under WRN28.
Figure 18: Clean Accuracy and Adversarial Accuracy on Typical Set of CIFARIO
3.00
1.70
1.00
1.50
3.55
3.00
0.45
Aegicsk
0.90
3.50
0.93
3.5
(a) Clean (left) & Adv Acc. (right) under ResNet32.
(b) Clean (left) Adv Acc. (right) under WRN28.
Figure 19: Clean Accuracy and Adversarial Accuracy on Typical Set of CIFAR100
B.3 AN ABLATION STUDY FOR "POISONING ATYPICAL SAMPLES"
In this subsection, we conduct additional experiments to verify the claims about "poisoning atypical samples". In particular, aim to demonstrate that: the atypical images with high "poisoning scores" are likely to obtain features which visually resemble the images from a (wrong) different class, so fitting them likely to degrade the model performance. To validate claim, print 200 images of atypical samples from CIFAR 100 training set, with highest/ lowest poisoning scores as defined in (6) of the paper. Then, we let two individual human annotators to label each image, by choosing one of 4 options including: (a.) This image belongs to class yı. (b.) This image belongs class y2- (c.) Both class y1 and 3/2 are likely. (d.) Neither class y1 or 3/2 Here, y11 is the ground truth label of the sample; Y2 is the class other than y1 which the model has the maximal confidence: y2 = ar m Fx),, and the model F is obtained via PGD adversarial training, with all atypical samples removed from the training set. We report the percentage of the answers of the human annotators. We report the percentage of the answers the human annotators.
From the results, we can see, for the samples with high poisoning scores, people are more likely to believe that "the image is from y2" or "the image is both likely from yı or 1/2". is because the highly-poisoning images can persist lot of similar features from these two classes. It also worth to mention that for samples which people labeled them to y1, they usually also have features belonging to the class V2. For example, a "bowl" which is round and with orange color. Although people believe it is a bowl confidently. does have features such as "round" and "orange color". which are also features of class "orange".
18

---

Under review as a conference paper at ICLR 2022
Class y1
Class U/
Both Likely
Neither.
Total
High Poison Score
59.0
12.0
26.0
4.0
100.0
Low Poison Score
96.0
0.0
1.5
2.5
100.0
C PROOF OF THEORIES
In Section 3.2 we argue that the poisoning effect during adversarial training can be significantly stronger than traditional ERM. In this section, provide detailed discussions about this claim as well as complete proofs. Our analysis is based on a binary classification task with two-dimensional Gaussian Mixture distribution:
u.a.T y
(11,+1], 0 (n, x
N (0, (-0,0011)
if yf if yy=-1,
(11)
and we consider linear models F = sign(w + b), where w and are the model weight and bias. Moreover, assume there an atypical training sample vo IR² labeled "-1", with Euclidean distance to the center of each class | (-0)
d-1, ||00- 0|12 = d41. Intuitively, this setting of zo resembles the cases in Figure 3f d+1 is relatively small, since has short distance the wrong class. We also constrain d_1 2 x y for zo to preserve certain similarity to the class "-1". The following theorem shows that there always exists model which fits the sample Fo, and achieve optimal accuracy (i.e. 99%) on both classes, although d41 can be small.
Theorem For a data distribution Eq. 4 given that n/a is large enough, we define I = /2. (0-999 a (1 + 101101110000 (0.99) -,)-1/2, For a point zo with d41 S 2n, if d41 then there exist a model F filting zo: F( = -1 and has classification error at most 1% on both classes.
Proof We prove this theorem by two main sleps.
Step 1. we show that for any zo satisfying d_ 2n, d+1 2 1, at least one of F1, and fits To: Bie [1,2), F((zo) -,, where:
क -- o)) F2(x) = sign((T, V1²² -2 x)
Fi (x)) == sign( [1,0) x
F2(x) sign([T, vn2 72
In the following parts, we let T 4-1. o for simplicity. Step 2. Both models Fi and Fi has classification accuracy at least 99% for both classes.
Proof of Step 1. For any given sample vn = lp. q). on hand, if <1- Fi(xo) = -
On the other hand, p 2 17 T, we will show all samples 2g = [p.g] which satisfies F F2 q))) # -, it must have d+1 < where I = /2a- + 1/n) Here, we assume S 0 without loss of generality. This can be treated as a quadratic maximization problem under convex closed bound:
(p 1)= +²² 1)
subject to
K1 = T p 1 -- 20
and K₂ == p 2 1 T;
(12)
and Kg = (p (-n))² 2 + d² < (2n)
and Ka k=:4q- 0
Figure 20 provides an illustration of the problem proposed in Eq. 12 The optima of Eq. 12 lies on the
border of the closed region (shadow area). Next, we solve the values of the object (p n) 9² on points S and S2, and compares the objective value when To = S or Xn S2-
For F0 S1, it satisfies:
T.P V72 -
--
p p= T;
19

---

Under review as a conference paper at ICLR 2022
K2
K
0 = [17 0]
S1
S2
K1
3/
Figure 20: Illustration Eq.
By solving the equations, we have (p - op -V2 (1+
For zo = St, it satisfies:
r.p + V12 72 =
(p- (-n) + (21)²
By solving the equations, have see that, when n/o sufficiently large, the objective will achieve larger value on S1, whose optimum is not larger than 12, Therefore, finished proving the step 1.
(1- (- ((( is not hard to
Proof of Step 2. The distance of the center of class "+1" to the decision boundaries of F1, F2 and Fs are all T. Therefore, the error of class "+l":
(r(F(() + +1yy = +1) = Pr(N(0,0²)> >r)
PP((N(0,1) 1/00)
PP((N(O, $ +-
1%
For the class "-1", since classifiers F2 and F3 are symmetric, they also have 1% error on class "-I". For classifier F1, is easy that it has smaller error than 1% class "_1". Based on the results of Step I and Step 2, we finished the proof of Theorem
Theorem 2 For the same data distribution in Eq. 4 and same definition 1 = 22 (1 + + (0.999) g)-1/2, We assume I d41 < k. I, where 1 S k << n/a. Then, any model F will have a classification error at least 50% on class "+1" when F fits the adversarial example of F(zu + 5) -1, v|10 S c, and
e 2 a (k /2 t -- (1 $ -01 - - 1/2
(13)
1
Proof 2 During adversarial training, if k 1, it will include the center of class "+1" into the adversarial training bound of sample To:
0 {a 1101|2 Se..
Therefore, if there is a model which fits any adversarial counterpart of the sample In, there does not exist linear model which has error smaller than 50% class "+1". Therefore, we finished the proof of Theorem
D THE DETAILED TRAINING SCHEME OF BAT
In this section, we provide the detailed training scheme of BAT in Algorithm [ In particular, BAT algorithm starts from randomly initialized neural network. On each mini-batch, it applies PGD
20

---

Under review as a conference paper at ICLR 2022
attack to generate (training) adversarial examples (Step 5). Following the Eq 6 and Eq 7 BAT calculates which samples are likely to be Poisoning Atypical Samples and their corresponding weight values (Step 6). Under the current mini- next, BAT calculates the Discrimination Loss of the typical samples Dyy (Step 8). Finally, BAT uses SGD to update the model parameter to minimize the reweighted adversarial loss regularized by times discrimination loss (Step 8).
Algorithm The Benign Adversarial Training (BAT) Algorithm
1: Input: Training dataset D, with typical set Dyp = {- E D; mem(2:) S o). atypical set Dayp {ir memm(12) x)) σ). Targeted type of adversarial attack: - attack. Hyperparameters
a. a R*.
2: Randomly initialize the network F
3: repeat
4: Fetch mini-batch data {(r4, v:)} current epoch
5: Using PGD to generate adversarial training sample {
6: Calculate Poisoning Score (_x?))) and weight w; as in Equation and Equation 7
7:
Calculate Discrimination Loss (F) within the current mini-batch, in Equation
8: Update F by SGD the objective: LBAT Σ, E,w. Ee [w ((F(2)))) v.] + B (F)
9: until End of Training
E ADDITIONAL RESULTS FOR EXPERIMENT
In this section, provide additional experimental results to validate the effectiveness of BAT method. In Table and Table 4, we provide the results of BAT and baseline models on CIFAR100 dataset under ResNet18 and WRN28 architectures. In Table and Table and of BAT and baseline models on Tiny ImageNet dataset under ResNet32 and WRN28 architectures. In the experiments, weight decay Se-4, and decay the learning rate by 0.1 at the epoch 80 and 120. To have a more comprehensive and reliable adversarial robustness, in addition PGD adversarial attack Madry et al.) (017)), we also measure the model's adversarial accuracy via other attack algorithms,, including FGSM attack Goodfellow al. (2014), CW attack Carlimi & Wagner (2017) and Auto Attack Croce & Hein 2020). For PGD) attack, we implement the algorithm with T00 steps. For CW and AA attack, we follow the suggestion from the original papers to set the hyperparameters.
provide the results
train the models for 160 epochs with learning rate 0.1, momentum 0.9,
Table 3: Performance of BAT vs. Baselines on CIFAR100 Under ResNet18
Method
Clean Ace.
FGSM
PGD
CW
AA.
PGD Adv. Training
56.9 (-2.6)
(-1.3)
27.4(+0.11
(-122)
23.6 (-0.7)
TRADES (1/X- 5)
56.6 (-2.9)
36.5 (-0.8)
26.9 (-0.4)
(-1.3)
23.9 (-0.4)
MART Wang et al.
(0119)
51.8 (-7.7)
36.1 (-1.2)
30.4
(-0.8)
24.4 (40.1)
GAIRAT Zhang et al
(2020)
58.2 (-1.3)
36,5 (-1.2)
27.8
25.9 (-0.7)
23.8 (-0.2)
BAT (a 1, 0.2)
59.5 (+0.0)
37.3 (+0.0)
27.3
26.6 (40.0)
BAT (a = 2, 0.2)
59.3 (-0.2)
37.1 (-0.2)
(70.)
26.5 (-0.1)
24.0 (+0.7)
Table 4: Performance of BAT vs. Baselines on CIFARI00 Under WRN28
Method
Clean Acc.
FGSM
PGD
CW
AA.
PGD Adv. Training
59.7 (-2.3)
34.9
24.7 (-3.8)
(-2.3)
22.5 (-2.3)
TRADES (1/A
57.3 (-4.7)
34,5 (-4.1)
24.9 (-3.6)
(-1.9)
22.9 (-1.9)
MART (Wang et al.
2019
56.5 (-5.5)
36.1 (-2.5)
26.8 (-1.7)
25.3 (-1.2)
23.8 (-1.0)
GAIRAT (Zhang al.
2020)
60.2 (-1.8)
34.8 (-3.8)
24.4 (-4.1)
24.8 (-1.5)
22.9 (-1.9)
BAT (a = 1, 0.2)
62.0 (+0.0)
38.6 (+0.0)
28.5 (40.0)
26.5
(+0.0)
24.8
(+0.0)
BAT (a = 2, 0.2)
61.4 (-0.6)
38.9 (+0.3)
28.2
(-0.3)
26.3 (-0.2)
24.8 (-0.0)
The results show that the BAT method can consistently outperform baseline models, BAT has better clean accuracy vs. adversarial accuracy trade-off, especially on the larger model architecture WRN28. Under 18 8, the BAT method achieves comparable adversarial accuracy to the best
21

---

Under review as a conference paper at ICLR 2022
Table 5: Performance of BAT vs. Baselines on Tiny ImageNet Under ResNet32
Method
Clean Acc.
FGSM
PGD
CW
A.A.
PGD Adv. Training
56.3 (-3.1)
37.5 (-2.9)
32.3 (+0.3)
(-1.7)
29.8 (-2.2)
TRADES (1/A 5)
55.4 (-4.0)
35.2 (-5.2)
28.8 (-3.2)
27.0
27.0 (-5.0)
MART Wang al. (019
56,2 (-3.2)
38.1 (-2.3)
34.5 (+2.5)
(40.3)
32.0 (+0.0)
GAIRAT Zhang et al (2020)
58.4 (-1.0)
37.3 (-3.1)
30.4
28,9 (-2.6)
(-3.0)
BAT (a L, 0.2) (+0.0)
59.4 (+0.0)
40.4 (+0.0)
32.0 (+0.0)
31.5 (+0.0)
(+0.0)
BAT (a 2,8 0.2)
59.4 (40.0)
41.3 (+0.9)
32.9 (+0.9)
31.8 (+0.3)
32.4 (+0.4)
Table 6: Performance of BAT vs. Baselines on Tiny ImageNet Under WRN28
Method
Clean Acc.
FGSM
PGD
CW
AA.
PGD Adv. Training
58.9 (-3.4)
3-.79)
31.7 (-4.1)
30.0 (-3.7)
30.0 (-4.3)
TRADES (1/A- 5)
59.7 (-2.6)
37.4
32.0
(-1.9)
32.0 3-203)).3
MART (Wang et al. 2019)
58.2 (-4.1)
(-006)
(-0.2)
33.9
34.1 (-0.2)
GAIRAT Zhang et al 2020
60.5 (-1.8)
(-2.5)
33.1 (-2.7)
31.9 (-1.8)
32.3 (-2.0)
BAT (o = 1, 0.2)
62.3 (+0.0)
41.6 (40.0)
35.8 (40.0)
33.7 (40.0)
34.3 (40.0)
BAT (o = 2, 0.2)
62.4 (+0.1)
43.1 (+1.5)
37..6)
35.6 (+0.7)
baseline methods, such as MART and GAIRAT, under the strongest attacking methods such as AA attack. However, BATs around 7. 7.7% higher clean accuracy than MART, and 1.3% GAIRAT. Under larger model architecture like WRN, BAT's have both the highest clean accuracy and adversarial accuracy on various attacking methods. Interestingly, BAT methods take better advantages under large model architectures. The potential reason is that larger neural networks have greater ability to those atypical samples during the early stages of training. As a result, the poisoning effect from fitting the "poisoning" atypical samples in baseline methods is stronger. However, BATs can effectively eliminate the poisoning effect and hence enjoy better performance.
F BOARDER IMPACT
Nowadays, deep neural networks (DNNs) have been widely applied to solve various machine learning tasks, especially on many safety-criticn tasks such as autonomous vehicle [Fagnant & Kockelman (2015), Al healthcare et al. (2017) and ID authentication Mohammed et al. (2011). etc. However, the existence of adversarial attacks Xu al. (2019) brings huge threats to the safety of these DNNs applications. As one of the most successful approaches to defend DNNs against adversarial attacks, adversarial training methods Madry et al. 2017): Zhang et (2016) still suffer from several disadvantages. For example, for a DNN model to achieve good adversarial robustness, it usually sacrifices its clean accuracy. Adversarially trained DNNs also present strong overfitting effects. In our study, we draw important findings explain these drawbacks of adversarial training from the data perspective and empirically validate these issues' relation to atypical samples in the data distribution. Morcover, the currently existed adversarially trained models only achieve satisfactory performance on simple datasets such as MNIST and CIFARIO. Motivated from our method improve the performance of adversarial training, especially more complex datasets. We anticipate that our findings and method are helpful for further studies to improve the effectiveness and feasibility of adversarial training and eventually build safer DNNs.
findings, we propose
22


Reviews:
[{"review_id": "WeMGJ_pfnS4", "reviewer": "Reviewer_ytJj", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The empirical observations on the effect of memorizing atypical samples are interesting.\n- The writing is clear, with detailed descriptions of the proposed method.\n- It is good to have ablation studies on different modules in BAT.\n\nWeaknesses:\n- Since there are several extra modules in the objective of BAT (Eq. (10)), there should be evaluations under adaptive attacks, whose objective involves the terms like $\\mathcal{L}\\_{DL}$.\n- As described in the paper, it seems that the authors do not apply early-stopping during the adversarial training process. This largely weakens the performance of baselines [b]. Besides, more advanced methods listed in [c] should be considered on CIFAR-100.\n- CIFAR-100 and Tiny ImageNet are not quite widely used in the adversarial community. It would be more informative to have the results like Table 1 on CIFAR-10, where many benchmarks exist to better estimate the effectiveness of BAT.\n\nMinors:\n- As suggested by [a], weak attacks like FGSM should not be treated as an evaluation, at least not in the main text.\n- There should be more details on the settings (e.g., iteration steps) of PGD and CW attacks.\n- The $1/\\lambda$ is 6 in TRADES paper, while it is set as 5 in, e.g., Table 1.\n\nReferences:\n[a] On Evaluating Adversarial Robustness.\n[b] Overfitting in adversarially robust deep learning.\n[c] RobustBench: a standardized adversarial robustness benchmark.", "summary_of_the_paper": "This paper investigates the memorization effects in adversarial training, especially on the effect of memorizing atypical examples. Based on the empirical observations, the authors propose benign adversarial training (BAT), which is evaluated on CIFAR-100 and Tiny ImageNet.", "summary_of_the_review": "Interesting observations on the memorization effects of atypical samples in adversarial training, but the empirical evaluations should involve stronger baselines (e.g., using early-stop) and adaptive attacks (e.g., considering the BAT modules), while it would be much more informative to have, e.g., AA results on CIFAR-10.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "OffFEsa9y41", "reviewer": "Reviewer_sm19", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The premise of the paper is interesting. While previous work has discussed \"atypical examples\" in the context of adversarial training, this is (to the best of my knowledge) the first work that performs such a study in a rigorous manner by considering proper measures of atypicality through influences. The findings are thus not always surprising given prior work, but are nevertheless interesting and the proposed method does show promise.\n\nThere are however a few issues that I would like to point out:\n\n- The theoretical setting is actually quite similar to the one in [Sanyal et al.]([https://arxiv.org/abs/2007.04028](https://arxiv.org/abs/2007.04028)) (cited). Their focus is on label noise instead of atypical examples (which is not a very different setting after all), but both analyses boil down to robustly classifying samples from a Gaussian in the presence of outliers. This does diminish the impact of the paper somewhat. In any case, the discussion of that paper in the manuscript should be expanded.\n- The discussion around \"poisoning atypical examples\" (e.g., Figure 3) is intriguing but somewhat speculative. Is there a more rigorous way to explore this phenomenon? E.g., by choosing samples with high influence and verifying them through human annotation?\n- It is not entirely clear why the discrimination loss regularization would be beneficial in this setting. The authors do provide a justification based on the class margin in representation space. However, since the model is trained end-to-end, why wouldn't standard adversarial training discover this solution?\n\nMinor comments (not affecting score):\n\n- The title is confusing, what does \"towards the ... effect\" mean?\n- [Balaji et al. 2019]([https://arxiv.org/abs/1910.08051](https://arxiv.org/abs/1910.08051)) is quite relevant and worth discussing.\n- Why are the PGD-generated adversarial examples during adversarial training referred to as \"manually generated\"?\n- Figure 4 is hard to parse in 3-D. It seems that a table would provide the same information in a more readable format.", "summary_of_the_paper": "The authors study the interaction between adversarial training (meant to produce robust models) and atypical examples in standard datasets (examples where generalization is driven by a handful of examples, cf [Feldman 2020]). They find that, in contrast to standard training, adversarial training has a hard time generalizing to atypical test examples in the adversarial setting. They also find that adding atypical examples to adversarial training can actually hurt test accuracy, in contrast to standard training where such examples typically help. Finally, the authors propose a method that down-weights  atypical examples during robust training and introduces a contrastive regularizer, which they term BAT (benign adversarial training).", "summary_of_the_review": "Overall, despite its shortcomings, the paper presents some findings that would be of interest to the community.\n\n--- \n\nPost-rebuttal update: Given the authors' clarification of their theoretical contributions and the provided small-scale user study, I increase my score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2Noju9nGZ7", "reviewer": "Reviewer_iXiX", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n1. This paper makes a connection between the memorization effect of DNN with adversarial training and provides some comprehensive experimental results on analyzing the effect of training data for adversarial robustness. And this study shows a promising perspective for adversarial robustness, which focuses on the data perspective. \n2. The proposed benign adversarial training gains consistent improvement (at least using the two benchmark datasets) on both natural and robust performance across different evaluations (including the strong attack method---AA). \n\nCons:\n\n1. The major concern is the novelty of the main contributions. The similar \"important\" findings in this paper have already been pointed out and analyzed by previous studies [1,2,3]. For example, in [1], the authors also investigated memorization behavior in adversarial training and have pointed out that overfitting some rare examples in training can just improve the clean accuracy, and ignoring these rare examples helps in adversarial robustness. And in [3], the authors show that using high-quality (not so rare or hard) examples can indeed improve both natural performance and adversarial robustness, which has a consistent high-level idea with benign adversarial training.\n2. Some claims are not well-supported. It may be better to add some citations to justify or provide some empirical results, especially for the memorization behavior in adversarial training. For example, how to confirm that these atypical samples have deviated from the main sub-population? Whether DNN can indeed fit every atypical samples' labels since we know that the model capacity for adversarial training is always not enough (in other words, adversarial training may require DNNs with a large model capacity than standard training [4]).\n3. The motivation for exploring or connecting memorization with adversarial training is not clear and not strong (At the top of page 2). The recent study mainly focuses on the \"overfitting\" phenomena in adversarial training, so what's the relationship of memorization in adversarial training with such \"overfitting\" phenomena? (Personally, since there seems to be a logic gap, it is hard to understand why we need to explore the memorization behavior in adversarial training).\n4. The reweighting strategy may need further explanation or illustration. \n5. What's the relationship of the proposed method with adopting curriculum learning in adversarial training?\n6. Empirically, if the data distribution is long-tail, how can we apply the benign adversarial training? Can benign adversarial training also effectively utilize the long-tailed data since it seems to be similar to the atypical samples mentioned in this paper?\n\nMinor:\n\nIt is better to use $f$ instead of $F$ in all the equations to appropriately indicate the loss function. \n\nReferences:\n\n[1] Sanyal, Amartya, Puneet K. Dokania, Varun Kanade, and Philip Torr. \"How Benign is Benign Overfitting?.\" In *International Conference on Learning Representations*. 2020.\n\n[2] Dong, Yinpeng, et al. \"Exploring Memorization in Adversarial Training.\" *arXiv preprint arXiv:2106.01606* (2021).\n\n[3] Dong, Chengyu, Liyuan Liu, and Jingbo Shang. \"Data Profiling for Adversarial Training: On the Ruin of Problematic Data.\" arXiv preprint arXiv:2102.07437 (2021).\n\n[4] Wu, Boxi, et al. \"Do Wider Neural Networks Really Help Adversarial Robustness?.\" *arXiv preprint arXiv:2010.01279* (2020).", "summary_of_the_paper": "This paper explores the \"memorization\" in adversarial training. Through some empirical experiments and theoretical analysis, this paper points out two findings: (a) memorizing atypical samples can improve DNN’s accuracy on clean atypical samples, but hardly improve their adversarial robustness and (b) memorizing some atypical samples can even hurt the DNN’s performance on typical samples. Based on these findings, the authors propose a method called benign adversarial training by reweighting. Empirical results show that it can achieve better clean accuracy or robustness than some baseline methods in CIFAR-100 and Tiny-ImageNet.", "summary_of_the_review": "Overall, this paper explores the memorization effect in adversarial training and proposes benign adversarial training which further enhances adversarial training. However, although the direction is promising, the major concern is about the novelty of the main contribution since the similar findings in this paper have already been pointed out and analyzed in previous studies [1,2,3], and simply dropping some called \"atypical\" data can also improve adversarial training [3]. Considering the similar results from the perspective of data quality for adversarial robustness,  the current submission may need some further exploration to provide some different and in-depth findings.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0sr9vuvv0zJ", "reviewer": "Reviewer_qCTN", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Interesting observations and analysis provided on the behavior of atypical samples in the context of robust overfitting \n- Promising empirical results showing improved clean accuracy at similar or better levels of robustness with strong attacks\n- Analysis of the effect of individual components in ablation studies\n- Paper is well-written and easy to follow\n\nWeaknesses:\n- The choice of threshold for mem(x) used to define an atypical sample seems arbitrary. How do the results vary with different thresholds for mem(x) used to define the atypical set?  \n- Even though the paper is initially motivated by atypical samples, the mechanism relating atypical-ness to adversarial robustness is not clear and moreover it seems to me that the analysis in \"Poisoning Atypical Samples\" and the algorithm design that follows can be better understood from the perspective of margins and adversarial robustness; examples with smaller margins (closer to the decision boundary) are necessarily more susceptible to adversarial attack. In particular:\n  * Theorems 1/2 are not specific to the notion of atypical samples, and are about samples near the decision boundary (low margin). These theorems are essentially saying that by trying to accomodate these samples near the decision boundary for an optimal classifier, the model will be less robust.\n  * The two components of BAT involve other elements that are not related to atypicalness but rather margin - the reweighting involves a poisoning score that is essentially the margin of the example (\"likely to be close to the distribution of a wrong class\"), and the discrimination loss may be viewed as attempts to increase the margin between the different classes.\n\n  It would be useful to isolate the effect and importance of the atypical-ness of an example in the algorithm, for instance by applying the algorithm to all examples not just the atypical ones.\n\nOther comments/suggestions: \n- Metric learning was previously used for adversarial robustness in Metric Learning for Adversarial Robustness, NeurIPS 2019 though in a different way\n- In light of the relation to margin, some related work from the margin perspective that could be discussed include \n  * Helper-based Adversarial Training: Reducing Excessive Margin to Achieve a Better Accuracy vs. Robustness Trade-off, ICML Workshop on Adversarial ML 2021\n  * MMA Training: Direct Input Space Margin Maximization through Adversarial Training, ICLR 2020\n  \n  In particular, the first paper achieves similar results as the current work by restricting the margin increase through different means; the downweighting of low (or even negative margin) examples in BAT has a similar effect.\n\n- Typos:\n  * Sec 3.2 on p4: trails -> trials\n  * Theorem 1: the condition $d_{+1} \\leq 2\\eta$ should be on $d_{-1}$ instead\n", "summary_of_the_paper": "This paper studies the role of atypical samples in the context of adversarial robustness, and further introduces a variant of adversarial training based on these observations. The proposed Benign Adversarial Training (BAT) incorporates an additional temperature-scaled n-pair loss and weights adversarial examples according to their margin. BAT shows improved clean accuracy at similar levels of robustness compared to baselines.", "summary_of_the_review": "Overall the proposed BAT algorithm achieves promising results that improve clean accuracy at similar robustness levels, but the design of the algorithm seems to involve elements that are orthogonal from the initial motivation and study of atypical examples. The importance of the atypical-ness of an example should be clarified during the discussion period to better motivate the method. \n\n**Post response update:** \nThe authors have comprehensively addressed my concerns about relation to margin and importance of the atypicality to the components of BAT through ablation studies. I have increased my score accordingly.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper tries to improve the training of adversarial deep neural networks by avoiding fitting the “harmful” atypical samples and fitting more “benign” atypical samples. 

Overall, the main concerns are

1. The current presentation can easily cause some misunderstandings on the observations made in Section 3, especially [1] and [3] mentioned by the reviewer iXiX. 
- The authors may consider moving "related work" to the first half of the submission, and organize existing findings with rare/hard/atypical in a more principle manner. 
- Besides, as author mentioned in Section 3.1: "it is equivalent to a classification task based on an extremely small dataset, with one or a few training samples given". Such findings are natural and not novel to the deep learning community. Authors may consider shorten Section 3.1 and elaborate more in Section 3.2.

2. Theorem 1 and 2 do not help much. 
- It does not talk about the training algorithm and models, which over simplifies the learning problem. 
- Besides, the authors can consider some theoretical results how BAT improve the performance of typical samples, but still preserve the ability to fit those "useful" atypical samples. 
This helps to bridge the gap between motivation behind BAT and its algorithm design (raise by reviewer  ytJj and sm19).

3. It is also suggested to make observations more convincing. 
- Since authors want to claim their findings are universal, it is better to consider more adv training methods and datasets; it is also better to change the ratio of "normal samples" v.s. "atypical samples". In this way, the effect of atypical samples in adversarial training can be more carefully quantized.
Decision: Reject