Towards the Memorization Effect of Neural Networks in Adversarial Training

1 INTRODUCTION . It is evident from recent studies that the memorization effect ( or benign overfitting ) ( Feldman , 2020 ; Feldman & Zhang , 2020 ; Bartlett et al. , 2020 ; Chatterji & Long , 2020 ; Muthukumar et al. , 2020 ) is one necessary factor for overparametrized deep neural networks ( DNNs ) to achieve the close-tooptimal generalization error . From the empirical perspective , the works ( Feldman , 2020 ; Feldman & Zhang , 2020 ) suggest that modern benchmark datasets , such as CIFAR10 , CIFAR100 ( Krizhevsky et al. , 2009 ) and ImageNet ( Krizhevsky et al. , 2012 ) , always have very diverse data distributions , especially containing a large fraction of “ atypical ” samples . These atypical samples are both visually and statistically very different from other samples in their labeled class . For example , the images in the class “ bird ” may have a variety of sub-populations or species , with many ( typical ) samples in the main sub-population and other ( atypical ) samples in less-frequent and distinct sub-populations . Since these atypical samples are deviated from the main sub-population , DNNs can only fit these atypical samples by “ memorizing ” their labels . While , memorizing / fitting these atypical samples will not hurt the model performance on typical samples , but can boost DNNs ’ accuracy by correctly classifying the atypical samples appearing in the test set . Similar to the classification models which are trained via empirical risk minimization ( ERM ) algorithms , adversarial training methods ( Madry et al. , 2017 ; Kurakin et al. , 2016 ) are also devised to fit the whole training dataset . Specifically , adversarial training minimizes the model ’ s error against adversarial perturbations ( Goodfellow et al. , 2014 ; Szegedy et al. , 2013 ) by fitting the model on manually generated adversarial examples of all training data . Although adversarial training can fit and memorize all training data as well as their adversarially perturbed counterparts , they always suffer from both poor clean accuracy and adversarial accuracy ( or robustness ) 1 on the test set ( Tsipras et al. , 1Clean Accuracy : Model ’ s accuracy on unperturbed samples . Adversarial accuracy ( robustness ) : Model ’ s accuracy on adversarially perturbed samples . Without loss of generality , this paper discusses the adversarial accuracy under l∞-8/255 PGD attack ( Madry et al. , 2017 ) . 2018 ; Schmidt et al. , 2018 ) . Recent study ( Rice et al. , 2020 ) indicates that , during the adversarial training process , one model ’ s test adversarial accuracy will even keep decreasing as it hits higher training adversarial accuracy ( during the fine-tuning epochs ) . Thus , it is natural to ask a question : What is the effect of memorization in adversarial training ? In particular , can memorizing the atypical samples and their adversarial counterparts benefit the model ’ s accuracy and robustness ? To answer this question , we first conduct preliminary studies to explore whether memorizing atypical samples in adversarial training can benefit the DNNs ’ test performance , especially on those test atypical samples . In Section 3.1 , we implement PGD adversarial training ( Madry et al. , 2017 ) on CIFAR100 under WideResNet models ( He et al. , 2016 ) and fine-tune them until achieving the optimal training performance . From the results in Section 3.1 , we observe that the memorization in adversarial training can only benefit the clean accuracy of test atypical samples . When the DNNs gradually fit/memorize more atypical samples , they can finally achieve fair clean accuracy close to ∼ 40 % on the test atypical set . However , the adversarial accuracy on test atypical set is constantly low ( ∼ 10 % ) during the whole training process , even though the models can fit almost all atypical ( adversarial ) training samples . Based on the theoretical study ( Schmidt et al. , 2018 ) , the adversarial robustness is hard to generalize especially when the training data size is limited . Since every single atypical sample is distinct from the main sub-population and rarely appears in the training set , the data complexity for each specific atypical sample is very low . Thus , its adversarial robustness can be extremely hard to generalize . Notably , for datasets such as CIFAR100 , the entire atypical set covers at least ∼ 40 % samples of the whole dataset . Therefore , completely failing on atypical samples could be one of the key reasons leading to the poor robustness generalization of DNNs . Furthermore , we find that in adversarial training , fitting atypical samples will even hurt DNNs ’ performance on those “ typical ” samples ( the samples in the main sub-population ) . In the Section 3.2 , we again implement PGD adversarial training ( Madry et al. , 2017 ) on CIFAR100 for several trails , which are trained with different amount of atypical samples . Based on the results from Section 3.2 , an adversarially trained model on the training set without any atypical samples has 95 % clean accuracy and 55 % adversarial accuracy on the test typical set . While , the model trained with 100 % atypical samples only has 90.2 % /50.4 % clean/adversarial accuracy respectively . In other words , atypical samples act more like “ poisoning data ” ( Biggio et al. , 2012 ) to deteriorate model performance on typical samples . Furthermore , our study in Section 3.2 also demonstrates that this poisoning effect is absent in traditional ERM , where fitting atypical samples will not reduce the model accuracy . To deepen our understanding on this finding , we build a theoretical analysis based on Gaussian mixture models . We prove that , given certain atypical samples , any model which fits its adversarial counterpart ( adv . training ) must have a poor accuracy on typical samples . While , under the same setting , the models only fitting the clean version of this atypical sample ( traditional ERM ) can achieve optimal accuracy ( i.e . > 99 % ) . Our empirical and theoretical results highlight the key difference between the effect of memorization in adversarial training and traditional ERM . Motivated by our findings , we propose a novel algorithm called Benign Adversarial Training ( BAT ) , which can eliminate the negative influence from memorizing those “ poisoning ” atypical samples , meanwhile preserving the model ’ s ability to memorize those “ benign / useful ” atypical samples . It is worth mentioning that , by fitting those “ benign ” atypical samples , the BAT method can achieve good clean accuracy on the atypical set ; by eliminating those poisoning atypical samples , the BAT method can improve the clean & adv . accuracy on the typical set . Compared with PGD adversarial training ( Madry et al. , 2017 ) when it achieves the highest adversarial robustness , BAT has higher clean accuracy as well as adversarial accuracy . Compared to other popular variants of adversarial training such as ( Zhang et al. , 2019 ; Wang et al. , 2019 ; Zhang et al. , 2020 ) , BAT is the only one obtaining both better ( or comparable ) clean and adversarial accuracy than ( Madry et al. , 2017 ) , on complex datasets such as CIFAR100 and Tiny Imagenet ( Le & Yang , 2015 ) . 2 DEFINITION AND NOTATION . 2.1 ATYPICAL SAMPLES AND MEMORIZATION . In this section , we start by introducing necessary concepts and definitions about the memorization effects . As well known , the practice of training deep neural networks ( DNNS ) can not be well explained by standard theories about model generalization ( Evgeniou et al. , 2000 ; Bartlett & Mendelson , 2002 ) . At a high level , the standard theories underline the importance of regularization on the model complexity , to avoid “ overfitting ” the outliers and nonuseful samples . While , for DNNs , we always tune the model to hit almost perfect training accuracy and enjoy good test performance . Fortunately , recent works ( Feldman , 2020 ; Feldman & Zhang , 2020 ; Bartlett et al. , 2020 ; Muthukumar et al. , 2020 ) make significant progress to close this gap from both theoretical and empirical perspectives . They suggest that the memorization is one key property for DNNs to achieve optimal generalization performance . In detail , the works ( Feldman , 2020 ; Feldman & Zhang , 2020 ) point out that the common benchmark datasets , such as CIFAR10 , CIFAR100 , ImageNet , contain a large portion of atypical samples ( or namely , rare samples , sub-populations , etc. ) . These atypical samples look very different from the other samples in the main distribution of its labeled class , and are statistically indistinguished from outliers or mislabeled samples . DNNs can only fit these samples by memorizing their labels . Moreover , without memorizing these atypical samples during training , the DNNs can totally fail to predict the atypical samples in the test set ( Feldman & Zhang , 2020 ) . Identify Atypical Samples . To identify such atypical samples in common datasets in practice , one representative strategy ( Feldman & Zhang , 2020 ) proposes to examine which training samples can only be fitted by memorization , and measure each training sample ’ s “ memorization value ” . Formally , for a training algorithmA ( i.e. , ERM ) , the memorization value “ mem ( A , D , xi ) ” of a training sample ( xi , yi ) ∈ D in training set D is defined as : mem ( A , D , xi ) = Pr . F←A ( D ) ( F ( xi ) = yi ) − Pr . F←A ( D\xi ) ( F ( xi ) = yi ) , ( 1 ) which calculates the difference between the model F ’ s accuracy on xi with and without xi removed from the training set D. In practice , ( Feldman & Zhang , 2020 ) trains a DNN model using ERM method for 1,000 trials , with each trial preserves 70 % samples of the whole training dataset . Based on this metric , one can identify all common datasets having a large fraction of atypical samples . For example , CIFAR10 , CIFAR100 and Tiny ImageNet have more than 11 % , 40 % and 49 % samples with a large memorization value > 0.15 . The similar strategy can also facilitate to find atypical samples in the test set , which are the samples that are strongly influenced by atypical training samples . In detail , by removing an atypical training sample ( xi , yi ) , we calculate its “ influence value ” on each test sample ( x′j , y ′ j ) ∈ D′ in test set D′ : infl ( A , D , xi , x′j ) = Pr . F←A ( D ) ( F ( x′j ) = y ′ j ) − Pr . F←A ( D\xi ) ( F ( x′j ) = y ′ j ) . ( 2 ) If the sample pair ( xi , x′j ) has a high influence value , removing the atypical sample xi will drastically decrease the model ’ s accuracy on x′j . In Appendix A , we provide more detailed discussions on the memorization effect and atypical samples , containing : ( a ) the distribution and frequency of atypical samples in common datasets and ( b ) alternative ( more efficient ) metrics to identify atypical samples , such as confidence-based method and ensemble disagreement ( Carlini et al. , 2019 ) .

Summary: The authors study the interaction between adversarial training (meant to produce robust models) and atypical examples in standard datasets (examples where generalization is driven by a handful of examples, cf [Feldman 2020]). They find that, in contrast to standard training, adversarial training has a hard time generalizing to atypical test examples in the adversarial setting. They also find that adding atypical examples to adversarial training can actually hurt test accuracy, in contrast to standard training where such examples typically help. Finally, the authors propose a method that down-weights  atypical examples during robust training and introduces a contrastive regularizer, which they term BAT (benign adversarial training).