Towards the Memorization Effect of Neural Networks in Adversarial Training

1 INTRODUCTION . It is evident from recent studies that the memorization effect ( or benign overfitting ) ( Feldman , 2020 ; Feldman & Zhang , 2020 ; Bartlett et al. , 2020 ; Chatterji & Long , 2020 ; Muthukumar et al. , 2020 ) is one necessary factor for overparametrized deep neural networks ( DNNs ) to achieve the close-tooptimal generalization error . From the empirical perspective , the works ( Feldman , 2020 ; Feldman & Zhang , 2020 ) suggest that modern benchmark datasets , such as CIFAR10 , CIFAR100 ( Krizhevsky et al. , 2009 ) and ImageNet ( Krizhevsky et al. , 2012 ) , always have very diverse data distributions , especially containing a large fraction of “ atypical ” samples . These atypical samples are both visually and statistically very different from other samples in their labeled class . For example , the images in the class “ bird ” may have a variety of sub-populations or species , with many ( typical ) samples in the main sub-population and other ( atypical ) samples in less-frequent and distinct sub-populations . Since these atypical samples are deviated from the main sub-population , DNNs can only fit these atypical samples by “ memorizing ” their labels . While , memorizing / fitting these atypical samples will not hurt the model performance on typical samples , but can boost DNNs ’ accuracy by correctly classifying the atypical samples appearing in the test set . Similar to the classification models which are trained via empirical risk minimization ( ERM ) algorithms , adversarial training methods ( Madry et al. , 2017 ; Kurakin et al. , 2016 ) are also devised to fit the whole training dataset . Specifically , adversarial training minimizes the model ’ s error against adversarial perturbations ( Goodfellow et al. , 2014 ; Szegedy et al. , 2013 ) by fitting the model on manually generated adversarial examples of all training data . Although adversarial training can fit and memorize all training data as well as their adversarially perturbed counterparts , they always suffer from both poor clean accuracy and adversarial accuracy ( or robustness ) 1 on the test set ( Tsipras et al. , 1Clean Accuracy : Model ’ s accuracy on unperturbed samples . Adversarial accuracy ( robustness ) : Model ’ s accuracy on adversarially perturbed samples . Without loss of generality , this paper discusses the adversarial accuracy under l∞-8/255 PGD attack ( Madry et al. , 2017 ) . 2018 ; Schmidt et al. , 2018 ) . Recent study ( Rice et al. , 2020 ) indicates that , during the adversarial training process , one model ’ s test adversarial accuracy will even keep decreasing as it hits higher training adversarial accuracy ( during the fine-tuning epochs ) . Thus , it is natural to ask a question : What is the effect of memorization in adversarial training ? In particular , can memorizing the atypical samples and their adversarial counterparts benefit the model ’ s accuracy and robustness ? To answer this question , we first conduct preliminary studies to explore whether memorizing atypical samples in adversarial training can benefit the DNNs ’ test performance , especially on those test atypical samples . In Section 3.1 , we implement PGD adversarial training ( Madry et al. , 2017 ) on CIFAR100 under WideResNet models ( He et al. , 2016 ) and fine-tune them until achieving the optimal training performance . From the results in Section 3.1 , we observe that the memorization in adversarial training can only benefit the clean accuracy of test atypical samples . When the DNNs gradually fit/memorize more atypical samples , they can finally achieve fair clean accuracy close to ∼ 40 % on the test atypical set . However , the adversarial accuracy on test atypical set is constantly low ( ∼ 10 % ) during the whole training process , even though the models can fit almost all atypical ( adversarial ) training samples . Based on the theoretical study ( Schmidt et al. , 2018 ) , the adversarial robustness is hard to generalize especially when the training data size is limited . Since every single atypical sample is distinct from the main sub-population and rarely appears in the training set , the data complexity for each specific atypical sample is very low . Thus , its adversarial robustness can be extremely hard to generalize . Notably , for datasets such as CIFAR100 , the entire atypical set covers at least ∼ 40 % samples of the whole dataset . Therefore , completely failing on atypical samples could be one of the key reasons leading to the poor robustness generalization of DNNs . Furthermore , we find that in adversarial training , fitting atypical samples will even hurt DNNs ’ performance on those “ typical ” samples ( the samples in the main sub-population ) . In the Section 3.2 , we again implement PGD adversarial training ( Madry et al. , 2017 ) on CIFAR100 for several trails , which are trained with different amount of atypical samples . Based on the results from Section 3.2 , an adversarially trained model on the training set without any atypical samples has 95 % clean accuracy and 55 % adversarial accuracy on the test typical set . While , the model trained with 100 % atypical samples only has 90.2 % /50.4 % clean/adversarial accuracy respectively . In other words , atypical samples act more like “ poisoning data ” ( Biggio et al. , 2012 ) to deteriorate model performance on typical samples . Furthermore , our study in Section 3.2 also demonstrates that this poisoning effect is absent in traditional ERM , where fitting atypical samples will not reduce the model accuracy . To deepen our understanding on this finding , we build a theoretical analysis based on Gaussian mixture models . We prove that , given certain atypical samples , any model which fits its adversarial counterpart ( adv . training ) must have a poor accuracy on typical samples . While , under the same setting , the models only fitting the clean version of this atypical sample ( traditional ERM ) can achieve optimal accuracy ( i.e . > 99 % ) . Our empirical and theoretical results highlight the key difference between the effect of memorization in adversarial training and traditional ERM . Motivated by our findings , we propose a novel algorithm called Benign Adversarial Training ( BAT ) , which can eliminate the negative influence from memorizing those “ poisoning ” atypical samples , meanwhile preserving the model ’ s ability to memorize those “ benign / useful ” atypical samples . It is worth mentioning that , by fitting those “ benign ” atypical samples , the BAT method can achieve good clean accuracy on the atypical set ; by eliminating those poisoning atypical samples , the BAT method can improve the clean & adv . accuracy on the typical set . Compared with PGD adversarial training ( Madry et al. , 2017 ) when it achieves the highest adversarial robustness , BAT has higher clean accuracy as well as adversarial accuracy . Compared to other popular variants of adversarial training such as ( Zhang et al. , 2019 ; Wang et al. , 2019 ; Zhang et al. , 2020 ) , BAT is the only one obtaining both better ( or comparable ) clean and adversarial accuracy than ( Madry et al. , 2017 ) , on complex datasets such as CIFAR100 and Tiny Imagenet ( Le & Yang , 2015 ) . 2 DEFINITION AND NOTATION . 2.1 ATYPICAL SAMPLES AND MEMORIZATION . In this section , we start by introducing necessary concepts and definitions about the memorization effects . As well known , the practice of training deep neural networks ( DNNS ) can not be well explained by standard theories about model generalization ( Evgeniou et al. , 2000 ; Bartlett & Mendelson , 2002 ) . At a high level , the standard theories underline the importance of regularization on the model complexity , to avoid “ overfitting ” the outliers and nonuseful samples . While , for DNNs , we always tune the model to hit almost perfect training accuracy and enjoy good test performance . Fortunately , recent works ( Feldman , 2020 ; Feldman & Zhang , 2020 ; Bartlett et al. , 2020 ; Muthukumar et al. , 2020 ) make significant progress to close this gap from both theoretical and empirical perspectives . They suggest that the memorization is one key property for DNNs to achieve optimal generalization performance . In detail , the works ( Feldman , 2020 ; Feldman & Zhang , 2020 ) point out that the common benchmark datasets , such as CIFAR10 , CIFAR100 , ImageNet , contain a large portion of atypical samples ( or namely , rare samples , sub-populations , etc. ) . These atypical samples look very different from the other samples in the main distribution of its labeled class , and are statistically indistinguished from outliers or mislabeled samples . DNNs can only fit these samples by memorizing their labels . Moreover , without memorizing these atypical samples during training , the DNNs can totally fail to predict the atypical samples in the test set ( Feldman & Zhang , 2020 ) . Identify Atypical Samples . To identify such atypical samples in common datasets in practice , one representative strategy ( Feldman & Zhang , 2020 ) proposes to examine which training samples can only be fitted by memorization , and measure each training sample ’ s “ memorization value ” . Formally , for a training algorithmA ( i.e. , ERM ) , the memorization value “ mem ( A , D , xi ) ” of a training sample ( xi , yi ) ∈ D in training set D is defined as : mem ( A , D , xi ) = Pr . F←A ( D ) ( F ( xi ) = yi ) − Pr . F←A ( D\xi ) ( F ( xi ) = yi ) , ( 1 ) which calculates the difference between the model F ’ s accuracy on xi with and without xi removed from the training set D. In practice , ( Feldman & Zhang , 2020 ) trains a DNN model using ERM method for 1,000 trials , with each trial preserves 70 % samples of the whole training dataset . Based on this metric , one can identify all common datasets having a large fraction of atypical samples . For example , CIFAR10 , CIFAR100 and Tiny ImageNet have more than 11 % , 40 % and 49 % samples with a large memorization value > 0.15 . The similar strategy can also facilitate to find atypical samples in the test set , which are the samples that are strongly influenced by atypical training samples . In detail , by removing an atypical training sample ( xi , yi ) , we calculate its “ influence value ” on each test sample ( x′j , y ′ j ) ∈ D′ in test set D′ : infl ( A , D , xi , x′j ) = Pr . F←A ( D ) ( F ( x′j ) = y ′ j ) − Pr . F←A ( D\xi ) ( F ( x′j ) = y ′ j ) . ( 2 ) If the sample pair ( xi , x′j ) has a high influence value , removing the atypical sample xi will drastically decrease the model ’ s accuracy on x′j . In Appendix A , we provide more detailed discussions on the memorization effect and atypical samples , containing : ( a ) the distribution and frequency of atypical samples in common datasets and ( b ) alternative ( more efficient ) metrics to identify atypical samples , such as confidence-based method and ensemble disagreement ( Carlini et al. , 2019 ) .

Summary: This paper studies the role of atypical samples in the context of adversarial robustness, and further introduces a variant of adversarial training based on these observations. The proposed Benign Adversarial Training (BAT) incorporates an additional temperature-scaled n-pair loss and weights adversarial examples according to their margin. BAT shows improved clean accuracy at similar levels of robustness compared to baselines.