Data Quality Matters For Adversarial Training: An Empirical Study

1 INTRODUCTION . Adversarial training ( Goodfellow et al. , 2015 ; Huang et al. , 2015 ; Kurakin et al. , 2017 ; Madry et al. , 2018 ) is arguably the most effective way to establish the robustness of deep neural networks against adversarial perturbations . Nevertheless , several intriguing problems and phenomena hover in adversarial training practice , including ( 1 ) robust overfitting ( Rice et al. , 2020 ) , ( 2 ) robustness overestimation1 ( Uesato et al. , 2018 ; Mosbach et al. , 2018 ; Croce & Hein , 2020b ; Chen & Gu , 2020 ) , and ( 3 ) robustness-accuracy trade-off ( Papernot et al. , 2016 ; Su et al. , 2018 ; Tsipras et al. , 2019 ; Zhang et al. , 2019 ) . In this work , we focus on the data in adversarial training and show that the three aforementioned problems are all interconnected with the quality of the data employed in adversarial training . Specifically , we measure the data quality based on their stability of being learned through training and show that unstably learned examples are of low quality to adversarial training . We show that low-quality data can be one common cause of the three aforementioned problems . To the best of our knowledge , this is the first time the effect of the data quality being systematically studied in adversarially robust learning . This is also the first time that all these problems are investigated in a holistic manner . As a demonstration , we partition the training set of CIFAR-10 ( Krizhevsky , 2009 ) into two exclusive subsets with equal size and balanced classes , while ensuring all the examples in one subset have higher quality than all the examples in the other one in a class-wise manner . We then conduct both standard training and adversarial training with Projected Gradient Descent ( PGD-AT ) ( Madry et al. , 2018 ) . The model is fixed as pre-activation ResNet-18 ( He et al. , 2016 ) . As shown in Figure 1 , adversarial training on the high-quality half yields high robustness , almost no robust overfitting , genuine robustness against the popular PGD attack ( Madry et al. , 2018 ) , and minor robustnessaccuracy trade-off that is only noticeable in the late stage of training , which is in sharp contrast to that on the low-quality half . We further conduct controlled experiments on three real-world datasets , CIFAR-10 , CIFAR100 ( Krizhevsky , 2009 ) and Tiny-ImageNet ( Le & Yang , 2015 ) , to explore the interconnection 1Robustness overestimation refers to the problem that the adversarial robustness may be spuriously high against certain types of adversaries . between data quality and those three problems in adversarial training . Our experiments cover various sample sizes , training methods ( e.g. , PGD-AT and TRADES ( Zhang et al. , 2019 ) ) , and neural architectures ( e.g. , pre-activation ResNet and Wide ResNet ( WRN ) ( Zagoruyko & Komodakis , 2016 ) ) . Our major findings are summarized as follows . • Low-quality data may not be useful or even detrimental to adversarial training . • Low-quality data causes robust overfitting in adversarial training . • Adversarial training with low-quality data can lead to overestimated robustness against certain adversaries such as the PGD attack . • Robustness-accuracy trade-off is only prominent when the dataset contains low-quality data . When using high-quality data only , the standard accuracy achieved by adversarial training and standard training is comparable . The remainder of this paper is organized as follows . In Section 2 , we briefly review the existing works that are most related to our data-centric investigation of adversarial training . In Section 3 , we propose a strategy to estimate the data quality based on learning stability , and show that unstablylearned data are of low quality to adversarial training . Sections 4.1 , 4.2 and 4.3 investigate the interconnection between data quality and major problems in adversarial training including robust overfitting , robustness overestimation , and robustness-accuracy trade-off respectively . Conclusions and further implications are discussed in Section 5 . 2 RELATED WORK . Increasing attention has recently been paid to analyzing and improving the quality of the ( training ) data . Compared with standard learning , only a limited number of all learnable features would be beneficial to the adversarial robustness ( Ilyas et al. , 2019 ) , thus making data quality more important in adversarially robust learning . Existing works have been focusing on the effect of sample size and data distribution in adversarial training . Schmidt et al . ( 2018 ) show that adversarially robust learning requires a much higher sample complexity than standard learning . They also show that such sample complexity requirement is highly sensitive to the data distribution . Ding et al . ( 2019 ) further observe that adversarial robustness achieved by adversarial training is sensitive to semantically-lossless distribution shift . Shafahi et al . ( 2019 ) show that the adversarial robustness achieved by any classifier is fundamentally limited by the properties of the data distribution . Our study is distinguished from previous works by investigating the datasets with the same size and within the same distribution . We also systematically study the other important problems in adversarial training beyond the performance . Towards advancing adversarial robustness , substantially more works have been devoted to customizing the inner maximization or outer minimization in adversarial training based on the properties and behaviors of training data . Representative works including Wang et al . ( 2020 ) , which shows that misclassified examples have greater impacts on adversarial robustness , and Zhang et al . ( 2020b ) , which observes that some training data are more robust to adversarial attack and will significantly improve the robustness against certain types of attacks if they are emphasized in adversarial training . Following a curriculum adversarial training framework ( Cai et al. , 2018 ) , Wang et al . ( 2019 ) proposes to gradually increase the convergence quality of adversarial examples through training by observing that training on adversarial examples with high convergence quality in the later stages leads to better robustness . Selecting high-quality data may be more crucial when including extra data into training that are either unlabeled ( Uesato et al. , 2019 ; Carmon et al. , 2019 ; Gowal et al. , 2020 ) or generated by generative models ( Sehwag et al. , 2021 ; Gowal et al. , 2021 ) , where the predictive probability of a model is comomly used to measure the data quality . We review more works on the different measures of data quality and their applications in Appendix D. Our study is distinguished from these works by both methodology and objective . In terms of the methodology , we identify the low-quality examples based on their behaviors of being learned throughout the adversarial training . Our approach to measure data quality yields more fine-grained results than misclassification to distinguish the individual examples , and more consistent results than the resistance to perturbation or predictive probability among different training methods and neural architectures ( see Appendix D ) . In terms of the objective , we are not aiming to compete with the state-of-the-art methods , but rather to reveal the important influence of data quality on the problems in adversarial training through rigorously controlled experiments and reliable robustness evaluation . Such comparative and systematic analyses are lacking in the current adversarially robust learning research . Due to the space constraint , we will discuss more works specifically related to the benefit of data in adversarially robust learning , robust overfitting , robustness overestimation , and robustness-accuracy trade-off when it comes to Sections 3.2 , 4.1 , 4.2 and 4.3 , respectively . More broadly related work regarding data quality in standard and robust learning will be discussed in Appendix A . 3 DATA QUALITY IN ADVERSARIAL TRAINING . 3.1 DATA QUALITY MEASURE : LEARNING STABILITY AS AN EXAMPLE . We propose to measure the quality of the data in adversarial training based on the frequency of events that an example is predicted correctly under adversarial perturbation throughout the training , which we will refer as the learning stability in the rest of the paper . Specifically , given an example x , the model f and the optimizer ω , the learning stability s ( x ; f , ω ) can be formulated as s ( x ; f , ω ) = 1 T | { t|f ( t , x+ δ ) = y ( x ) , t ∈ { 1 , · · · , T } } | , ( 1 ) where δ is a ` ∞ norm-bounded adversarial perturbation , f ( t , · ) denotes the classifier at epoch t , y ( x ) denotes the true label of example x , and T is the total number of epochs the model will be trained . A similar measure in standard training has been utilized by Moon et al . ( 2020 ) to regularize the confidence estimates of deep neural networks , and is shown to be roughly proportional to the probability that an example will be correctly predicted by a model . We find that , although the absolute value of the learning stability can vary a lot ( see Appendix C.2 ) , the ranking of examples based on the learning stability is consistent across different experimental setups , such as random initializations , training epochs , training methods , and neural architectures as shown in Figure 2 . This suggests that the learning stability can reflect the intrinsic “ hardness ” of an example in a relative manner . Towards a more accurate estimation of such hardness , we calculate the ranking of the learning stability based on the ensemble of multiple experiments ( see Appendix C.1 ) . Figure 3a and Figure 3b show samples of the examples with the lowest learning stability and the examples with the highest learning stability , respectively , in the CIFAR-10 training set . One can find that unstably-learned examples appear more ambiguous and may not align well with their given labels . Under adversarial perturbation with relatively small size , they can not easily be identified even by humans . This aligns with our intuition that unstably-learned examples may be intrinsically hard . In standard learning , hard examples are often essential to the model performance , which is manifested in real-world applications such as hard example mining ( Kumar et al. , 2010 ; Shrivastava et al. , 2016 ) and ablation studies on the effect of removing hard examples ( Toneva et al. , 2019 ) . Empirical evidence can also be found in our Figure 1 ( c ) where standard training on the unstably-learned half yields a higher standard accuracy than that on the stably-learned half . In adversarial training , rather surprisingly , we observe an opposite case . Unstably-learned examples are often not useful and may even hurt the robustness achieved by adversarial training ( see Section 3.2 ) . Therefore , in the rest of the paper , we refer those unstably-learned examples as low-quality examples , and measure the data quality relatively using the learning stability . It is worth mentioning that , in Appendix D , we show that it is possible to estimate the data quality based on other behaviors and properties of the data ( e.g. , predictive probability , minimum perturbation , and learning order ) and all these estimations will yield consistent findings . 3.2 UNSTABLY-LEARNED DATA IS OF LOW-QUALITY FOR ADVERSARIAL TRAINING . In this section , we conduct controlled experiments on three real-world datsets and show that lowquality examples are not useful and may even be detrimental to adversarial training . Related work . In adversarially robust learning , solely increasing sample size will not always yield better performance and sometimes even be detrimental . Ding et al . ( 2019 ) observes that the robustness achieved by adversarial training plateaus after training size is sufficiently large on MNIST ( LeCun et al. , 1998 ) . Introducing additional data examples may not always help the adversarial robustness if they are unrelated to the original dataset ( Uesato et al. , 2019 ; Gowal et al. , 2020 ) . Yang et al . ( 2020a ) shows that a proper pruning of the dataset will benefit the adversarial robustness for non-parametric classifiers . A recent study also shows that adversarial vulnerability may originate from the label noise in the dataset ( Sanyal et al. , 2020 ) , which apparently hurts the adversarial robustness . However , we note that low-quality data is not simply label noise after manually inspecting the samples as those in Figure 3a . Moreover , in-depth studies show that the fractions of label noise in benchmark datasets are only marginal ( Northcutt et al. , 2021 ) . Experimental setup of controlled experiments . We conduct controlled experiments to dissect the effect of training data on adversarial training . Specifically , we gradually remove more examples from the training set and record the robustness achieved by adversarial training from scratch on an increasingly smaller training subset , while all other experiment settings remain the same . The training examples are removed either randomly , or in an ascending order of data quality , i.e. , low-quality data will be removed first . By removing the same number of random data and low-quality data , it should be able to eliminate the effect of sample size in the comparison and reveal the effect of data quality . We consider the robustness against ` ∞ norm-bounded adversarial attack with perturbation radius 8/255 throughout the paper . We evaluate the adversarial robustness against AutoAttack ( AA ) ( Croce & Hein , 2020b ) , one of the adversarial attacks known to be relatively reliable . We employ popular adversarial training methods including PGD-AT and TRADES combined with early stopping ( Rice et al. , 2020 ) , which have been shown to attain the improvements of almost all other adversarial training variants ( Croce & Hein , 2020b ; Chen & Gu , 2020 ; Pang et al. , 2021 ) . We conduct experiments with pre-activation ResNet-18 in the main paper and Wide ResNet in the appendix . We present results on three datasets including CIFAR-10 , CIFAR-100 and Tiny-ImageNet . Other experiment details can be found in Appendix F. The same experimental setup will be applied to Sections 4 , 5 and 6 as well . Results & Discussions . Figure 4 shows that removing random examples from the training set degrades the robustness achieved by adversarial training rapidly . In contrast , when removing low-quality examples first , the achieved robustness maintains or even increases . For instance , by PGD-AT , up to 40 % , 60 % , and 50 % low-quality training examples can be removed without reducing the robustness significantly ( < 1 % ) on CIFAR-10 , CIFAR-100 , and Tiny-ImageNet , respectively . More surprisingly , removing examples with the lowest data quality can achieve robustness even higher than that achieved on the entire training set for both PGD-AT and TRADES on CIFAR-10 , and TRADES for other datasets , although the training set is actually becoming smaller . We also validate the above observations with different perturbation sizes and neural achitectures in Appendix B.2 .

Summary: The paper investigates the impact of data quality, measured by the proportion of epochs in which the model classifies a specific input correctly during training, on the robustness, generalization, and robustness-accuracy tradeoff of adversarially trained models.  Unlike with standard training, the authors find that more difficult inputs (lower quality inputs) can hurt adversarially trained models.  They find that compared to randomly removing data during training, removing low quality data can lead to higher robustness, less robust overfitting, less robustness overestimation, and less robustness-accuracy tradeoff.