On the Impact of Hard Adversarial Instances on Overfitting in Adversarial Training

1 INTRODUCTION . The existence of adversarial examples ( Szegedy et al. , 2014 ) causes serious safety concerns when deploying modern deep learning models . For example , for classification tasks , imperceptible perturbations of the input instance can fool state-of-the-art classifiers . Many strategies to obtain models that are robust against adversarial attacks have been proposed ( Buckman et al. , 2018 ; Dhillon et al. , 2018 ; Ma et al. , 2018 ; Pang et al. , 2020 ; 2019 ; Samangouei et al. , 2018 ; Xiao et al. , 2020 ) , but most of them have been found to be ineffective in the presence of adaptive attacks ( Athalye et al. , 2018 ; Croce & Hein , 2020b ; Tramer et al. , 2020 ) . Ultimately , this leaves adversarial training ( Madry et al. , 2018 ) and its variants ( Alayrac et al. , 2019 ; Carmon et al. , 2019 ; Gowal et al. , 2020 ; Hendrycks et al. , 2019 ; Kumari et al. , 2019 ; Wu et al. , 2020 ; Zhang et al. , 2019a ) as the most effective and popular approach to construct robust models . Unfortunately , adversarial training yields much worse performance on the test data than vanilla training . In particular , it strongly suffers from overfitting ( Rice et al. , 2020 ) , with the model ’ s performance decaying significantly on the test set in the later phase of adversarial training . While this can be mitigated by early stopping ( Rice et al. , 2020 ) or model smoothing ( Chen et al. , 2021b ) , the reason behind the overfitting of adversarial training remains poorly understood . In this paper , we study this phenomenon from the perspective of training instances , i.e. , training input-target pairs . We introduce a quantitative metric to measure the difficulty of the instances and analyze the model ’ s behavior , such as its loss and intermediate activations , on training instances of different difficulty levels . This lets us discover that the model ’ s generalization performance decays significantly when it fits the hard adversarial instances in the later training phase . To more rigorously study this phenomenon , we then perform theoretical analyses on both linear and nonlinear models . For linear models , we study logistic regression on a Gaussian mixture model , in which we can calculate the analytical expression of the model parameters upon convergence and thus the robust test accuracy . Our theorem demonstrates that adversarial training on harder instances leads to larger generalization gaps . We further prove that the gap in robust test accuracy between models trained by hard instances and ones trained by easy instances increases with the size of the adversarial budget . In the case of nonlinear models , we derive the lower bound of the model ’ s Lipschitz constant when the model is well fit to the adversarial training examples . This bound increases with the difficulty level of the training instances and the size of the adversarial budget . Since a larger Lipschitz constant indicates a higher adversarial vulnerability ( Ruan et al. , 2018 ; Weng et al. , 2018a ; b ) , our theoretical analysis confirms our empirical observations . Our findings can be broadly applied to obtain robust models . To evidence this , in addition to the standard adversarial training settings of ( Madry et al. , 2018 ) , we study the following two scenarios : fast adversarial training and adversarial fine-tuning with additional training data . Our proposed method assigns adaptive targets or adaptive weights on training instances to avoid fitting hard inputtarget pairs . We show it can mitigate adversarial overfitting and improve models ’ performance . For fast adversarial training , our results are better than other accelerated adversarial training methods available on RobustBench ( Croce et al. , 2020 ) . For adversarial fine-tuning with additional training data , we show improved performance over the methods in ( Alayrac et al. , 2019 ; Carmon et al. , 2019 ) . Contributions . In summary , our contributions are as follows : 1 ) Based on a quantitative metric of instance difficulty , we show that fitting hard adversarial instances leads to degraded generalization performance in adversarial training . 2 ) We conduct a rigorous theoretical analysis on both linear and nonlinear models . For linear models , we show analytically that models trained on harder instances have larger robust test error than the ones trained on easy instances ; the gap increases with the size of the adversarial budget . For nonlinear models , we derive a lower bound of the model ’ s Lipschitz constant . The lower bound increases with the difficulty of the training instances and the size of the adversarial budget , indicating both factors make adversarial overfitting more severe . 3 ) We show that the adaptive use of the easy and hard training instances can improve the performance in fast adversarial training and adversarial fine-tuning with additional training data . Notation and terminology . In this paper , x and x′ are the clean input and its adversarial counterpart . We use fw to represent a model parameterized by w and omit the subscript w unless ambiguous . o = fw ( x ) and o′ = fw ( x′ ) are the model ’ s output of the clean input and the adversarial input . Lw ( x , y ) and Lw ( x′ , y ) represent the loss of the clean and adversarial instances , receptively , in which we sometimes omit w and y for notation simplicity . We use ‖w‖ and ‖X‖ to represent the l2 norm of the vector w and the largest singular value of the matrix X , respectively . sign is an elementwise function which returns +1 for positive elements , −1 for negative elements and 0 for 0 . 1y is the one-hot vector with only the y-th dimension being 1 . The term adversarial budget refers to the allowable perturbations applied to the input instance . It is characterized by lp norm and the size as a set S ( p ) ( ) = { ∆|‖∆‖p ≤ } , with defining the budget size . Therefore , given the training set D , the robust learning problem can be formulated as the min-max optimization minw Ex∼Dmax∆∈S ( p ) ( ) Lw ( x+ ∆ ) . A notation table is provided in Appendix A . In this paper , vanilla training refers to training on the clean inputs , and vanilla adversarial training to the adversarial training method in ( Madry et al. , 2018 ) . RN18 and WRN34 are the 18-layer ResNet ( He et al. , 2016 ) and the 34-layer WideResNet ( Zagoruyko & Komodakis , 2016 ) used in ( Madry et al. , 2018 ) and ( Wong et al. , 2020 ) , respectively . To avoid confusion with the general term overfitting , which denotes the gap between the training and test accuracy , we employ the term adversarial overfitting to indicate the phenomenon that robust accuracy on the test set decreases significantly in the later phase of vanilla adversarial training . This phenomenon was pointed out in ( Rice et al. , 2020 ) and does not occur in vanilla training . Our code is submitted on GoogleDrive anonymously1 . 2 RELATED WORK . We concentrate on white-box attacks , where the attacker has access to the model parameters . Such attacks are usually based on first-order information and stronger than black-box attacks ( Andriushchenko et al. , 2020 ; Dong et al. , 2018 ) . For example , the fast gradient sign method ( FGSM ) ( Goodfellow et al. , 2014 ) perturbs the input based on its gradient ’ s sign , i.e. , ∆ = sign ( OxLw ( x ) ) . The iterative fast gradient sign method ( IFGSM ) ( Kurakin et al. , 2016 ) iteratively runs FGSM using a smaller step size and projects the perturbation back to the adversarial budget after each iteration . On top of IFGSM , projected gradient descent ( PGD ) ( Madry et al. , 2018 ) use random initial perturbations and restarts to boost the strength of the attack . Many methods have been proposed to defend a model against adversarial attacks ( Buckman et al. , 2018 ; Dhillon et al. , 2018 ; Ma et al. , 2018 ; Pang et al. , 2020 ; 2019 ; Samangouei et al. , 2018 ; Xiao 1https : //drive.google.com/file/d/1vb6ehNMkBeNIM3dLr_igKMUcCgBd9ZmK/view ? usp=sharing et al. , 2020 ) . However , most of them were shown to utilize obfuscated gradients ( Athalye et al. , 2018 ; Croce & Hein , 2020b ; Tramer et al. , 2020 ) , that is , training the model to tackle some specific types of attacks instead of achieving true robustness . This makes these falsely robust models vulnerable to stronger adaptive attacks . By contrast , several works have designed training algorithms to obtain provably robust models ( Cohen et al. , 2019 ; Gowal et al. , 2019 ; Raghunathan et al. , 2018 ; Salman et al. , 2019 ; Wong & Kolter , 2018 ) . Unfortunately , these methods either do not generalize to modern network architectures or have a prohibitively large computational complexity . As a consequence , adversarial training ( Madry et al. , 2018 ) and its variants ( Alayrac et al. , 2019 ; Carmon et al. , 2019 ; Hendrycks et al. , 2019 ; Kumari et al. , 2019 ; Wu et al. , 2020 ; Zhang et al. , 2019a ) have become the de facto approach to obtain robust models in practice . In essence , these methods generate adversarial examples , usually using PGD , and use them to optimize the model parameters . While effective , adversarial training is more challenging than vanilla training . It was shown to require larger models ( Xie & Yuille , 2020 ) and to exhibit a poor convergence behavior ( Liu et al. , 2020 ) . Furthermore , as observed in ( Rice et al. , 2020 ) , it suffers from adversarial overfitting : the robust accuracy on the test set significantly decreases in the late adversarial training phase . ( Rice et al. , 2020 ) thus proposed to perform early stopping based on a separate validation set to improve the generalization performance in adversarial training . Furthermore , ( Chen et al. , 2021b ) introduced logit smoothing and weight smoothing strategies to reduce adversarial overfitting . In parallel to this , several techniques to improve the model ’ s robust test accuracy were proposed ( Wang et al. , 2020 ; Wu et al. , 2020 ; Zhang et al. , 2021 ) , but without solving the adversarial overfitting issue . By contrast , other works ( Balaji et al. , 2019 ; Huang et al. , 2020 ) were empirically shown to mitigate adversarial overfitting but without providing any explanations as to how this phenomenon was addressed . In this paper , we study the causes of adversarial overfitting from both an empirical and a theoretical point of view . We also identify the reasons why prior attempts ( Balaji et al. , 2019 ; Chen et al. , 2021a ; Huang et al. , 2020 ) successfully mitigate it . 3 A METRIC FOR INSTANCE DIFFICULTY . Parametric models are trained to minimize a loss objective based on several input-target pairs called training set , and are then evaluated on a held-out set called test set . By comparing the loss value of each instance , we can understand which ones , in either the training or the test set , are more difficult for the model to fit . In this section , we introduce a metric for instance difficulty , which mainly depends on the data and on the perturbations applied to the instances . Let L ( x ) denote the average loss of x ’ s corresponding perturbed input across all training epochs . We define the difficulty of an instance x within a finite set D as d ( x ) =P ( L ( x ) < L ( x̃ ) |x̃ ∼ U ( D ) ) + 1 2 P ( L ( x ) = L ( x̃ ) |x̃ ∼ U ( D ) ) , ( 1 ) where x̃ ∼ U ( D ) indicates x̃ is uniformly sampled from the finite set D. d ( x ) is a bounded function , close to 0 for the hardest instances , and 1 for the easiest ones . We discuss the motivation and properties of d ( x ) in Appendix D.1 , and show that it mainly depends on the data and the perturbation applied , the model architecture or the training duration can hardly affect the difficulty function . That is , d ( x ) can represent the difficulty of x within a set under a specific type of perturbation .

Summary: This paper proposes a metric to measure the difficulty of training examples and find that hard training examples influence the generalization of adversarial training and cause overfitting in adversarial training. The authors also provide a theoretical analysis. To mitigate the issue caused by hard training examples, the authors propose to assign different weights for training examples based on loss values. The evaluation results show that the proposed training methods can improve robustness in fast adversarial training and adversarial fine-tuning with additional data.