Exploring Memorization in Adversarial Training

1 INTRODUCTION . Deep neural networks ( DNNs ) usually exhibit excellent generalization ability in pattern recognition tasks , despite their sufficient capacity to overfit or memorize the entire training set with completely random labels ( Zhang et al. , 2017 ) . The memorization behavior in deep learning has aroused tremendous attention to identifying the differences between learning on true and random labels ( Arpit et al. , 2017 ; Neyshabur et al. , 2017 ) , and examining what and why DNNs memorize ( Feldman , 2020 ; Feldman & Zhang , 2020 ; Maennel et al. , 2020 ) . This phenomenon has also motivated a growing body of works on model capacity ( Arpit et al. , 2017 ; Belkin et al. , 2019 ) , convergence ( Allen-Zhu et al. , 2019 ; Du et al. , 2019 ; Zou et al. , 2020 ) , and generalization ( Neyshabur et al. , 2017 ; Bartlett et al. , 2017 ) , which consequently provide a better understanding of the DNN working mechanism . In this paper , we explore the memorization behavior for a different learning algorithm—adversarial training ( AT ) . Owing to the security threat of adversarial examples , i.e. , maliciously generated inputs by adding imperceptible perturbations to cause misclassification ( Szegedy et al. , 2014 ; Goodfellow et al. , 2015 ) , various defense methods have been proposed to improve the adversarial robustness of DNNs ( Kurakin et al. , 2017 ; Madry et al. , 2018 ; Liao et al. , 2018 ; Wong & Kolter , 2018 ; Cohen et al. , 2019 ; Zhang et al. , 2019b ; Pang et al. , 2019 ; 2020 ; Dong et al. , 2020a ) . AT is arguably the most effective defense technique ( Athalye et al. , 2018 ; Dong et al. , 2020b ) , in which the network is trained on the adversarially augmented samples instead of the natural ones ( Madry et al. , 2018 ) . Despite the popularity , the memorization behavior in AT is less explored . Schmidt et al . ( 2018 ) show that a model is able to fully ( over ) fit the training set against an adversary , i.e. , reaching almost 100 % robust training accuracy , while the performance on test data is much inferior , witnessing a significant generalization gap . The overfitting phenomenon in AT is further investigated in Rice et al . ( 2020 ) . However , it is not clear whether DNNs could memorize adversarial examples of training data with completely random labels . Answering this question could help to examine the effects of memorization in AT under the “ extreme ” circumstance and facilitate a deeper understanding of capacity , convergence , generalization , and robust overfitting of the adversarially trained models . In general , it is difficult for a classifier to memorize adversarial examples with random labels since the model entails a much more complicated decision boundary , as illustrated in Fig . 1 . Even though the networks have sufficient capacity , AT may not necessarily converge . Therefore , we aim to comprehensively study this problem and explore how the analysis can motivate better algorithms . Our contributions . We first empirically investigate the memorization behavior in AT by performing PGD-AT ( Madry et al. , 2018 ) and TRADES ( Zhang et al. , 2019b ) with random labels sampled uniformly over all classes . Different from standard training ( ST ) that can easily memorize random labels ( Zhang et al. , 2017 ) , AT may fail to converge , with PGD-AT being a typical example . Nevertheless , TRADES can converge under this circumstance . It demonstrates that DNNs have sufficient capacity to memorize adversarial examples of training data with completely random labels . This phenomenon is commonly observed on multiple datasets , network architectures , and threat models . The memorization analysis has further implications for understanding the convergence and generalization of AT . We conduct a convergence analysis on gradient magnitude and stability to explain the counter-intuitive different convergence properties of PGD-AT and TRADES with random labels since they behave similarly when trained on true labels ( Rice et al. , 2020 ) . We corroborate that PGD-AT suffers from a gradient instability issue while the gradients of TRADES are relatively stable thanks to its adversarial loss . Moreover , by considering models trained on random labels , our generalization analysis indicates that several recently suggested complexity measures are inadequate to explain robust generalization , which is complementary to the findings in ST ( Neyshabur et al. , 2017 ) . Accordingly , an appropriate explanation of robust generalization remains largely under-addressed . Lastly , but most importantly , we identify a significant drawback of memorization in AT that it could result in robust overfitting ( Rice et al. , 2020 ) . We argue that the cause of robust overfitting lies in the memorization of one-hot labels in the typical AT methods . The one-hot labels can be inappropriate or even noisy for some adversarial examples because some data may naturally lie close to the decision boundary , and the corresponding adversarial examples should be assigned low predictive confidence ( Stutz et al. , 2020 ; Cheng et al. , 2020 ) . When the network starts to memorize these noisy examples , robust overfitting occurs , as revealed by the learning curves . To address this problem , we propose a new mitigation algorithm that impedes over-confident predictions by a regularization term for avoiding the excessive memorization of adversarial examples with possibly noisy labels . Experiments validate that our approach can eliminate robust overfitting to a large extent across multiple datasets , network architectures , threat models , and AT methods , achieving better robustness under a variety of adversarial attacks than the baselines . 2 BACKGROUND . 2.1 ADVERSARIAL TRAINING . Adversarial training is a commonly used technique to learn robust DNN classifiers against adversarial examples . We let D = { ( xi , yi ) } ni=1 denote the training dataset with n samples , where xi ∈ Rd is a natural example and yi ∈ { 1 , ... , C } is its true label often encoded as an one-hot vector 1yi with totally C classes . AT can be formulated as a robust optimization problem ( Madry et al. , 2018 ) : min θ n∑ i=1 max x′i∈S ( xi ) L ( fθ ( x′i ) , yi ) , ( 1 ) where fθ is a DNN classifier with parameters θ that predicts probabilities over all classes , L is the classification loss ( i.e. , the cross-entropy loss as L ( fθ ( x ) , y ) = −1 > y log fθ ( x ) ) , and S ( x ) = { x′ : ‖x′ − x‖p ≤ } is an adversarial region centered at x with radius > 0 under the ` p-norm threat models ( e.g. , ` 2 and ` ∞ norms that we consider ) . The robust optimization problem ( 1 ) is solved by using adversarial attacks to approximate the inner maximization and updating the model parameters θ via gradient descent . A typical method uses projected gradient descent ( PGD ) ( Madry et al. , 2018 ) for the inner problem , which starts at a randomly initialized point in S ( xi ) and iteratively updates the adversarial example under the ` ∞-norm threat model by x′i = ΠS ( xi ) ( x′i + α · sign ( ∇xL ( fθ ( x′i ) , yi ) ) ) , ( 2 ) where Π ( · ) is the projection operator and α is the step size . Besides PGD-AT , another typical AT method is TRADES ( Zhang et al. , 2019b ) , which balances the trade-off between robustness and natural accuracy by minimizing a different adversarial loss min θ n∑ i=1 { L ( fθ ( xi ) , yi ) + β · max x′i∈S ( xi ) D ( fθ ( xi ) ‖fθ ( x′i ) ) } , ( 3 ) where L is the clean cross-entropy loss on the natural example , D is the Kullback–Leibler divergence , and β is a balancing parameter . The inner maximization of TRADES is also solved by PGD . Recent progress of AT includes designing new adversarial losses ( Mao et al. , 2019 ; Qin et al. , 2019 ; Pang et al. , 2020 ; Wang et al. , 2020 ; Dong et al. , 2020a ) and network architecture ( Xie et al. , 2019 ) , training acceleration ( Shafahi et al. , 2019 ; Zhang et al. , 2019a ; Wong et al. , 2020 ) , and exploiting more training data ( Hendrycks et al. , 2019 ; Alayrac et al. , 2019 ; Carmon et al. , 2019 ; Zhai et al. , 2019 ) . Recent works highlight the training tricks in AT ( Gowal et al. , 2020 ; Pang et al. , 2021 ) . 2.2 RELATED WORK ON DNN MEMORIZATION . It has been observed that DNNs can easily memorize training data with random labels ( Zhang et al. , 2017 ) , which requires “ rethinking ” of conventional techniques ( e.g. , VC dimension ) to explain generalization . Arpit et al . ( 2017 ) identify qualitative differences between learning on true and random labels . Further works attempt to examine what and why DNNs memorize ( Feldman , 2020 ; Feldman & Zhang , 2020 ; Maennel et al. , 2020 ) . Motivated by the memorization phenomenon in deep learning , convergence of training has been analyzed in the over-parameterized setting ( Allen-Zhu et al. , 2019 ; Du et al. , 2019 ; Zou et al. , 2020 ) , while generalization has been studied with numerous theoretical and empirical complexity measures ( Neyshabur et al. , 2015 ; 2017 ; Bartlett et al. , 2017 ; Novak et al. , 2018 ; Arora et al. , 2018 ; Cao & Gu , 2019 ; Jiang et al. , 2020 ; Chen et al. , 2020 ) . In contrast , the memorization behavior in AT has been less explored . The previous works demonstrate that DNNs can fit training data against an adversary ( Madry et al. , 2018 ; Schmidt et al. , 2018 ; Rice et al. , 2020 ) , e.g. , achieving nearly 100 % robust training accuracy against a PGD adversary , but this behavior is not explored when trained on random labels . This paper is dedicated to investigating the memorization in AT under the extreme condition with random labels , while drawing connections to capacity , convergence , generalization , and robust overfitting , with the overarching goal of better understanding the AT working mechanism . 3 MEMORIZATION IN AT AND IMPLICATIONS . In this section , we first explore the memorization behavior in AT through an empirical study . Our analysis raises new questions about the convergence and generalization of AT , many of which can not be answered by existing works . Thereafter , we provide further analytical studies on the convergence and generalization of AT by considering models trained on random labels particularly . 3.1 AT WITH RANDOM LABELS . We explore the memorization behavior of PGD-AT ( Madry et al. , 2018 ) and TRADES ( Zhang et al. , 2019b ) as two studying cases . The experiments are conducted on CIFAR-10 ( Krizhevsky & Hinton , 2009 ) with a Wide ResNet model ( Zagoruyko & Komodakis , 2016 ) of depth 28 and widen factor 10 ( WRN-28-10 ) . Similar to Zhang et al . ( 2017 ) , we train a network on the original dataset with true labels and on a copy of the dataset in which the true labels are corrupted by random ones . For training and robustness evaluation , a 10-step ` ∞ PGD adversary with = 8/255 and α = 2/255 is adopted . For TRADES , the PGD adversary maximizes the KL divergence during training , while maximizes the cross-entropy loss for robustness evaluation , as common practice ( Zhang et al. , 2019b ) . We set β = 6.0 . In the sequel , we denote accuracy of a classifier against the 10-step PGD adversary as “ robust accuracy ” , and accuracy on natural examples as “ natural accuracy ” . Fig . 2 ( a ) and Fig . 2 ( b ) show the learning curves of PGD-AT and TRADES without explicit regularizations . Both methods achieve almost 100 % natural and robust training accuracies when trained on true labels . When the labels are random , we observe the totally different behaviors between PGDAT and TRADES—PGD-AT fails to converge while TRADES still reaches nearly 100 % training accuracies . This phenomenon is somewhat striking because PGD-AT and TRADES perform similarly on true labels ( Rice et al. , 2020 ) . We find that the different memorization behaviors between PGD-AT and TRADES when trained on random labels can commonly be observed across a variety of datasets , model architectures , and threat models ( shown in Appendix A.1 , indicating that it is a general phenomenon of memorization in the two AT methods . Therefore , our finding is : DNNs have sufficient capacity to memorize adversarial examples of training data with completely random labels , but the convergence depends on the AT algorithms . Partially corrupted labels . We then inspect the behavior of AT under varying levels of label noise from 0 % ( true labels ) to 100 % ( completely random labels ) . The generalization gap ( i.e. , difference between training and test accuracies ) presented in Fig . 2 ( c ) grows steadily as we increase the noise rate before the network fails to converge . The learning curves are provided in Appendix A.1 . Explicit regularizations . We study the role of common regularizers in AT memorization , including data augmentation , weight decay , and dropout ( Srivastava et al. , 2014 ) . We train TRADES on true and random labels with several combinations of regularizers . We observe the explicit regularizers do not significantly affect the model ’ s ability to memorize adversarial examples , similar to the finding in ST ( Zhang et al. , 2017 ; Arpit et al. , 2017 ) . The detailed results are provided in Appendix A.1 .

Summary: This paper presents the memorization effect/behavior in adversarial training in perspective of model capacity, convergence and generalization. They first investigate the difference of memorization behavior between two adversarial training methods, PGD-AT and TRADES, with random labels and provide the proof of gradient instability for PGD-AT in terms of Lipschitz constant. Also they suggest that previous complexity measures are inadequate to explain robust generalization. Lastly, they investigate the cause of robust overfitting in AT and propose a mitigation algorithm by a regularization term for avoiding the excessive memorization of adversarial examples.