Exploring Memorization in Adversarial Training

1 INTRODUCTION . Deep neural networks ( DNNs ) usually exhibit excellent generalization ability in pattern recognition tasks , despite their sufficient capacity to overfit or memorize the entire training set with completely random labels ( Zhang et al. , 2017 ) . The memorization behavior in deep learning has aroused tremendous attention to identifying the differences between learning on true and random labels ( Arpit et al. , 2017 ; Neyshabur et al. , 2017 ) , and examining what and why DNNs memorize ( Feldman , 2020 ; Feldman & Zhang , 2020 ; Maennel et al. , 2020 ) . This phenomenon has also motivated a growing body of works on model capacity ( Arpit et al. , 2017 ; Belkin et al. , 2019 ) , convergence ( Allen-Zhu et al. , 2019 ; Du et al. , 2019 ; Zou et al. , 2020 ) , and generalization ( Neyshabur et al. , 2017 ; Bartlett et al. , 2017 ) , which consequently provide a better understanding of the DNN working mechanism . In this paper , we explore the memorization behavior for a different learning algorithm—adversarial training ( AT ) . Owing to the security threat of adversarial examples , i.e. , maliciously generated inputs by adding imperceptible perturbations to cause misclassification ( Szegedy et al. , 2014 ; Goodfellow et al. , 2015 ) , various defense methods have been proposed to improve the adversarial robustness of DNNs ( Kurakin et al. , 2017 ; Madry et al. , 2018 ; Liao et al. , 2018 ; Wong & Kolter , 2018 ; Cohen et al. , 2019 ; Zhang et al. , 2019b ; Pang et al. , 2019 ; 2020 ; Dong et al. , 2020a ) . AT is arguably the most effective defense technique ( Athalye et al. , 2018 ; Dong et al. , 2020b ) , in which the network is trained on the adversarially augmented samples instead of the natural ones ( Madry et al. , 2018 ) . Despite the popularity , the memorization behavior in AT is less explored . Schmidt et al . ( 2018 ) show that a model is able to fully ( over ) fit the training set against an adversary , i.e. , reaching almost 100 % robust training accuracy , while the performance on test data is much inferior , witnessing a significant generalization gap . The overfitting phenomenon in AT is further investigated in Rice et al . ( 2020 ) . However , it is not clear whether DNNs could memorize adversarial examples of training data with completely random labels . Answering this question could help to examine the effects of memorization in AT under the “ extreme ” circumstance and facilitate a deeper understanding of capacity , convergence , generalization , and robust overfitting of the adversarially trained models . In general , it is difficult for a classifier to memorize adversarial examples with random labels since the model entails a much more complicated decision boundary , as illustrated in Fig . 1 . Even though the networks have sufficient capacity , AT may not necessarily converge . Therefore , we aim to comprehensively study this problem and explore how the analysis can motivate better algorithms . Our contributions . We first empirically investigate the memorization behavior in AT by performing PGD-AT ( Madry et al. , 2018 ) and TRADES ( Zhang et al. , 2019b ) with random labels sampled uniformly over all classes . Different from standard training ( ST ) that can easily memorize random labels ( Zhang et al. , 2017 ) , AT may fail to converge , with PGD-AT being a typical example . Nevertheless , TRADES can converge under this circumstance . It demonstrates that DNNs have sufficient capacity to memorize adversarial examples of training data with completely random labels . This phenomenon is commonly observed on multiple datasets , network architectures , and threat models . The memorization analysis has further implications for understanding the convergence and generalization of AT . We conduct a convergence analysis on gradient magnitude and stability to explain the counter-intuitive different convergence properties of PGD-AT and TRADES with random labels since they behave similarly when trained on true labels ( Rice et al. , 2020 ) . We corroborate that PGD-AT suffers from a gradient instability issue while the gradients of TRADES are relatively stable thanks to its adversarial loss . Moreover , by considering models trained on random labels , our generalization analysis indicates that several recently suggested complexity measures are inadequate to explain robust generalization , which is complementary to the findings in ST ( Neyshabur et al. , 2017 ) . Accordingly , an appropriate explanation of robust generalization remains largely under-addressed . Lastly , but most importantly , we identify a significant drawback of memorization in AT that it could result in robust overfitting ( Rice et al. , 2020 ) . We argue that the cause of robust overfitting lies in the memorization of one-hot labels in the typical AT methods . The one-hot labels can be inappropriate or even noisy for some adversarial examples because some data may naturally lie close to the decision boundary , and the corresponding adversarial examples should be assigned low predictive confidence ( Stutz et al. , 2020 ; Cheng et al. , 2020 ) . When the network starts to memorize these noisy examples , robust overfitting occurs , as revealed by the learning curves . To address this problem , we propose a new mitigation algorithm that impedes over-confident predictions by a regularization term for avoiding the excessive memorization of adversarial examples with possibly noisy labels . Experiments validate that our approach can eliminate robust overfitting to a large extent across multiple datasets , network architectures , threat models , and AT methods , achieving better robustness under a variety of adversarial attacks than the baselines . 2 BACKGROUND . 2.1 ADVERSARIAL TRAINING . Adversarial training is a commonly used technique to learn robust DNN classifiers against adversarial examples . We let D = { ( xi , yi ) } ni=1 denote the training dataset with n samples , where xi ∈ Rd is a natural example and yi ∈ { 1 , ... , C } is its true label often encoded as an one-hot vector 1yi with totally C classes . AT can be formulated as a robust optimization problem ( Madry et al. , 2018 ) : min θ n∑ i=1 max x′i∈S ( xi ) L ( fθ ( x′i ) , yi ) , ( 1 ) where fθ is a DNN classifier with parameters θ that predicts probabilities over all classes , L is the classification loss ( i.e. , the cross-entropy loss as L ( fθ ( x ) , y ) = −1 > y log fθ ( x ) ) , and S ( x ) = { x′ : ‖x′ − x‖p ≤ } is an adversarial region centered at x with radius > 0 under the ` p-norm threat models ( e.g. , ` 2 and ` ∞ norms that we consider ) . The robust optimization problem ( 1 ) is solved by using adversarial attacks to approximate the inner maximization and updating the model parameters θ via gradient descent . A typical method uses projected gradient descent ( PGD ) ( Madry et al. , 2018 ) for the inner problem , which starts at a randomly initialized point in S ( xi ) and iteratively updates the adversarial example under the ` ∞-norm threat model by x′i = ΠS ( xi ) ( x′i + α · sign ( ∇xL ( fθ ( x′i ) , yi ) ) ) , ( 2 ) where Π ( · ) is the projection operator and α is the step size . Besides PGD-AT , another typical AT method is TRADES ( Zhang et al. , 2019b ) , which balances the trade-off between robustness and natural accuracy by minimizing a different adversarial loss min θ n∑ i=1 { L ( fθ ( xi ) , yi ) + β · max x′i∈S ( xi ) D ( fθ ( xi ) ‖fθ ( x′i ) ) } , ( 3 ) where L is the clean cross-entropy loss on the natural example , D is the Kullback–Leibler divergence , and β is a balancing parameter . The inner maximization of TRADES is also solved by PGD . Recent progress of AT includes designing new adversarial losses ( Mao et al. , 2019 ; Qin et al. , 2019 ; Pang et al. , 2020 ; Wang et al. , 2020 ; Dong et al. , 2020a ) and network architecture ( Xie et al. , 2019 ) , training acceleration ( Shafahi et al. , 2019 ; Zhang et al. , 2019a ; Wong et al. , 2020 ) , and exploiting more training data ( Hendrycks et al. , 2019 ; Alayrac et al. , 2019 ; Carmon et al. , 2019 ; Zhai et al. , 2019 ) . Recent works highlight the training tricks in AT ( Gowal et al. , 2020 ; Pang et al. , 2021 ) . 2.2 RELATED WORK ON DNN MEMORIZATION . It has been observed that DNNs can easily memorize training data with random labels ( Zhang et al. , 2017 ) , which requires “ rethinking ” of conventional techniques ( e.g. , VC dimension ) to explain generalization . Arpit et al . ( 2017 ) identify qualitative differences between learning on true and random labels . Further works attempt to examine what and why DNNs memorize ( Feldman , 2020 ; Feldman & Zhang , 2020 ; Maennel et al. , 2020 ) . Motivated by the memorization phenomenon in deep learning , convergence of training has been analyzed in the over-parameterized setting ( Allen-Zhu et al. , 2019 ; Du et al. , 2019 ; Zou et al. , 2020 ) , while generalization has been studied with numerous theoretical and empirical complexity measures ( Neyshabur et al. , 2015 ; 2017 ; Bartlett et al. , 2017 ; Novak et al. , 2018 ; Arora et al. , 2018 ; Cao & Gu , 2019 ; Jiang et al. , 2020 ; Chen et al. , 2020 ) . In contrast , the memorization behavior in AT has been less explored . The previous works demonstrate that DNNs can fit training data against an adversary ( Madry et al. , 2018 ; Schmidt et al. , 2018 ; Rice et al. , 2020 ) , e.g. , achieving nearly 100 % robust training accuracy against a PGD adversary , but this behavior is not explored when trained on random labels . This paper is dedicated to investigating the memorization in AT under the extreme condition with random labels , while drawing connections to capacity , convergence , generalization , and robust overfitting , with the overarching goal of better understanding the AT working mechanism . 3 MEMORIZATION IN AT AND IMPLICATIONS . In this section , we first explore the memorization behavior in AT through an empirical study . Our analysis raises new questions about the convergence and generalization of AT , many of which can not be answered by existing works . Thereafter , we provide further analytical studies on the convergence and generalization of AT by considering models trained on random labels particularly . 3.1 AT WITH RANDOM LABELS . We explore the memorization behavior of PGD-AT ( Madry et al. , 2018 ) and TRADES ( Zhang et al. , 2019b ) as two studying cases . The experiments are conducted on CIFAR-10 ( Krizhevsky & Hinton , 2009 ) with a Wide ResNet model ( Zagoruyko & Komodakis , 2016 ) of depth 28 and widen factor 10 ( WRN-28-10 ) . Similar to Zhang et al . ( 2017 ) , we train a network on the original dataset with true labels and on a copy of the dataset in which the true labels are corrupted by random ones . For training and robustness evaluation , a 10-step ` ∞ PGD adversary with = 8/255 and α = 2/255 is adopted . For TRADES , the PGD adversary maximizes the KL divergence during training , while maximizes the cross-entropy loss for robustness evaluation , as common practice ( Zhang et al. , 2019b ) . We set β = 6.0 . In the sequel , we denote accuracy of a classifier against the 10-step PGD adversary as “ robust accuracy ” , and accuracy on natural examples as “ natural accuracy ” . Fig . 2 ( a ) and Fig . 2 ( b ) show the learning curves of PGD-AT and TRADES without explicit regularizations . Both methods achieve almost 100 % natural and robust training accuracies when trained on true labels . When the labels are random , we observe the totally different behaviors between PGDAT and TRADES—PGD-AT fails to converge while TRADES still reaches nearly 100 % training accuracies . This phenomenon is somewhat striking because PGD-AT and TRADES perform similarly on true labels ( Rice et al. , 2020 ) . We find that the different memorization behaviors between PGD-AT and TRADES when trained on random labels can commonly be observed across a variety of datasets , model architectures , and threat models ( shown in Appendix A.1 , indicating that it is a general phenomenon of memorization in the two AT methods . Therefore , our finding is : DNNs have sufficient capacity to memorize adversarial examples of training data with completely random labels , but the convergence depends on the AT algorithms . Partially corrupted labels . We then inspect the behavior of AT under varying levels of label noise from 0 % ( true labels ) to 100 % ( completely random labels ) . The generalization gap ( i.e. , difference between training and test accuracies ) presented in Fig . 2 ( c ) grows steadily as we increase the noise rate before the network fails to converge . The learning curves are provided in Appendix A.1 . Explicit regularizations . We study the role of common regularizers in AT memorization , including data augmentation , weight decay , and dropout ( Srivastava et al. , 2014 ) . We train TRADES on true and random labels with several combinations of regularizers . We observe the explicit regularizers do not significantly affect the model ’ s ability to memorize adversarial examples , similar to the finding in ST ( Zhang et al. , 2017 ; Arpit et al. , 2017 ) . The detailed results are provided in Appendix A.1 .

Summary: Memorization has been investigated in deep neural network classifiers but not that much in adversarial training. The authors investigate the ability of adversarial training to memorize random datasets. They use PGD-AT and TRADES to do the adversarial training and see different convergence properties, which differs significantly from training on true labels. They also discover that robust overfitting can occur in adversarial training with memorization and use temporal ensembling to mitigate it.