Improving Adversarial Robustness Requires Revisiting Misclassified Examples

1 INTRODUCTION . Despite their great success in applications such as computer vision ( He et al. , 2016 ) , speech recognition ( Wang et al. , 2017 ) and natural language processing ( Devlin et al. , 2018 ; Zeng et al. , 2019 ) , deep neural networks ( DNNs ) are extremely vulnerable to adversarial examples crafted by adding small adversarial perturbations to natural examples ( Szegedy et al. , 2013 ; Goodfellow et al. , 2015 ; Wu et al. , 2020 ) . Given a DNN classifier hθ with parameter θ and a correctly classified natural example x with class label y ( hθ ( x ) = y ) , an adversarial example x′ can be generated by perturbing x such that hθ ( x′ ) 6= y , i.e. , the natural example is correctly classified before perturbation but misclassified after perturbation . The perturbation required for misclassification is often small and bounded by an Lp-norm ‖x′ − x‖p ≤ , which keeps x′ within the -ball centered at x , so that it is visually the “ same ” for human observers . This vulnerability of DNNs raises serious security concerns about their practicability in security critical applications ( Chen et al. , 2015 ; Kurakin et al. , 2016 ; Jiang et al. , 2019 ; Finlayson et al. , 2019 ; Ma et al. , 2019 ) . Compared with pre/post-processing methods such as feature squeezing ( Xu et al. , 2017 ) , input denoising ( Guo et al. , 2018 ; Liao et al. , 2018 ; Samangouei et al. , 2018 ; Bai et al. , 2019 ) and adversarial detection ( Feinman et al. , 2017 ; Ma et al. , 2018 ; Lee et al. , 2018 ) , several defense techniques have been proposed to train DNNs that are inherently robust to adversarial examples including defensive distillation ( Papernot et al. , 2016 ) , gradient regularization ( Gu & Rigazio , 2014 ; Papernot et al. , 2017 ; Ross & Doshi-Velez , 2018 ; Tramèr et al. , 2018 ) , model compression ( Das et al. , 2018 ; Liu et al. , 2018 ) and activation pruning ( Dhillon et al. , 2018 ; Rakin et al. , 2018 ) , among which ∗Equal contribution . †Corresponding author . adversarial training has been demonstrated to be the most effective ( Athalye et al. , 2018 ) . Adversarial training can be regarded as a data augmentation technique that trains DNNs on adversarial examples , and can be viewed as solving the following min-max optimization problem ( Madry et al. , 2018 ) : min θ 1 n n∑ i=1 max ‖x′i−xi‖p≤ ` ( hθ ( x ′ i ) , yi ) , ( 1 ) where n is the number of training examples and ` ( · ) is the classification loss , such as the commonly used cross-entropy ( CE ) loss . The inner maximization generates adversarial examples that can be used by the outer minimization to train robust DNNs . Recently , adversarial training with adversarial examples generated by Projected Gradient Descent ( PGD ) ( Madry et al. , 2018 ) has been demonstrated to be the only method that can train moderately robust DNNs without being fully attacked ( Athalye et al. , 2018 ) . However , there is still a significant gap between adversarial robustness ( test accuracy on adversarial examples ) and natural accuracy ( test accuracy on natural examples ) , even for simple image datasets like CIFAR-10 ( Krizhevsky & Hinton , 2009 ) . Compared with natural training ( on natural examples ) , training adversarially robust DNNs is particularly difficult ( Madry et al. , 2018 ) . Nakkiran ( 2019 ) showed that a model requires more capacity to be robust ( i.e. , simple models can have high natural accuracy but are less likely to be robust ) . In addition , the sample complexity of adversarial training can be significantly higher than that of natural training , that is , training robust DNNs tends to require more data either labeled ( Schmidt et al. , 2018 ) or unlabeled ones ( Uesato et al. , 2019 ; Carmon et al. , 2019 ; Najafi et al. , 2019 ; Zhai et al. , 2019 ) . Moreover , Tsipras et al . ( 2019 ) ; Zhang et al . ( 2019 ) demonstrated that adversarial robustness may be inherently at odds with natural accuracy . Parallel to these studies , in this paper , we provide some new insights on the adversarial examples used for adversarial training . Recall that the formal definition of an adversarial example is conditioned on it being correctly classified∗ ( Carlini et al. , 2019 ) . From this perspective , adversarial examples generated from misclassified examples are “ undefined ” . Most adversarial training variants neglect this distinction , where all training examples are treated equally in both the maximization and the minimization processes , regardless of whether or not they are correctly classified . The only exception we are aware of is Ding et al . ( 2018 ) , which proposes to use maximal margin optimization for correctly classified examples . Yet they did not pay sufficient attention to misclassified examples . A deeper understanding about the influence of misclassified and correctly classified examples on the robustness is still missing in the literature . Therefore , we raise the following questions : Are the adversarial examples generated from i ) misclassified and ii ) correctly classified examples , equally important for adversarial robustness ? If not , how can one make better use of the difference to improve robustness ? In this paper , we investigate this intriguing , yet thus far overlooked aspect of adversarial training , and find that misclassified and correctly classified examples exhibit a distinctive influence on the final robustness . To illustrate this phenomenon , we conduct a proof-of-concept experiment on CIFAR10 in a white-box setting with L∞ maximum perturbation = 8/255 . We first train an 8-layer ∗In this paper , both “ correctly classified ” and “ misclassified ” refer to predictions on natural training examples . Convolutional Neural Network ( CNN ) using standard adversarial training with 10-step PGD ( PGD10 ) and step size /4 , then use this network ( 87 % training accuracy ) to select two subsets of natural training examples to investigate : 1 ) a subset of misclassified examples S− ( 13 % of training data ) , and 2 ) a subset of correctly classified examples S+ ( also 13 % of training data , |S+| = |S−| ) . Using these two subsets , we explore different ways to re-train the same network , and evaluate its robustness against white-box PGD20 ( step size /10 ) attacks on the test dataset . In Figure 1 ( a ) , we find that misclassified examples have a significant impact on the final robustness . Compared with standard adversarial training ( dashed blue line ) , the final robustness drops drastically , if examples in subset S− are not perturbed ( solid green line ) during adversarial training ( other examples are still perturbed by PGD10 ) . In contrast , the same operation on subset S+ only slightly affects the final robustness ( solid orange line ) . Previous work has found that removing a small proportion of training examples does not reduce the robustness ( Ding et al. , 2019 ) , which seems to be true for correctly classified examples , but is apparently not true for misclassified examples . To further understand the distinctive influence of misclassified and correctly classified examples , we test different techniques on them within either the maximization or the minimization process of adversarial training . Firstly , we apply different maximization techniques while keeping the minimization loss CE unchanged . As shown in Figure 1 ( b ) , the final robustness is barely affected when we use a weak attack ( e.g. , Fast Gradient Sign Method ( FGSM ) ( Goodfellow et al. , 2015 ) ) to perturb misclassified examples S− ( all other training examples are still perturbed by PGD10 ) . This suggests that different maximization techniques on misclassified examples S− may have a negligible influence on the final robustness , provided that the inner maximization problem is solved to a moderate precision ( Wang et al. , 2019 ) . However , for subset S+ , a weak attack for the maximization tends to degenerate the robustness . Secondly , we test different minimization techniques with the inner maximization still solved by PGD10 . Interestingly , we find that different minimization techniques on misclassified examples make a significant difference to the final robustness . As shown in Figure 1 ( c ) , compared with standard adversarial training ( dashed blue line ) with the CE loss , the final robustness is significantly improved when the outer minimization on misclassified examples is “ regularized ” ( solid green line ) by an additional term ( a KL-divergence term that was used previously in Zheng et al . ( 2016 ) ; Zhang et al . ( 2019 ) ) . The same regularization applied to correctly classified examples also helps the final robustness ( solid orange line ) , though not as significantly as for misclassified examples . Motivated by the above observations , we reformulate the adversarial risk to incorporate an explicit differentiation of misclassified examples in a form of regularization . We then propose a new defense algorithm to achieve this in a dynamic way during adversarial training . Our main contributions are : • We investigate the distinctive influence of misclassified and correctly classified examples on the final robustness of adversarial training . We find that the manipulation on misclassified examples has more impact on the final robustness , and the minimization techniques are more crucial than maximization ones under the min-max optimization framework . • We propose a regularized adversarial risk which incorporates an explicit differentiation of misclassified examples as a regularizer . Based on that , we further propose a new defense algorithm , called Misclassification Aware adveRsarial Training ( MART ) . • Experimentally , we show that adversarial robustness can be significantly improved over the stateof-the-art , by a specific focus on misclassified examples . It also helps improve recently proposed adversarial training with unlabeled data . 2 MISCLASSIFICATION AWARE ADVERSARIAL RISK . In this section , we propose a regularized adversarial risk that incorporates an explicit differentiation of misclassified examples . 2.1 PRELIMINARIES . We first define some notations . We use lower case and lower case bold face to denote scalars and vectors , respectively . We use upper case calligraphic symbols to denote sets . For a K-class ( K ≥ 2 ) classification problem , given a dataset { ( xi , yi ) } i=1 , ... , n with xi ∈ Rd as a natural example and yi ∈ { 1 , . . . , K } as its associated label , a DNN classifier hθ with model parameter θ predicts the class of an input example xi : hθ ( xi ) = arg max k=1 , ... , K pk ( xi , θ ) , pk ( xi , θ ) = exp ( zk ( xi , θ ) ) / K∑ k′=1 exp ( zk′ ( xi , θ ) ) , ( 2 ) where zk ( xi , θ ) is the logits output of the network with respect to class k , and pk ( xi , θ ) is the probability ( softmax on logits ) of xi belonging to class k. The adversarial risk ( Madry et al. , 2018 ) on dataset { ( xi , yi ) } i=1 , ... , n and classifier hθ can be defined with respect to the 0-1 loss ( Zhang et al. , 2019 ) as : R ( hθ ) = 1 n n∑ i=1 max x′i∈B ( xi ) 1 ( hθ ( x ′ i ) 6= yi ) , ( 3 ) where 1 ( · ) is the indicator function and B ( xi ) = { x : ‖x − xi‖p ≤ } denotes the Lp-norm ball centered at xi with radius . We will focus on the L∞-ball in this paper .

Summary: The paper improves adversarial training by introducing two modifications to the loss function: (i) a "boosted" version of the cross-entropy loss that involves a term similar to a large-margin loss, and (ii) weighting the adversarial loss differently depending on how correctly classified an example is. When put together, these modifications achieve state-of-the-art robustness on CIFAR-10, improving over the previously best robust accuracy by about 3.5%. The authors perform multiple ablation studies and demonstrate that their modified loss function also improves when additional unlabeled data is added (again achieving state-of-the-art robustness).