Improving Adversarial Robustness Requires Revisiting Misclassified Examples

1 INTRODUCTION . Despite their great success in applications such as computer vision ( He et al. , 2016 ) , speech recognition ( Wang et al. , 2017 ) and natural language processing ( Devlin et al. , 2018 ; Zeng et al. , 2019 ) , deep neural networks ( DNNs ) are extremely vulnerable to adversarial examples crafted by adding small adversarial perturbations to natural examples ( Szegedy et al. , 2013 ; Goodfellow et al. , 2015 ; Wu et al. , 2020 ) . Given a DNN classifier hθ with parameter θ and a correctly classified natural example x with class label y ( hθ ( x ) = y ) , an adversarial example x′ can be generated by perturbing x such that hθ ( x′ ) 6= y , i.e. , the natural example is correctly classified before perturbation but misclassified after perturbation . The perturbation required for misclassification is often small and bounded by an Lp-norm ‖x′ − x‖p ≤ , which keeps x′ within the -ball centered at x , so that it is visually the “ same ” for human observers . This vulnerability of DNNs raises serious security concerns about their practicability in security critical applications ( Chen et al. , 2015 ; Kurakin et al. , 2016 ; Jiang et al. , 2019 ; Finlayson et al. , 2019 ; Ma et al. , 2019 ) . Compared with pre/post-processing methods such as feature squeezing ( Xu et al. , 2017 ) , input denoising ( Guo et al. , 2018 ; Liao et al. , 2018 ; Samangouei et al. , 2018 ; Bai et al. , 2019 ) and adversarial detection ( Feinman et al. , 2017 ; Ma et al. , 2018 ; Lee et al. , 2018 ) , several defense techniques have been proposed to train DNNs that are inherently robust to adversarial examples including defensive distillation ( Papernot et al. , 2016 ) , gradient regularization ( Gu & Rigazio , 2014 ; Papernot et al. , 2017 ; Ross & Doshi-Velez , 2018 ; Tramèr et al. , 2018 ) , model compression ( Das et al. , 2018 ; Liu et al. , 2018 ) and activation pruning ( Dhillon et al. , 2018 ; Rakin et al. , 2018 ) , among which ∗Equal contribution . †Corresponding author . adversarial training has been demonstrated to be the most effective ( Athalye et al. , 2018 ) . Adversarial training can be regarded as a data augmentation technique that trains DNNs on adversarial examples , and can be viewed as solving the following min-max optimization problem ( Madry et al. , 2018 ) : min θ 1 n n∑ i=1 max ‖x′i−xi‖p≤ ` ( hθ ( x ′ i ) , yi ) , ( 1 ) where n is the number of training examples and ` ( · ) is the classification loss , such as the commonly used cross-entropy ( CE ) loss . The inner maximization generates adversarial examples that can be used by the outer minimization to train robust DNNs . Recently , adversarial training with adversarial examples generated by Projected Gradient Descent ( PGD ) ( Madry et al. , 2018 ) has been demonstrated to be the only method that can train moderately robust DNNs without being fully attacked ( Athalye et al. , 2018 ) . However , there is still a significant gap between adversarial robustness ( test accuracy on adversarial examples ) and natural accuracy ( test accuracy on natural examples ) , even for simple image datasets like CIFAR-10 ( Krizhevsky & Hinton , 2009 ) . Compared with natural training ( on natural examples ) , training adversarially robust DNNs is particularly difficult ( Madry et al. , 2018 ) . Nakkiran ( 2019 ) showed that a model requires more capacity to be robust ( i.e. , simple models can have high natural accuracy but are less likely to be robust ) . In addition , the sample complexity of adversarial training can be significantly higher than that of natural training , that is , training robust DNNs tends to require more data either labeled ( Schmidt et al. , 2018 ) or unlabeled ones ( Uesato et al. , 2019 ; Carmon et al. , 2019 ; Najafi et al. , 2019 ; Zhai et al. , 2019 ) . Moreover , Tsipras et al . ( 2019 ) ; Zhang et al . ( 2019 ) demonstrated that adversarial robustness may be inherently at odds with natural accuracy . Parallel to these studies , in this paper , we provide some new insights on the adversarial examples used for adversarial training . Recall that the formal definition of an adversarial example is conditioned on it being correctly classified∗ ( Carlini et al. , 2019 ) . From this perspective , adversarial examples generated from misclassified examples are “ undefined ” . Most adversarial training variants neglect this distinction , where all training examples are treated equally in both the maximization and the minimization processes , regardless of whether or not they are correctly classified . The only exception we are aware of is Ding et al . ( 2018 ) , which proposes to use maximal margin optimization for correctly classified examples . Yet they did not pay sufficient attention to misclassified examples . A deeper understanding about the influence of misclassified and correctly classified examples on the robustness is still missing in the literature . Therefore , we raise the following questions : Are the adversarial examples generated from i ) misclassified and ii ) correctly classified examples , equally important for adversarial robustness ? If not , how can one make better use of the difference to improve robustness ? In this paper , we investigate this intriguing , yet thus far overlooked aspect of adversarial training , and find that misclassified and correctly classified examples exhibit a distinctive influence on the final robustness . To illustrate this phenomenon , we conduct a proof-of-concept experiment on CIFAR10 in a white-box setting with L∞ maximum perturbation = 8/255 . We first train an 8-layer ∗In this paper , both “ correctly classified ” and “ misclassified ” refer to predictions on natural training examples . Convolutional Neural Network ( CNN ) using standard adversarial training with 10-step PGD ( PGD10 ) and step size /4 , then use this network ( 87 % training accuracy ) to select two subsets of natural training examples to investigate : 1 ) a subset of misclassified examples S− ( 13 % of training data ) , and 2 ) a subset of correctly classified examples S+ ( also 13 % of training data , |S+| = |S−| ) . Using these two subsets , we explore different ways to re-train the same network , and evaluate its robustness against white-box PGD20 ( step size /10 ) attacks on the test dataset . In Figure 1 ( a ) , we find that misclassified examples have a significant impact on the final robustness . Compared with standard adversarial training ( dashed blue line ) , the final robustness drops drastically , if examples in subset S− are not perturbed ( solid green line ) during adversarial training ( other examples are still perturbed by PGD10 ) . In contrast , the same operation on subset S+ only slightly affects the final robustness ( solid orange line ) . Previous work has found that removing a small proportion of training examples does not reduce the robustness ( Ding et al. , 2019 ) , which seems to be true for correctly classified examples , but is apparently not true for misclassified examples . To further understand the distinctive influence of misclassified and correctly classified examples , we test different techniques on them within either the maximization or the minimization process of adversarial training . Firstly , we apply different maximization techniques while keeping the minimization loss CE unchanged . As shown in Figure 1 ( b ) , the final robustness is barely affected when we use a weak attack ( e.g. , Fast Gradient Sign Method ( FGSM ) ( Goodfellow et al. , 2015 ) ) to perturb misclassified examples S− ( all other training examples are still perturbed by PGD10 ) . This suggests that different maximization techniques on misclassified examples S− may have a negligible influence on the final robustness , provided that the inner maximization problem is solved to a moderate precision ( Wang et al. , 2019 ) . However , for subset S+ , a weak attack for the maximization tends to degenerate the robustness . Secondly , we test different minimization techniques with the inner maximization still solved by PGD10 . Interestingly , we find that different minimization techniques on misclassified examples make a significant difference to the final robustness . As shown in Figure 1 ( c ) , compared with standard adversarial training ( dashed blue line ) with the CE loss , the final robustness is significantly improved when the outer minimization on misclassified examples is “ regularized ” ( solid green line ) by an additional term ( a KL-divergence term that was used previously in Zheng et al . ( 2016 ) ; Zhang et al . ( 2019 ) ) . The same regularization applied to correctly classified examples also helps the final robustness ( solid orange line ) , though not as significantly as for misclassified examples . Motivated by the above observations , we reformulate the adversarial risk to incorporate an explicit differentiation of misclassified examples in a form of regularization . We then propose a new defense algorithm to achieve this in a dynamic way during adversarial training . Our main contributions are : • We investigate the distinctive influence of misclassified and correctly classified examples on the final robustness of adversarial training . We find that the manipulation on misclassified examples has more impact on the final robustness , and the minimization techniques are more crucial than maximization ones under the min-max optimization framework . • We propose a regularized adversarial risk which incorporates an explicit differentiation of misclassified examples as a regularizer . Based on that , we further propose a new defense algorithm , called Misclassification Aware adveRsarial Training ( MART ) . • Experimentally , we show that adversarial robustness can be significantly improved over the stateof-the-art , by a specific focus on misclassified examples . It also helps improve recently proposed adversarial training with unlabeled data . 2 MISCLASSIFICATION AWARE ADVERSARIAL RISK . In this section , we propose a regularized adversarial risk that incorporates an explicit differentiation of misclassified examples . 2.1 PRELIMINARIES . We first define some notations . We use lower case and lower case bold face to denote scalars and vectors , respectively . We use upper case calligraphic symbols to denote sets . For a K-class ( K ≥ 2 ) classification problem , given a dataset { ( xi , yi ) } i=1 , ... , n with xi ∈ Rd as a natural example and yi ∈ { 1 , . . . , K } as its associated label , a DNN classifier hθ with model parameter θ predicts the class of an input example xi : hθ ( xi ) = arg max k=1 , ... , K pk ( xi , θ ) , pk ( xi , θ ) = exp ( zk ( xi , θ ) ) / K∑ k′=1 exp ( zk′ ( xi , θ ) ) , ( 2 ) where zk ( xi , θ ) is the logits output of the network with respect to class k , and pk ( xi , θ ) is the probability ( softmax on logits ) of xi belonging to class k. The adversarial risk ( Madry et al. , 2018 ) on dataset { ( xi , yi ) } i=1 , ... , n and classifier hθ can be defined with respect to the 0-1 loss ( Zhang et al. , 2019 ) as : R ( hθ ) = 1 n n∑ i=1 max x′i∈B ( xi ) 1 ( hθ ( x ′ i ) 6= yi ) , ( 3 ) where 1 ( · ) is the indicator function and B ( xi ) = { x : ‖x − xi‖p ≤ } denotes the Lp-norm ball centered at xi with radius . We will focus on the L∞-ball in this paper .

Summary: Neural Networks (NN) have been shown to be susceptible to various adversarial attacks i.e. if we perturb the "x" just a little, the output prediction changes. So, there has been much research devoted to how we can make NNs robust to such attacks. Typically, the adversarial examples that are used to train adversarially robust methods are generated from the correctly classified examples.