Distinguishability of Adversarial Examples

1 INTRODUCTION . Machine learning models including SVMs ( Biggio et al. , 2013 ) and especially deep neural networks ( Szegedy et al. , 2014 ) can be easily fooled by adversarial examples which are generated from the natural examples with small perturbations . Quite often , both machine learning models and humans can classify the natural examples such as the images of pandas with high accuracy , and humans can still classify the adversarial examples as pandas with high accuracy because the small perturbations are imperceptible ; however , machine learning models are fooled to misclassify adversarial examples as some targets such as gibbons ( Goodfellow et al. , 2015 ) desired by attackers . This intriguing property or vulnerability of machine learning models poses a critical challenge to machine learning security , and it impedes the wide application of machine learning in many important domains such as computer vision ( e.g. , for self driving cars ) and even in malware detection ( Biggio et al. , 2013 ; Xu et al. , 2016 ) . Furthermore , the discovery of new and powerful adversarial example generation methods such as ( Szegedy et al. , 2014 ; Goodfellow et al. , 2015 ; Carlini & Wagner , 2017 ; Chen et al. , 2018 ; Kurakin et al. , 2017b ; Madry et al. , 2018 ; Dong et al. , 2018 ; Papernot et al. , 2016a ; Moosavi-Dezfooli et al. , 2016 ; Sabour et al. , 2016 ; Uesato et al. , 2018 ) goes on without cessation , indicating to a certain extent the unlimited capabilities for attackers to continuously and easily fool machine learning models . On the other hand , even state-of-the-art defense approaches such as adversarial training ( Szegedy et al. , 2014 ; Goodfellow et al. , 2015 ) and defensive distillation ( Papernot et al. , 2016b ) still suffer from major limitations and can be circumvented ( Section 2 ) . Therefore , the unfortunate status quo is that attackers prevail over defenders . In this paper , from a unique angle , we propose to investigate two important research questions that concern the distinguishability of adversarial examples . Question 1 : are adversarial examples distinguishable from natural examples ? Question 2 : are adversarial examples generated by different methods distinguishable from each other ? If the answer to Question 1 will be positive , i.e. , given a certain classification task such as image classification , generated adversarial examples ( regardless of the objects they represent ) largely belong to one class while natural examples belong to the other class , then defenders can simply discard those adversarial examples to protect the machine learning models . If the answer to Question 2 will be positive , i.e. , adversarial examples generated by different methods clearly belong to different classes , defenders can better protect the machine learning models , for example , by incorporating the corresponding examples into the adversarial training process to enhance the robustness of the models . Besides such practical benefits , answering these two questions may also help researchers further identify the nature of adversarial examples . Formally , we consider a classification problem in adversarial environments as a multi-label classification problem . That is , upon seeing a new input such as an image , while the original task such as classifying the image as a certain object is important , it is also important to classify the image as a generated adversarial vs. a natural example in the first place . We formulate this multi-label classification problem in Section 3 to guide us in answering the two questions , and term the corresponding defense approach as defensive distinction , which distinguishes adversarial vs. natural examples and distinguishes adversarial examples generated by different methods to protect against the attacks . We design and perform experiments using the MNIST dataset to investigate the two research questions and evaluate the effectiveness of our defensive distinction approach . In our experiments , we consider multiple scenario-case combinations that defenders either know or do not know the neural network , source images , and methods as well as parameters used by attackers for generating adversarial examples . We obtain highly positive answers to both research questions . For example , in some typical cases , adversarial vs. natural examples can be distinguished perfectly with 100 % accuracy , while adversarial examples generated by different methods can be distinguished with over 90 % accuracy . Our experimental results demonstrate the strong distinguishability of adversarial examples , and demonstrate the value of the defensive distinction approach . We recommend that this unique defense approach should be seriously considered to complement other defense approaches . We make four main contributions in this paper : ( 1 ) we propose to investigate two important research questions that concern the distinguishability of adversarial examples ; ( 2 ) we formulate a classification problem in adversarial environments as a multi-label classification problem to answer the two questions ; ( 3 ) we propose and explore a unique defense approach termed as defensive distinction ; ( 4 ) we design and perform experiments to empirically demonstrate the strong distinguishability of adversarial examples and the value of our defensive distinction approach . 2 RELATED WORK . Adversarial examples can easily fool machine learning models , and they can be generated by a number of powerful methods . One representative method is L-BFGS ( Szegedy et al. , 2014 ) , which formulates the adversarial example generation problem as a box-constrained optimization problem and uses line-search to identify approximated optimal solutions . Another representative method is FGSM ( Fast Gradient Sign Method ) ( Goodfellow et al. , 2015 ) , which efficiently obtains an optimal max-norm constrained perturbation for generating adversarial examples . FGSM inspired researchers to pursue a few fine-grained refinements over it , resulting in powerful methods such as BIM ( Basic Iterative Method ) by Kurakin et al . ( 2017b ) , PGD ( Projected Gradient Descent ) by Madry et al . ( 2018 ) , and MIM ( Momentum Iterative Method ) by Dong et al . ( 2018 ) . L-BFGS also inspired researchers to design more powerful or general methods such as C & W by Carlini & Wagner ( 2017 ) and EAD ( Elastic Net Method ) by Chen et al . ( 2018 ) . JSMA ( Jacobian-based Saliency Map Approach ) ( Papernot et al. , 2016a ) is another representative method , which constructs adversarial saliency maps to guide the selection of input features for perturbation in multiple iterations . On the defense side , one representative approach is adversarial training ( Szegedy et al. , 2014 ; Goodfellow et al. , 2015 ) , which basically uses both the generated adversarial examples and the natural ( a.k.a clean ) examples to train the machine learning models with better regularization . Another representative approach is defensive distillation ( Papernot et al. , 2016b ) , which basically uses the same neural network architecture to train a distilled network based on the probability vectors produced by the original network , thus reducing the models ’ sensitivity to small input perturbations . Both approaches in essence focus on improving the generalization and thus robustness of the machine learning models that perform the original classification task . While representing the state-of-theart , both approaches still suffer from major limitations and can be circumvented ( Goodfellow et al. , 2018 ) . For example , defensive distillation was found to be ineffective against C & W attacks ( Carlini & Wagner , 2017 ) and black-box attacks ( Papernot et al. , 2017 ) , while adversarial training was found to be ineffective against adversarial examples generated by many iterative methods such as BIM ( Kurakin et al. , 2017a ) , PGD Madry et al . ( 2018 ) , and MIM ( Dong et al. , 2018 ) . Overall , powerful adversarial example generation methods exist and will continue to be proposed , while existing defense approaches are far from being perfect yet . The unfortunate status quo is that attackers prevail over defenders . Our work is likely one step toward making a change . 3 DEFENSIVE DISTINCTION PROTECTION APPROACH . We now present the problem formulation and the corresponding defensive distinction protection approach ; we also define a threat model that can be used for evaluating our approach . 3.1 MULTI-LABEL CLASSIFICATION FORMULATION OF THE PROBLEM . To date , a classification problem in adversarial environments is viewed as a single-label classification problem , where a function or model f : X → Y is learned to assign each input example xi ∈ X a single label yi ∈ Y , where the range Y is a set of m singletons representing m possible classes such as 10 possible digits . In other words , a single concept or semantic meaning is associated with each example . State-of-the-art defense approaches such as adversarial training and defensive distillation reviewed in Section 2 do not change this basic problem formulation . However , considering the two important distinguishability questions that we propose to investigate ( Question 1 : are adversarial examples distinguishable from natural examples ? Question 2 : are adversarial examples generated by different methods distinguishable from each other ? ) , it is straightforward for us to formulate a classification problem in adversarial environments as a multi-label classification problem . That is , a function or model f ′ : X → Y′ is learned to assign each input example xi a pair of labels represented by y′i = ( ai , bi ) ∈ Y′ . Here , the new range Y′ = Y × Z is the Cartesian product of Y ( i.e. , the range in the original classification task ) and Z , where Z is a set of n singletons representing n possible classes regarding if xi is an adversarial example . In the simplest situation , n = 2 and bi indicates either an example is natural ( i.e. , clean ) or adversarial . In a more complex situation , n > 2 and bi can indicate either an example is natural or generated by one of n− 1 different adversarial example generation methods . Under this new problem formulation , two concepts or semantic meanings 1 are associated with each example . The solution to the problem will help us answer the two research questions , and help defenders simply discard or leverage the identified adversarial examples to better protect their machine learning models as described in Section 1 . Moreover , one major advantage of a multi-label classification formulation and the corresponding multi-label learning is on exploring the intrinsic correlations between multiple labels . In real-world adversarial environments , this advantage is significant because it will allow researchers and defenders to perform in-depth analysis of the behavior of attackers . For example , do attackers create most adversarial examples targeting at some specific classes such as gibbons ? do attackers use different generation methods for different targeted classes ? and , do attackers use examples of some specific source classes to create adversarial examples targeting at some specific classes ( Appendix A ) ? These and perhaps other types of label correlation analyses will be valuable for inferring the real intentions of attackers , and even for revolutionizing the design of the original machine learning systems . 3.2 DEFENSIVE DISTINCTION APPROACH . We simply define the defensive distinction protection approach as training a model for a multi-label classification problem formulated in adversarial environments to identify adversarial examples and protect against them . So a training set will include both natural examples and generated adversarial examples with their ground truth bi values correspondingly labeled in y′i = ( ai , bi ) ∈ Y′ . The ground truth ai values for natural examples are obviously known based on some existing dataset such as MNIST . The ground truth ai values for adversarial examples can be missing ( Yu et al. , 2014 ) or not needed ( as in our solution described below ) depending on specific learning algorithms . 1More labels could be added to include more concepts or semantic meanings as described in Appendix A . Many multi-label learning algorithms have been proposed with different strategies or considerations on the order-of-correlations among labels ( Zhang & Zhou , 2014 ) . In this paper , we experiment with the simplest first-order strategy that basically transforms a multi-label classification problem into multiple single-label problems . More specifically , three models will be independently trained : • DDP-Model ( defensive distinction primary model ) : a single-label binary model for distinguishing an input example xi as either natural or adversarial . • Original-Model : a single-label multi-class model for the original task on classifying the input example xi as one of multiple possible objects such as one of 10 digits . • DDS-Model ( defensive distinction secondary model ) : a single-label multi-class model 2 for distinguishing known adversarial examples as generated by different specific methods . This simple strategy provides a baseline solution to the defensive distinction approach . It is simple in model formulation , computation , and evaluation , albeit high-order strategies would be more powerful in exploring label correlations ( Read et al. , 2009 ; Zhang & Zhou , 2014 ) .

Summary: Defensive Distinction (DD) is an interesting model for detecting adversarial examples. However, it leaves some key aspects of defense and distinction out. Firstly, one can argue that if you know the adversaries of your model you can simply regularize the model for them. Even if regularization doesn't work fully, the DD model still suffers since it can have its own adversarial examples. From distinction perspective, it would be hard to believe that every single adversarial example will be detected, at least not without some solid theoretical background. It seems that  and natural examples are being thrown at the DD model without an elegant approach.