Representation Quality Explain Adversarial Attacks

1 INTRODUCTION . Adversarial samples are slightly perturbed inputs that can make neural networks misclassify . They are carefully crafted by searching for variations in the input that , for example , could decrease the soft-labels of the correct class . Since they were discovered some years ago ( Szegedy , 2014 ) , the number of adversarial samples have grown in both number and types . Random noise was shown to be recognized with high confidence by neural networks ( Nguyen et al. , 2015 ) , universal perturbations , that can be added to almost any image to generate an adversarial sample , were shown to exist ( Moosavi-Dezfooli et al. , 2017 ) , and the addition of crafted patches was shown to cause networks to misclassify ( Brown et al. , 2017 ) . Only one pixel is enough to make networks misclassify ( Su et al. , 2017 ) . Such attacks can also be easily transferred to real-world scenarios ( Kurakin et al. , 2016 ) , ( Athalye & Sutskever , 2018 ) , which confers a big issue as well as a security risk for current deep neural networks ’ applications . Albeit the existence of many defences , there is not any known learning algorithm or procedure that can defend against adversarial attacks consistently . Many works have tried to defend by hiding or modifying the gradients to make neural networks harder to attack . However , a recent paper shows that most of these defences fall into the class of obfuscated gradients which have their shortcomings ( e.g. , they can be easily bypassed by transferable attacks ) ( Athalye et al. , 2018 ) . Additionally , the use of an augmented dataset with adversarial samples ( named adversarial training ) is perhaps one of the most successful approaches to construct robust neural networks ( Goodfellow et al. , 2014 ) , ( Huang et al. , 2015 ) , ( Madry et al. , 2018 ) . However , it is still vulnerable to attacks and has a strong bias in the type of adversarial samples used in training ( Tramèr et al. , 2018 ) . This shows that a deeper understanding of the issues is needed to enable more consistent defences to be created . Few works focused on understanding the reason behind such lack of robustness . In ( Goodfellow et al. , 2014 ) , it is argued that Deep Neural Networks ’ s ( DNN ) linearity are one of the main reasons . Recent investigations reveal that attacks are changing where the algorithm is paying attention ( Vargas & Su , 2019 ) , other experiments show that deep learning neural networks learn false structures that are easier to learn rather than the ones expected ( Thesing et al. , 2019 ) and an accuracy and robustness trade-off for models were shown to exist ( Tsipras et al. , 2019 ) . In this paper , we propose a methodology of how to evaluate the representation of machine learning methods . Based on these metrics , we reveal a link between deep representations ’ quality and attack susceptibility . Specifically , we propose a test called Raw Zero-Shot and two metrics to evaluate DNN ’ s representations . 1.1 RECENT ADVANCES IN ATTACKS AND DEFENSES . DNNs were shown vulnerable to many types of attacks . For example , the output high confidence results to noise images ( Nguyen et al. , 2015 ) , universal perturbations in which a single perturbation can be added to almost any input to create an adversarial sample are possible ( Moosavi-Dezfooli et al. , 2017 ) , the addition of image patches can also make them misclassify ( Brown et al. , 2017 ) . Moreover , the vulnerability can be exploited even with a single pixel , i.e. , changing a single pixel is often enough to make a DNNs misclassify ( Su et al. , 2017 ) . Most of these attacks can be transformed into real-world attacks by merely printing the adversarial samples ( Kurakin et al. , 2016 ) . Moreover , crafted glasses ( Sharif et al. , 2016 ) or even general 3d adversarial objects ( Athalye & Sutskever , 2018 ) can be used as attacks . Although many defensive systems were proposed to tackle the current problems , there is still no consistent solution available . Defensive distillation in which a smaller neural network squeezes the content learned by the original DNN was proposed ( Papernot et al. , 2016 ) . However , it was shown not to be robust enough ( Carlini & Wagner , 2017 ) . Adversarial training was also proposed as a defence , in which adversarial samples are used to augment the training dataset ( Goodfellow et al. , 2014 ) , ( Huang et al. , 2015 ) , ( Madry et al. , 2018 ) . With adversarial training , DNNs increase slightly in robustness but not without a bias towards the adversarial samples used and while still being vulnerable to attacks in general ( Tramèr et al. , 2018 ) . There are many recent variations of defenses in which the objective is to hide the gradients ( obfuscated gradients ) ( Ma et al. , 2018 ) , ( Guo et al. , 2018 ) ( Song et al. , 2018 ) . However , they can be bypassed by various types of attacks ( such as attacks not using gradients , transfer of adversarial samples , etc . ) ( Athalye et al. , 2018 ) , ( Uesato et al. , 2018 ) . There are a couple of works which are trying to understand the reason behind such lack of robustness . To citep some , in ( Goodfellow et al. , 2014 ) , it is argued that the main reason may lie in DNNs ’ lack of non-linearity . Another work argues that the perturbations cause a change in the saliency of images which makes the model switch the attention to another part of it ( Vargas & Su , 2019 ) . False structures that are easier to learn were also shown related to the problem ( Thesing et al. , 2019 ) . Moreover , in ( Tsipras et al. , 2019 ) , the accuracy and robustness trade-off was shown to exist . 1.2 ZERO-SHOT LEARNING . Zero-Shot learning is a method used to estimate unknown classes which do not appear in the training data . The motivation of Zero-Shot learning is to transfer knowledge from training classes to unknown classes . Existing methods approach the problem by estimating unknown classes from an attribute vector defined manually . Attribute vectors are annotated to both known and unknown classes , and for each class , whether an attribute , such as “ colour ” and “ shape ” , belongs to the class or not is represented by 1 or 0 . For example , in ( Lampert et al. , 2009 ) the authors proposed Direct Attribute Prediction ( DAP ) model , which learns each parameter for estimating the attributes from the target data . It estimates an unknown class of the source data which is estimated from the target data by using these parameters . Based on this research , other zero-shot learning methods have been proposed which uses an embedded representation generated using a natural language processing algorithm instead of a manually created attribute vector ( Zhang & Saligrama , 2016 ; Fu et al. , 2015 ; Norouzi et al. , 2013 ; Akata et al. , 2015 ; Bucher et al. , 2016 ) . In ( Zhang & Saligrama , 2015 ) , a different approach to estimate unknown classes is proposed . This method constructs the histogram of known classes distribution for an unknown class . In this approach , it is assumed that the unknown classes are the same if these histograms generated in the target domain and the source domain are similar . This perspective is similar to our approach because our method approach to represent an unknown class as the distribution of known classes . However , our objective is not estimating the unknown class , and we do not use the source domain . Our objective here is to analyze DNNs ’ representation by using this distribution . 2 ON THE LINKS BETWEEN ROBUSTNESS EVALUATION AND REPRESENTATION . In the canonical classification setting , the goal of a classifier is to achieve low expected loss : E ( x , y ) ∼D [ L ( x , y ; θ ) ] ( 1 ) Robustness against adversarial attacks is a slightly different setting . To achieve a high robustness in this setting , a classifier should have a lower adversarial loss in noise δ ∈ ∆ 1 : E ( x , y ) ∼D [ L ( x+ δ , y ; θ ) ] . ( 2 ) Considering Mean Squared Error ( MSE ) , we have : E ( x , y ) ∼D [ L ( x+ δ , y ; θ ) ] = E ( x , y ) ∼D [ ( f ( x+ δ ) − h ( x+ δ ) ) 2 ] + E ( x , y ) ∼D [ ( h ( x+ δ ) − ŷ ( x+ δ ) ) 2 ] , where h ( x ) = E [ hD ( x ) ] is the expected behavior of the prediction when averaged over many datasets , f ( x ) is the ground-truth and ŷ ( x ) = hD ( x ) is the output after learned on a given dataset D. For robustness to increase , adversarial training requires that datasets should have many noisy samples , i.e. , x+δ ∈ D. However , the more noise is added to images the moreD becomes close to all possible images RM∗N : lim ∆→∞ D = RM∗N ( 3 ) However , f ( x ) is undefined 2 for D ∈ RM∗N in which y /∈ C for the set of known classes C. Even a small amount of noise may be enough to cause y /∈ C and thus f ( x ) undefined . Would it be possible to evaluate the robustness and/or the quality of a model without a well defined y ? To answer this question we take into account an ideal representation z and the representation learned by the model ẑ : E [ ( f ( x+ δ ; z ) − h ( x+ δ ; ẑ ) ) 2 ] + E [ ( h ( x+ δ ; ẑ ) − ŷ ( x+ δ ; ẑ ) ) 2 ] Interestingly , although y is undefined , z represents the features learned and is well defined for any input . Moreover , by considering learned classes to be clustered in z space , unsupervised learning evaluation can be used to evaluate z even without a well defined y . We use here a famous clustering analysis index to evaluate clusters in z by their intracluster distance . In z , it is also possible to evaluate the representation of known and unknown classes which should share some features . Moreover , we hypothesize here that unknown classes should evaluate z with less bias because a direct map from input to output is inexistent . Any projection of the input in any of the feature maps or the output layer could be used as z . To take the entire projection into account , we use here z as the final projection of the input to the classes , i.e . z is the soft label array e .

Summary: This paper proposes to evaluate the robustness of the neural networks by extrapolating to the unseen classes. However, the authors only include evaluation for non-robust trained models, without considering the robust trained model, such as Madry et al. [1]. The conclusion is not convincing that the authors studied the robustness using only non-robust models, because it is well known that the accuracy for attacking non-robust model can be 100% (for CIFAR-10). It is useful if the authors study whether their method can be used to measure the robustness of the robust trained models.