Representation Quality Explain Adversarial Attacks

1 INTRODUCTION . Adversarial samples are slightly perturbed inputs that can make neural networks misclassify . They are carefully crafted by searching for variations in the input that , for example , could decrease the soft-labels of the correct class . Since they were discovered some years ago ( Szegedy , 2014 ) , the number of adversarial samples have grown in both number and types . Random noise was shown to be recognized with high confidence by neural networks ( Nguyen et al. , 2015 ) , universal perturbations , that can be added to almost any image to generate an adversarial sample , were shown to exist ( Moosavi-Dezfooli et al. , 2017 ) , and the addition of crafted patches was shown to cause networks to misclassify ( Brown et al. , 2017 ) . Only one pixel is enough to make networks misclassify ( Su et al. , 2017 ) . Such attacks can also be easily transferred to real-world scenarios ( Kurakin et al. , 2016 ) , ( Athalye & Sutskever , 2018 ) , which confers a big issue as well as a security risk for current deep neural networks ’ applications . Albeit the existence of many defences , there is not any known learning algorithm or procedure that can defend against adversarial attacks consistently . Many works have tried to defend by hiding or modifying the gradients to make neural networks harder to attack . However , a recent paper shows that most of these defences fall into the class of obfuscated gradients which have their shortcomings ( e.g. , they can be easily bypassed by transferable attacks ) ( Athalye et al. , 2018 ) . Additionally , the use of an augmented dataset with adversarial samples ( named adversarial training ) is perhaps one of the most successful approaches to construct robust neural networks ( Goodfellow et al. , 2014 ) , ( Huang et al. , 2015 ) , ( Madry et al. , 2018 ) . However , it is still vulnerable to attacks and has a strong bias in the type of adversarial samples used in training ( Tramèr et al. , 2018 ) . This shows that a deeper understanding of the issues is needed to enable more consistent defences to be created . Few works focused on understanding the reason behind such lack of robustness . In ( Goodfellow et al. , 2014 ) , it is argued that Deep Neural Networks ’ s ( DNN ) linearity are one of the main reasons . Recent investigations reveal that attacks are changing where the algorithm is paying attention ( Vargas & Su , 2019 ) , other experiments show that deep learning neural networks learn false structures that are easier to learn rather than the ones expected ( Thesing et al. , 2019 ) and an accuracy and robustness trade-off for models were shown to exist ( Tsipras et al. , 2019 ) . In this paper , we propose a methodology of how to evaluate the representation of machine learning methods . Based on these metrics , we reveal a link between deep representations ’ quality and attack susceptibility . Specifically , we propose a test called Raw Zero-Shot and two metrics to evaluate DNN ’ s representations . 1.1 RECENT ADVANCES IN ATTACKS AND DEFENSES . DNNs were shown vulnerable to many types of attacks . For example , the output high confidence results to noise images ( Nguyen et al. , 2015 ) , universal perturbations in which a single perturbation can be added to almost any input to create an adversarial sample are possible ( Moosavi-Dezfooli et al. , 2017 ) , the addition of image patches can also make them misclassify ( Brown et al. , 2017 ) . Moreover , the vulnerability can be exploited even with a single pixel , i.e. , changing a single pixel is often enough to make a DNNs misclassify ( Su et al. , 2017 ) . Most of these attacks can be transformed into real-world attacks by merely printing the adversarial samples ( Kurakin et al. , 2016 ) . Moreover , crafted glasses ( Sharif et al. , 2016 ) or even general 3d adversarial objects ( Athalye & Sutskever , 2018 ) can be used as attacks . Although many defensive systems were proposed to tackle the current problems , there is still no consistent solution available . Defensive distillation in which a smaller neural network squeezes the content learned by the original DNN was proposed ( Papernot et al. , 2016 ) . However , it was shown not to be robust enough ( Carlini & Wagner , 2017 ) . Adversarial training was also proposed as a defence , in which adversarial samples are used to augment the training dataset ( Goodfellow et al. , 2014 ) , ( Huang et al. , 2015 ) , ( Madry et al. , 2018 ) . With adversarial training , DNNs increase slightly in robustness but not without a bias towards the adversarial samples used and while still being vulnerable to attacks in general ( Tramèr et al. , 2018 ) . There are many recent variations of defenses in which the objective is to hide the gradients ( obfuscated gradients ) ( Ma et al. , 2018 ) , ( Guo et al. , 2018 ) ( Song et al. , 2018 ) . However , they can be bypassed by various types of attacks ( such as attacks not using gradients , transfer of adversarial samples , etc . ) ( Athalye et al. , 2018 ) , ( Uesato et al. , 2018 ) . There are a couple of works which are trying to understand the reason behind such lack of robustness . To citep some , in ( Goodfellow et al. , 2014 ) , it is argued that the main reason may lie in DNNs ’ lack of non-linearity . Another work argues that the perturbations cause a change in the saliency of images which makes the model switch the attention to another part of it ( Vargas & Su , 2019 ) . False structures that are easier to learn were also shown related to the problem ( Thesing et al. , 2019 ) . Moreover , in ( Tsipras et al. , 2019 ) , the accuracy and robustness trade-off was shown to exist . 1.2 ZERO-SHOT LEARNING . Zero-Shot learning is a method used to estimate unknown classes which do not appear in the training data . The motivation of Zero-Shot learning is to transfer knowledge from training classes to unknown classes . Existing methods approach the problem by estimating unknown classes from an attribute vector defined manually . Attribute vectors are annotated to both known and unknown classes , and for each class , whether an attribute , such as “ colour ” and “ shape ” , belongs to the class or not is represented by 1 or 0 . For example , in ( Lampert et al. , 2009 ) the authors proposed Direct Attribute Prediction ( DAP ) model , which learns each parameter for estimating the attributes from the target data . It estimates an unknown class of the source data which is estimated from the target data by using these parameters . Based on this research , other zero-shot learning methods have been proposed which uses an embedded representation generated using a natural language processing algorithm instead of a manually created attribute vector ( Zhang & Saligrama , 2016 ; Fu et al. , 2015 ; Norouzi et al. , 2013 ; Akata et al. , 2015 ; Bucher et al. , 2016 ) . In ( Zhang & Saligrama , 2015 ) , a different approach to estimate unknown classes is proposed . This method constructs the histogram of known classes distribution for an unknown class . In this approach , it is assumed that the unknown classes are the same if these histograms generated in the target domain and the source domain are similar . This perspective is similar to our approach because our method approach to represent an unknown class as the distribution of known classes . However , our objective is not estimating the unknown class , and we do not use the source domain . Our objective here is to analyze DNNs ’ representation by using this distribution . 2 ON THE LINKS BETWEEN ROBUSTNESS EVALUATION AND REPRESENTATION . In the canonical classification setting , the goal of a classifier is to achieve low expected loss : E ( x , y ) ∼D [ L ( x , y ; θ ) ] ( 1 ) Robustness against adversarial attacks is a slightly different setting . To achieve a high robustness in this setting , a classifier should have a lower adversarial loss in noise δ ∈ ∆ 1 : E ( x , y ) ∼D [ L ( x+ δ , y ; θ ) ] . ( 2 ) Considering Mean Squared Error ( MSE ) , we have : E ( x , y ) ∼D [ L ( x+ δ , y ; θ ) ] = E ( x , y ) ∼D [ ( f ( x+ δ ) − h ( x+ δ ) ) 2 ] + E ( x , y ) ∼D [ ( h ( x+ δ ) − ŷ ( x+ δ ) ) 2 ] , where h ( x ) = E [ hD ( x ) ] is the expected behavior of the prediction when averaged over many datasets , f ( x ) is the ground-truth and ŷ ( x ) = hD ( x ) is the output after learned on a given dataset D. For robustness to increase , adversarial training requires that datasets should have many noisy samples , i.e. , x+δ ∈ D. However , the more noise is added to images the moreD becomes close to all possible images RM∗N : lim ∆→∞ D = RM∗N ( 3 ) However , f ( x ) is undefined 2 for D ∈ RM∗N in which y /∈ C for the set of known classes C. Even a small amount of noise may be enough to cause y /∈ C and thus f ( x ) undefined . Would it be possible to evaluate the robustness and/or the quality of a model without a well defined y ? To answer this question we take into account an ideal representation z and the representation learned by the model ẑ : E [ ( f ( x+ δ ; z ) − h ( x+ δ ; ẑ ) ) 2 ] + E [ ( h ( x+ δ ; ẑ ) − ŷ ( x+ δ ; ẑ ) ) 2 ] Interestingly , although y is undefined , z represents the features learned and is well defined for any input . Moreover , by considering learned classes to be clustered in z space , unsupervised learning evaluation can be used to evaluate z even without a well defined y . We use here a famous clustering analysis index to evaluate clusters in z by their intracluster distance . In z , it is also possible to evaluate the representation of known and unknown classes which should share some features . Moreover , we hypothesize here that unknown classes should evaluate z with less bias because a direct map from input to output is inexistent . Any projection of the input in any of the feature maps or the output layer could be used as z . To take the entire projection into account , we use here z as the final projection of the input to the classes , i.e . z is the soft label array e .

Summary: This paper aims at revealing the relationship between the quality of deep representations and the attack susceptibility of deep classification models. To this end, they propose the zero-shot test to investigate the "quality" of learned representations for unknown classes. Specifically, they leverage two kinds of quality metrics on data of unknown classes. The first one is based on clustering named Davies-Bouldin Index which measures the compactness of intra-cluster. The second one is based on the difference of soft-label histogram distributions with/without unknown classes during training, which may describe the generalization for unknown classes or bias towards known classes of learned features. Finally, with these two metrics, they rank the quality of different models and compare such ranking results with the attack robustness obtained by different attack techniques on CIFAR-10 dataset.