Explaining Adversarial Examples with Knowledge Representation

1 INTRODUCTION . With the more widespread use of deep neural networks , the robustness and security of these networks have aroused the attention of both academic and industrial eyes . Among these adversarial examples is one of the most interesting as well as intriguing . Since the discovery of adversarial examples in CNNs from 2013Szegedy et al . ( 2013 ) , security and robustness has become a hot topic . Researchers have put efforts into finding out sources for adversarial examples and also developing methods for automatically generating these adversarial examplesGoodfellow et al . ( 2014 ) . Most these research focus on how certain perturbations lead to changes in decision boundaries . This paper discusses the origin of adversarial examples from a more underlying knowledge representation point of view . It provides a possible reason why adversarial examples exist for current networks and uses some experiments to prove this idea . Experiments also in some way show that adversarial examples can be derived from only the training data and totally network-independent . In addition , adversarial examples may be in more forms than the usual small perturbations . At last , possible ways to alleviate this issue are discussed . 1.1 RELATED WORK . Current adversarial attacks have become a systematic procedure . Some algorithms have been developed to deliberately generate these kinds of adversarial examples Goodfellow et al . ( 2014 ) ; MoosaviDezfooli et al . ( 2016 ) . After these examples have been generated , they can be injected back into the model to skew the classificationPapernot et al . ( 2017 ) . This can even serve as a universal attack model for other machine learning techniques . Some adversarial example generation techniques arise from the properties of neural networks themselves and are dependent on model architectures . Most of this kind of work has been done on image classification tasks like handwriting recognition ( MINST dataset ) and object recognition ( ImageNet ) . However , from related research work these years , it has been widely recognized that there are even more machine architectures vulnerable to adversarial attacking other than neural networksPapernot et al . ( 2016 ) . More recently , some other research has shown that adversarial examples maybe more widespread in our physical worldKurakin et al . ( 2016 ) . And there even exist universal perturbationsMoosaviDezfooli et al . ( 2017b ; a ) for a certain neural network that can generate examples that are both universal and adversarial . A recent paper further shows the existence of single-pixel attacks on image classification tasksSu et al . ( 2017 ) . Opposite to the attack techniques are the defense techniquesPapernot et al . ( 2015 ) ; Lu et al . ( 2017a ) . Some research has also been on this area . The most straightforward idea for defense is including some adversarial examples as inputs of the training set and let the neural network also learn what adversarial examplesTramèr et al . ( 2017a ) are like.DeepDefenseYan et al . ( 2018 ) incorporates an adversarial perturbation-based regularization into classification objectives . A quite optimistic view comes from the research on multi-camera view indicating adversarial examples can be constrained by taking inputs of an object from different angles of viewLu et al . ( 2017b ) . And this multi-camera design aims to prove that adversarial examples may not be obvious for autonomous driving which must involve multiple cameras and scaling . However , some research shows that adversarial examples can directly work on image and scene segmentationsMetzen et al . ( 2017 ) ; Xie et al . ( 2017 ) . Other researchers also put many efforts into exploring the underlying reasons for these adversarial examples . The most direct one would be the linear vibration caused by vector computationGoodfellow et al . ( 2014 ) ; Tramèr et al . ( 2017b ) . This is the cause given by the fast gradient method paper . A more recent trend is that researchers try to use geometrical and topological methods to explore the perturbation effects on high-dimensional decision spaceWang et al . ( 2016 ) ; Tanay & Griffin ( 2016 ) ; Tramèr et al . ( 2017b ) ; Liu et al . ( 2016 ) . All these research shows a trend that people are more and more eager to undermine the principles of deep neural networks and extend their applications . 1.2 GENERAL IDEA . Even though we still have no clear idea how eyes and vision of human beings actually work on the neural level , human eyes are more error-resistant . It is not easy to see such kind of adversarial examples intuitively . Most of these kinds of examples are generated by carefully designed algorithms and procedures . This complexity to some extent shows that adversarial examples may only occupy a small percentage for total image space we can imagine with pixel representations . To the best of our knowledge , this paper should be the first one that discusses adversarial examples from the internal knowledge representation point of view . The rest of this paper is organized as follows : • The representation section gives a formal description of the explanation and illustrates why adversarial examples exist for current architectures . • The experiment section proposes a one-step distribution imitation procedure to do some distribution imitations . Experiments show that simply by distribution imitation on a training set without any knowledge of the classifier may still lead to obvious impacts on classification results using the network . The last discussion section concludes the paper and discusses some potential solutions from knowledge representation perspective . 2 KNOWLEDGE REPRESENTATION OF NETWORK . When people think of an object , the object usually can be depicted in a quite abstract way . For example , when asked what is a car , we can think of a vehicle with several wheels and a driver steering in it . Even though for one task , there could be a large number of neurons involved , at least on high-level the encoding of the information should still be in a sparse way . For computers , the modern machine also set sparsity and abstraction as goals to seek . However , due to the lack of accurate knowledge of optimal representations , current deep networks actually choose an alternative way with redundant parameters to depict tasks and objects . For human beings , both abstraction and sparsity can be achieved with hierarchical storage of knowledge . This can be similar to object-oriented programming . We can think of an object with its attributes . Going back to the object recognition tasks , human beings can actually learn from one or a few prototypes for a certain object and know how this prototype can be transformed into different plausible forms . As a result , when recognizing an object , we are actually recognizing the object as well as how it is transformed from the prototype of this object to what we see . Current multi-layer neural networks are partly enlightened by the hierarchical ideology . Even though throughout these years , the detailed architectures of deep neural networks have evolved a lot so as the accuracy , the most fundamental one is still the AlexNetKrizhevsky et al . ( 2012 ) . There are many layers but two main phases in this network for image classification tasks . The first phase mainly uses convolutional layers to extract local features from different scales . The second mainly uses fully connected layers to add up local features from the first phase with weights to construct a higher-level image entity . At last outputs from the second phase go through a Softmax classifier and give out probabilities for each possible class . As described above , the architecture of AlexNet is quite intuitive . However , there still exists a great gap between abstract and sparse representation from human beings and neural networks . An ideal execution procedure can be simulated here to see how neural networks actually represent knowledge through the process . Consider the two-step procedure of extraction and transformation . We can define the output of a network as : output = T ·X , where X refers to the extracted part after the convolution and pooling part and T refers to the transformations . For a given network , different inputs can have different extracted patterns of X , here define the extracted X = [ X1 , X2 ] consisting of two parts . Correspondingly , different inputs can activate different parts in the transformation , so we can have the output denoted as : output = T ·X = [ T1 , T2 ] · [ X1 , X2 ] T ( 1 ) We can have inputs from the same class ( same label ) . Some extreme cases can be considered here . Suppose for the first input , it is only extracted to be [ X1 , 0 ] . For this input : output1 = [ T1 , T2 ] · [ X1 , X2 ] T = T1 ·X1 + 0 = T1 ·X1 ( 2 ) In the same way , there could exist a second input that is only extracted to be [ 0 , X2 ] ( for example , symmetric data ) . For this input : output2 = [ T1 , T2 ] · [ X1 , X2 ] T = 0 + T2 ·X2 = T2 ·X2 ( 3 ) After the first input , parameters for the transformation T = [ T1 , 0 ] can be determined in the form of the extracted input . For the ideal condition , the classifier should give the largest value for the correct class and very small but equal values for other classes before the softmax gives out the final probability distribution . The extreme case for the ideal softmax is to give positive infinite for the correct and negative infinite for others . So the outputs discussed above should be a fixed value when two inputs belong to the same class ( they have the same output results ) . Similarly , in the second step , given the same output result , there should be : T2 ·X2 = T1 ·X1 ( 4 ) T2 = ( T1 ·X1 ) ·X−12 ( 5 ) Now the transformations are denoted by the first two inputs and we are given a third input and its output pattern can be denoted as X3 = [ A ·X1 , B ·X2 ] , where A , B are both matrices . Put this back into the learned system of output = [ T1 , T2 ] · [ X1 , X2 ] T , we can get the output result : output3 = [ T1 , T2 ] · [ A ·X1 , B ·X2 ] −1 ( 6 ) = [ T1 , ( T1 ·X1 ) ·X−12 ] · [ A ·X1 , B ·X2 ] −1 ( 7 ) So the system actually processes the test input into : output3 = T1 ·A ·X1 + ( T1 ·X1 ) ·X−12 ·B ·X2 ( 8 ) That actually indicates an interesting fact that what current neural networks have extracted at last are not only local features with combinations but also a weighted sampling of all training inputs . The visualization techniquesZeiler & Fergus ( 2014 ) can help show some elements of truth in this aspect . We can use the DeepDream tool to visualize individual feature channels and their combinations to explore the space of patterns learned by the neural networkMordvintsev et al . ( 2015 ) . The tool returns images that strongly activate the channels within the network to highlight the features learned by a neural network . Nine random channels are chosen , they are ’ balloon ’ , ’ container ship ’ , ’ grey whale ’ , ’ aircraft carrier ’ , ’ ashcan ’ , ’ radio ’ , ’ trolleybus ’ , ’ revolver ’ and ’ passenger car ’ . It can be seen in Figure 1 that in the last layer with corresponding labels , the objects can still be recognized from their round and very similar repeating structures . For relative complex objects like ’ container ship ’ or ’ revolver ’ they can only ensemble similar shapes that are obviously overlapping with each other . This , in some way , proves the point that the knowledge storage and representation of current neural networks are not exactly sparse prototype based . Current networks actually use a hybrid distribution of all its prototypes from training and possible transformations to represent a certain class of object . And the accumulation of these makes the space distances between different types of objects decision boundaries become smaller . When they become small enough , even small perturbations become possible to lead the classifier into a wrong result if imposed in certain directions . That may be the origin of adversarial examples . The Figure 2 above provides a simple description of this idea . The first one represents the way of differentiating transformations in the same class . One class includes two normal distributions ( after transformations ) N ( 10,3 ) and N ( 30,4 ) . There is one simple explanation of why one prototype of a class is reasonable to be denoted as an object . When a human is learning a class prototype from a physical object , there are actually an unlimited number of images from different angles and views streaming into human brains . And a large number of samples from a same class prototype will be converged to be a normal distribution . The other class includes one normal distribution N ( 45,6 ) . In this representation approach , we can actually easily differentiate between these prototypes with representations . Even their mean values are different enough . However , as we have discussed , the representation in current neural networks is more likely to be in the second figures form when multiple prototypes and transformations are involved . We can see in the second figure of Figure 2 , the representation of the first class combines two subclasses together . From the properties of normal distributions , the resulting hybrid representation becomes N ( 40 , 5 ) . This becomes more similar to the second class and makes it more difficult to classify . The absolute difference in means decreases from 45-30=15 to the smaller 45-40=5 . The classifier has a lowest precision it can recognize . If right now a perturbation is added to make this difference even smaller to a certain degree , the classification can not be guaranteed any more . In summary , human beings can detect different objects as well as their transformations at the same time . CNNs do not separate these two . This makes final layers extracting not pure objects but in reality , a probability density distribution of the objects and its different states after transformations . Adversarial examples can arise from this underlying form of hybrid knowledge representation by imitating the distributions of a target class .

Summary: The paper discusses two ways of constructing adversarial examples (images) using PCA+knn in the input space. Compared to the litterature on adversarial examples, the modifications proposed by the authors are clearly visible to the human eye and the resulting images do not seem natural (see Figure 4 and 5). The authors acknowledge this difference between their work and the state-of-the-art (e.g., "Modified images are sometimes visible but still can keep original structures and information, this shows that adversarial images can be in more forms than small perturbations", Section 4.1), but it remains unclear why generating such images would be interesting in practice.