Manifold Distance Judge, an Adversarial Samples Defense Strategy Based on Service Orchestration

Deep neural networks ( DNNs ) are playing an increasingly significant role in the modern world . However , they are weak to adversarial examples that are generated by adding specially crafted perturbations . Most defenses against adversarial examples focused on refining the DNN models , which often sacrifice the performance and computational cost of models on benign samples . In this paper , we propose a manifold distance detection method to distinguish between legitimate samples and adversarial samples by measuring the different distances on the manifold . The manifold distance detection method neither modifies the protected models nor requires knowledge of the process for generating adversarial samples . Inspired by the effectiveness of the manifold distance detection , we demonstrated a well-designed orchestrated defense strategy , named Manifold Distance Judge ( MDJ ) , which selects the best image processing method that will effectively expand the manifold distance between legitimate and adversarial samples , and thus , enhances the performance of the following manifold distance detection method . Tests on the ImageNet dataset , the MDJ is effective against the most adversarial samples under whitebox , graybox , and blackbox attack scenarios . We show empirically that the orchestration strategy MDJ is significantly better than Feature Squeezing on the recall rate . Meanwhile , MDJ achieves high detection rates against CW attack and DI-FGSM attack . 1 INTRODUCTION . Deep Neural Networks ( DNNs ) is the main research direction in artificial intelligence in recent years . Research and applications have shown that deep learning has replaced previous related technologies , and extraordinary breakthroughs have been made in image recognition , security-sensitive applications , and other fields ( Kaiming He & Sun , 2016 ) . However , DNNs are weak to adversarial attacks ( Christian Szegedy & Fergus , 2014 ) . Attackers can deceive the DNNs model by manipulating graph structure or node features , which results in graph counter disturbances , and limits their application in safety-critical systems . We can divide the attack into two different attack methods based on the attack principles . One is the gradient attack based on the known optimization model , and another one is based on the transfer attack model . Recent researches have shown that we can use the Manifold Hypothesis theory to explain deep learning models from the mapping of data in high-dimensional space and low-dimensional space ( Fefferman et al. , 2016 ) The Manifold Hypothesis states that real-world high-dimensional data lie on low-dimensional manifolds embedded within the high-dimensional space ( Aamari , 2019 ) . This Hypothesis is important in DNN . Manifold hypothesis can explain the learning and detection process of deep learning models in image classification , as well as the principle of attacking against attacks ( Fefferman & Narayanan , 2019 ) . The Manifold Hypothesis explains why machine learning techniques are able to find useful features and produce accurate predictions from datasets that have a potentially large number of dimensions . From another perspective , the manifold hypothesis also explains why the adversarial attack can attack successfully and achieve the effect of confusing the deep learning model . We can use the theory of manifold hypothesis to explain the three adversarial defense methods . Current defenses against adversarial examples follow three approaches : ( 1 ) Training the target classifier with adversarial examples , called adversarial training ( Christian Szegedy & Fergus , 2014 ; Ian J. Goodfellow & Szegedy , 2015 ) ; ( 2 ) Making target classifiers hard to attack by blocking gradient pathway ( Papernot et al. , 2016b ) , e.g. , defensive distillation and ( 3 ) training a classifier to distinguish between normal and adversarial examples ( Fischer et al. , 2017 ) , e.g. , Preprocessing . In the analysis of defense methods using the manifold hypothesis , we can prove that in the deep learning model , the accuracy and generalization of the model can not be taken into account at the same time . That is , when we improve the performance of one aspect , we will sacrifice another one . It also illustrates the necessity of building an adversarial sample defense system . Finally , we can also conclude that it is inevitable to produce adversarial samples . On the other hand , in systematic adversarial sample detection methods , there are also methods such as adversarial training , defensive distillation and preprocessing method . Among these methods , there are optimal defense methods for different attacks.Based on the previous manifold analysis and defense methods analysis , we propose an orchestration defense strategy , Manifold Distance Judge ( MDJ ) , which is combining the manifold and the image preprocessing method . This paper makes four contributions . First , through the analysis of the manifold hypothesis , we propose the manifold distance detection method to reduce the pass rate of adversarial samples by expanding the manifold distance between legitimate and adversarial samples . Second , simple image transformation can enhance the difference in feature distribution between adversarial samples and legitimate samples . Third , we propose a well-designed orchestration strategy , MDJ , for detecting adversarial examples . MDJ combines the manifold distance detection method and input image preprocessing method to obtain a high detection rate of adversarial samples . Meanwhile , the MDJ method we propose can be used as a supplement to other defense methods . Because the original model does not have to be changed , it is easy to combine with other defenses such as a detection method specifically for a certain type of attack . Moreover , we prove that there is an orchestration strategy which is a simple defense combination method in the image field , and its effect is better than the single simple defense , and the random combination of the weak defense combination method . 2 RELATED WORK . In this section , we classify adversarial attack methods and briefly summarize the defense methods of adversarial training , defensive distillation , and preprocessing-based techniques . 2.1 ADVERSARIAL SAMPLE GENERATION ALGORITHM . Gradient Optimization Model : In a gradient optimization attack model , the attacker knows all the information and parameters inside the model and can generate adversarial samples based on the gradient of a given model to attack the classification model ( Muñoz-González et al. , 2017 ) . We discuss the attacking algorithms used in our experiments , Fast Gradient Sign Method ( FGSM ) ( Goodfellow et al. , 2014 ) , Basic Iterative Method ( BIM ) ( Kurakin et al. , 2018 ) , DeepFool ( Moosavi-Dezfooli et al. , 2016 ) , Acobian Saliency Map Approach ( JSMA ) ( Papernot et al. , 2016a ) , Carlini/Wagner Attacks ( Carlini & Wagner , 2017 ) . Transfer Attacks : In the transfer attack model , because the attacker does not know the internal information of the model , it is necessary to query the target model multiple times , use the alternative model and other optimization methods to solve the parameters based on the query results , generate the adversarial sample according to the gradient of the alternative model , and classify model attacks . we discuss the attacking algorithms used in our experiment , Momentum Iterative Fast Gradient Sign Method ( MI-FGSM ) ( Dong et al. , 2018 ) , Diverse Input Iterative Fast Gradient Sign Method ( DI2FGSM ) ( Xie et al. , 2019 ) . 2.2 EXISTING DETECTION METHODS . Adversarial Training : The defense method is to superimpose the disturbance that maximizes the loss function in the sample , so it is effective for fixed or large disturbances , even if the attack method with obvious changes in the model gradient is easy to be detected.But adversarial training diminishes the ML model ’ s accuracy and can make the ML model more exposed to generalization ( Carmon et al. , 2019 ) . Another disadvantage of Adversarial training based defense techniques is that we need to retrain the model whenever some new attack samples are discovered . It will be hard to update all deployed ML models . Defensive Distillation : The essence of the defense method is to replace the model defense idea , that is , to train a new distillation network based on the original model and its output to classify or predict samples.But distillation techniques work by combining the double model , and the second model uses the first model knowledge to improve accuracy . The black-box attack ’ s recent improvement makes this out-of-date defense ( Chakraborty et al. , 2018 ) . The strong transfer-potential of adversarial samples across neural network models ( Papernot et al. , 2016c ) is the main reason for this method ’ s collapse . It is not robust as simplistic variation in a neural network can make the system exposed to attacks ( Carlini & Wagner , 2016 ) . ( He et al. , 2017 ) concluded that combining/ensemble weak defenses does not automatically improve a system ’ s robustness . Also , the ensemble technique remains static and vulnerable to a new attack . Processing-based Techniques : The defense method based on image preprocessing has a defensive effect on migration attacks and gradient attacks . Its essence is to reduce the noise of the sample , to reduce the interference of the adversarial disturbance to the model . For the choice of preprocessing method , we can not judge which attack is effective , and most of the selection of preprocessing method adopts enumeration method , which is complicated to calculate . 3 MANIFOLD HYPOTHESIS AND ADVERSARIAL ATTACK . In this section , we introduce the manifold hypothesis to explain the learning and detection process of the deep learning model , and discuss the effect of this hypothesis in adversarial attacks . 3.1 MANIFOLD HYPOTHESIS . The Manifold Hypothesis states that real-world high-dimensional data lie on low-dimensional manifolds embedded within the high-dimensional space ( Fefferman et al. , 2016 ) . It reflects the local smoothness of the decision function , helps to more accurately characterize the local area , and enables the decision function to better perform data simulation . It also explains why deep learning techniques are able to find useful features and produce accurate predictions from datasets that have a potentially large number of dimensions ( Fefferman et al. , 2016 ) . The fact that the actual data set of interest actually lives on in a space of low dimension , means that a given deep learning model only needs to learn to focus on a few key features of the dataset to make decisions . Many of the algorithms behind machine learning techniques focus on ways to determine these ( embedding ) functions . 3.2 MANIFOLD HYPOTHESIS APPLIED IN ADVERSARIAL DEFENSE . In the training stage , the models learn the manifold representation of the data and the optimal decision border of classification , taking classification task as an example . The model continuously maps different images on low dimensions to form a manifold and learns by confirming tags . In the model checking stage , the features of the picture are also mapped on low dimensions , and the distance from the manifold determined in the learning process is measured to determine the label output by the model . In the detection stage , the model extracts the characteristics of the input picture and compares it with the stored optimal decision border to determine the classification of the input picture . The optimal decision border affects the classification efficiency of the model . The reason is that too few data points cause the boundary shape to change during smoothing , which makes the appearance of adversarial samples inevitable ; on the other hand , due to the excessive pursuit of the accuracy of the boundary , it leads to overfitting , and reduce the generalization and robustness of the model . From the perspective of binary classification , there is no difference between adversarial samples and correctly classified negative samples . The reason is legitimate samples are extremely sparse in the distribution of high-dimensional space in the real world ( take images as an example , MxNx8 black and white images , only the value in the high-dimensional extremely sparse field is meaningful ) . So we can use low-dimensional manifolds to classify legitimate samples . For deep learning models , if the adversarial samples and legitimate samples usually do not appear in which the samples are discriminated in high-dimensional space , it can be found that they are closer to the manifold than the legitimate negative samples . It leads to the failure to correctly classify legitimate samples into negative samples on the low-dimensional manifold that successfully classifies them . From the perspective of two attack optimization models , there are two reasons for a classifier misclassifies an adversarial example : ( 1 ) The adversarial example is far from the boundary of the manifold of the task . ( 2 ) The adversarial example is close to the boundary of the manifold . If the classifier generalizes poorly off the manifold in the vicinity of the adversarial example , then misclassification occurs . The distribution probability of the adversarial sample and the legitimate sample is different , that is , the model decision probability of the adversarial sample and the legitimate sample should be different . There is a theoretical method that can distinguish between legitimate samples and adversarial samples , that is , using adversarial samples that do not appear in the high-dimensional space of legitimate samples for judgment . This type of attack method against samples is to cross the boundary of the manifold with the smallest distance , and their mapping positions in the low-dimensional space are different from the legitimate samples . We assume that if there is a method that can change the mapping position of the image in the high-dimensional space , and also change the mapping position of the adversarial sample in the low-dimensional space , we can let the model learn the manifold of these samples to compare them . Adversarial samples are distinguished and used as the basis for classifying and detecting samples .

Summary: This paper proposes a manifold distance based detection based against adversarial samples. In addition, it also proposes manifold distance judge for the adversarial defense. The focus of the paper is to make low-dimensional manifolds better express high-dimensional data; the experimental results seems that the proposed method has good performance, but there are still lots of space to improve.