Manifold Distance Judge, an Adversarial Samples Defense Strategy Based on Service Orchestration
Authors: Mengxin Zhang
Xiaofeng QIU
Venue: ICLR 2022 Submitted (2022)
Keywords: service orchestration
manifold distance detection
adversarial example
neural network.

Abstract: Deep neural networks (DNNs) are playing an increasingly significant role in the modern world. However, they are weak to adversarial examples that are generated by adding specially crafted perturbations. Most defenses against adversarial examples focused on refining the DNN models, which often sacrifice the performance and computational cost of models on benign samples. In this paper, we propose a manifold distance detection method to distinguish between legitimate samples and adversarial samples by measuring the different distances on the manifold. The manifold distance detection method neither modifies the protected models nor requires knowledge of the process for generating adversarial samples. Inspired by the effectiveness of the manifold distance detection, we demonstrated a well-designed orchestrated defense strategy, named Manifold Distance Judge (MDJ), which selects the best image processing method that will effectively expand the manifold distance between legitimate and adversarial samples, and thus, enhances the performance of the following manifold distance detection method. Tests on the ImageNet dataset, the MDJ is effective against the most adversarial samples under whitebox, graybox, and blackbox attack scenarios. We show empirically that the orchestration strategy MDJ is significantly better than Feature Squeezing on the recall rate. Meanwhile, MDJ achieves high detection rates against CW attack and DI-FGSM attack.

Under review as a conference paper at ICLR 2022
MANIFOLD DISTANCE JUDGE, AN ADVERSARIAL SAMPLES DEFENSE STRATEGY BASED ON SERVICE ORCHESTRATION
Anonymous authors Paper under double-blind review
ABSTRACT
Deep neural networks (DNNs) are playing an increasingly significant role in the modern world. However, they are weak to adversarial examples that are gener- ated by adding specially crafted perturbations. Most defenses against adversarial examples focused on refining the DNN models, which often sacrifice the per- formance and computational cost of models on benign samples. In this paper, we propose a manifold distance detection method to distinguish between legiti- mate samples and adversarial samples by measuring the different distances the manifold. The manifold distance detection method neither modifies the protected models nor requires knowledge of the process for generating adversarial samples. Inspired by the effectiveness of the manifold distance detection, we demonstrated well-designed orchestrated defense strategy, named Manifold Distance Judge (MDJ), which selects the best image processing method that will effectively ex- pand the manifold distance between legitimate and adversarial samples, and thus, enhances the performance the following manifold distance detection method. Tests the ImageNet dataset, the MDJ is effective against the most adversar- ial samples under whitebox, graybox, and blackbox attack scenarios. We show empirically that the orchestration strategy MDJ is significantly better than Fea- ture Squeezing on the recall rate. Meanwhile, MDJ achieves high detection rates against CW attack and DI-FGSM attack.
INTRODUCTION
Deep Neural Networks (DNNs) is the main research direction in artificial intelligence in recent years. Research and applications have shown that deep learning has replaced previous related tech- nologies, and extraordinary breakthroughs have been made in image recognition, security-sensitivv applications, and other fields(Kaiming He & Sun 2016).. However, DNNs are weak to adversarial attacks Szegedy & Fergus 2014). Attackers ing graph structure or node features, which results in graph counter disturbances, and limits their application in safety-critical systems. We can divide the attack into two different attack methods based on the attack principles. One is the gradient attack based on the known optimization model, and another one is based the transfer attack model.
deceive the DNNs model by manipulat-
Recent researches have shown that we can use the Manifold Hypothesis theory to explain deep learning models from the mapping of data space(Feiferman et al,, The Manifold Hypothesis states that real-world high-dimensional data lie on low-dimensional manifolds embedded within the high-dimensional space(Aamari 2019). This Hypothesis is important in DNN. Manifold hypothesis can explain the learning and detection process of deep learning models in image classification, as well as the principle of attacking against attacks(Fefferman & Narayanan, 2019). The Manifold Hypothesis explains why machine learning techniques are able to find useful features and produce accurate predictions from datasets that have potentially large number of dimensions. From another perspective, the manifold hypothesis also explains why the adversarial attack can attack successfully and achieve the effect of confusing the deep learning model.
-ith-eiinniional space and low-dimensional

---

Under review as a conference paper at ICLR 2022
We can use the theory of manifold hypothesis explain the three adversarial defense methods. Current defenses against adversarial examples follow three approaches: (1) Training the target clas- sifier with adversarial examples, called adversarial ChhritngCCrrstiann Szegedy & Fergus [014 lan
J. Goodfellow & Szegedy, 2013); (2) Making target classifiers hard to attack by blocking gradient pahhaayPPaerrnot al. 2016b), e-g. defensive distillation and (3) training classifier distinguish between normal and ad adversarial examples et al. 2017). c.g., Preprocessing.
In the analysis of defense methods using the manifold hypothesis, we can prove that in the deep learning model, the accuracy and generalization of the model cannot be taken into account at the same time. That is, when we improve the performance of one aspect, we will sacrifice another one. It also illustrates the necessity of building an adversarial sample defense system. Finally, we can also conclude that it inevitable to produce adversarial samples. On the other hand, in systematic adversarial sample detection methods, there are also methods such as adversarial training, defensive distillation and preprocessing method. Among these methods, there are optimal defense methods for different attacks. Based on the previous manifold analysis and defense methods analysis, we propose an orchestration defense strategy, Manifold Distance Judge (MDJ), which is combining the manifold and the image preprocessing method.
This paper makes four contributions. First, through the analysis of the manifold hypothesis, we propose the manifold distance detection method to reduce the pass rate of adversarial samples by expanding the manifold distance between legitimate and adversarial samples. Second, simple im- age transformation can enhance the difference in feature distribution between adversarial samples and legitimate samples. Third, we propose a well-designed orchestration strategy, MDJ, for detect- ing adversarial examples. MDJ combines the manifold distance detection method and input image preprocessing method to obtain a high detection rate of adversarial samples. Meanwhile, the MDJ method we propose can be used as supplement to other defense methods. Because the original model does not have to be changed, it is easy to combine with other defenses such as a detection method specifically for a certain type of attack. Moreover, we prove that there an orchestration strategy which is simple defense combination method in the image field, and its effect is better than the single simple defense, and the random combination of the weak defense combination method.
2 RELATED WORK
In this section, classify adversarial attack methods and briefly summarize the defense methods of adversarial training, defensive distillation, and preprocessing- based techniques.
2.1 ADVERSARIAL SAMPLE GENERATION ALGORITHM
Gradient Optimization Model: In gradient optimization attack model, the attacker knows all the information and parameters inside the model and can generate adversarial samples based on the gra- dient of a given model to attack the classification model(MMuñoz-Gnnzial
et al. (017). We discuss the attacking algorithms used in our experiments. Fast Gradient Sign Method
eet al. 2014). Basic Iterative Method (Kurki et al. 2018), DeepFooll MMoosviiDeztooli
jet al. 2016 Acobian Saliency Map Approach (JSMA) (Papernot et al. 2016a), Carlini/Wanner
Attacks (Ctaacs(CCalimii & Wagner 2017).
Transfer Attacks: In the transfer attack model, because the attacker does not know the internal information of the model, it is necessary to query the target model multiple times, the alternative model and other optimization methods to solve the parameters based on the query results, generate the adversarial sample according the gradient of the alternative model, and classify model attacks. we discuss the attacking algorithms used in our experiment, Momentum Iterative Fast Gradient Sign Method (MI-FGSM) et al. Divere Input Iterative Fast Gradient Sign Method (DI FGS et al. 2019).
2.2 EXISTING DETECTION METHODS
Adversarial Training: The defense method is to superimpose the disturbance that maximizes the loss function in the sample, so is effective for fixed or large disturbances, even if the attack method with obvious changes in the model gradient is easy be detected. But adversarial training diminishes
2

---

Under review as a conference paper at ICLR 2022
the ML model's accuracy and can make the ML model more exposed to generalization Carmon et al., 2019). Another disadvantage of Adversarial training based defense techniques is that we need to retrain the model whenever some new attack samples are discovered. It will be hard to update all deployed MI. models.
The essence of the defense method is replace the model defense idea, that is, to train a new distillation network based on the original model and its output to classify or predict samples. But distillation techniques work by combining the double model, and the second model uses the first model knowledge to improve accuracy. The black-box attack's recent improvement makes this out-of-date defense (Chakraborty et al. (2018). The strong pooteee-potential of adver- sarial samples across neural network models (Papernot et al., (016c) is the main reason for this method's collapse. It is not robust as simplistic variation exposed to attacks (Carlini & Wagner 2016). (He et al. 2017) concluded that combining/ensemble weak defenses does not automatically improve a system's 's robustness. Also, the ensemble technique remains static and vulnerable to a new attack.
Defensive Distillation:
neural network can make the system
Processing-basel Techniques:
The defense method based on image preprocessing has a defensive effect on migration attacks and gradient attacks. Its essence is to reduce the noise of the sample, to reduce the interference of the adversarial disturbance to the model. For the choice of preprocessing method, cannot judge which attack is effective, and most of the selection of preprocessing method adopts enumeration method, which is complicated to calculate.
3 MANIFOLD HYPOTHESIS AND ADVERSARIAL ATTACK
In this section, we introduce the manifold hypothesis to explain the learning and detection process of the deep learning model, and discuss the effect of this hypothesis in adversarial attacks.
3.1 MANIFOLD HYPOTHESIS
The Manifold Hypothesis states that real-world high-dimensional data lic on low-dimensional man- ifolds embedded within the high-dimensional space(Fefferman et al. 2016). reflects the local smoothness of the decision function, helps more accurately characterize the local area, and en- ables the decision function to better perform data simulation. It also explains why deep learning techniques are able find useful features and produce accurate predictions from datasets that have a potentially large number of dimensions (Farrionnsseeffaannnnm et al.) 2016). The fact that the actual data set of interest actually lives on in a space of low dimension, means that a given deep learning model only needs to learn to focus on a few key features of the dataset to make decisions. Many of the algorithms behind machine learning techniques focus on ways to determine these (embedding) functions.
3.2 MANIFOLD HYPOTHESIS APPLIED IN ADVERSARIAL DEFENSE
Legicaaal exauppee
0.8
0.4
02
Mimillll
0.0
Adversaial example
02
1 de esouple
05
Adecaazia coargle
Figure I: Manifold learning defense
3

---

Under review as a conference paper at ICLR 2022
In the training stage, the models learn the manifold representation of the data and the optimal deci- sion border of classification, taking classification task as an example. The model continuously maps different images on low dimensions to form manifold and learns by confirming tags. In the model checking stage, the features of the picture are also mapped on low dimensions, and the distance from the manifold determined in the learning process is measured to determine the label output by the model. In the detection stage, the model extracts the characteristics of the input picture and com- pares it with the stored optimal decision border to determine the classification of the input picture. The optimal decision border affects the classification efficiency of the model. The reason is that too few data points cause the boundary shape to change during smoothing, which makes the appearance of adversarial samples inevitable; on the other hand, due to the excessive pursuit of the accuracy of the boundary, leads to overfitting, and reduce the generalization and robustness of the model.
From the perspective of binary classification, there is no difference between adversarial samples and correctly classified negative samples. The reason is legitimate samples are extremely sparse in the distribution of high-dimensional space in the real world (take images as example, MxNx8 black and white images, only the value in the high-dimensional extremely sparse field is meaningful), So use low-dimensional manifolds to classify legitimate samples. For deep learning models, the adversarial samples and legitimate samples usually do not appear in which the samples are discriminated in nigh-dimensional space, it can be found that they are closer to the manifold than the legitimate negative samples. It leads to the failure to correctly classify legitimate samples into negative samples on the low-dimensional manifold that successfully classifies them.
From the perspective of two attack optimization models, there are two reasons for a classifier mis- classifies an adversarial example: (1) The adversarial example is far from the boundary of the mani- fold of the task. (2) The adversarial example is close to the boundary of the manifold. If the classifier generalizes poorly off the manifold in the vicinity of the adversarial example, then misclassification occurs. The distribution probability of the adversarial sample and the legitimate sample is different, that is, the model decision probability of the adversarial sample and the legitimate sample should be different.
There is theoretical method that can distinguish between legitimate samples and adversarial sam- ples, that is, using adversarial samples that do not appear in the high-dimensional space of legitimate samples for judgment. This type of attack method against samples is to cross the boundary of the manifold with the smallest distance, and their mapping positions in the low-dimensional space are different from the legitimate samples. We assume that if there is a method that can change the map- ping position of the image in the high-dimensional space, and also change the mapping position of the adversarial sample in the low-dimensional space, we can let the model learn the manifold of these samples to compare them. Adversarial samples are distinguished and used the basis for classifying and detecting samples.
3.3 BASIC IDEA OF ADVERSARIAL SAMPLE DETECTION
Regarding how the model classifies samples, the following propositions can prove that there is detection method that can detect adversarial samples and explain the existing detection methods. We can assume that the manifold is proposition A: then there manifold M and algorithm A, if is distributed as P(X), then x is on M. The defense detection uses its negative proposition B: that is, there are manifold M and algorithm A If the distribution of x is not P(X), then x is not on M*.
The classification model is easy to be attacked because the dimensionality of the manifold has been excessively reduced. Increasing the dimensionality of the manifold can distinguish adversarial sam- ples, but it is difficult to solve the direction of increasing dimensionality. Based on the above anal- ysis, we can see that is feasible to use the manifold analysis deep learning model, and it is also feasible to apply the concept of the manifold to classification detection and to explain the distinction between legitimate samples and adversarial samples.
The focus of using the principle of manifold hypothesis and improving detection performance is how to make low-dimensional manifolds better express nigh-dimensional data. For each defense method, there should be corresponding optimal defense attack method, and for each attack, there should be an optimal defense method. We take the analysis of three more common defense methods examples from the perspective of manifold.
4

---

Under review as a conference paper at ICLR 2022
Adversarial training: this method can only defend against attacks added in the training phase, the reason is the model extracts the features of the adversarial samples formed by the attack method during training and thus perform defense.
defensive distillation: whether it can build its optimized network is critical to the effectiveness of this defense method. This is determined by whether the optimized network can accurately obtain the optimal discrimination boundary. If the learning effect the boundary is perfect, the performance of defense against known attacks will be greatly improved. But for unknown attacks, the sample classification effect still cannot be achieved.
preprocessing methods: preprocessing technologies can defend against different attacks. This is because different preprocessing methods have different effects on the samples, leading to changes in the representation of the picture in the high-dimensional space. For different attack methods, the changes of the attack samples are not the same, and the characteristies of the samples in the low-dimesiional space also change, resulting in different attack methods for defense.
Therefore, this paper uses the Heuristics method to learn the original high- digh-dimensional sample space which changes in the low-dimensional manifold. We propose a new idea of service orchestration strategy to combine different methods to expand the manifold distance.
4 THE MEANING OF ORCHESTRATION STRATEGY
Regarding the attack method, whether it is an attack based on an optimization model or a migration model, the ultimate goal is to make the label classification of the attack sample different from the original correct classification. From the perspective of manifold analysis, the purpose of the attack method is find the fastest direction of gradient descent, derive the gradient, that the shape of the adversarial sample in the high-dimensional space changes. It can also affect the mapping graph of the sample in the low-dimensional space, so that the adversarial sample crosses the boundary of the graph classification, to achieve the goal of a successful attack.
The general idea of the orchestration strategy should be able to change the mapping position of the picture in the high-dimensional space through picture processing and display this change. Now that the defense method can be connected with the attack method through the manifold, how to use this idea becomes the key to the orchestration strategy. Since the probability output of the deep learning model can reflect the distance of manifold distance between different samples, combined with the above analysis of the preprocessing method that has the characteristics of defending against gradient optimization models and transfer attack models, it becomes an ideal orchestration strategy idea. Here, propose the orchestration strategy Manifold Distance Judge (MDJ) and it uses the output of deep learning and preprocessing methods to expand the distance between manifolds. The MDJ expressed in the form of combination methods in the experimentand achieved excellent results in the ImageNet dataset(Deng et al. 2009).
OUR METHODS: MANIFOLD DISTANCE JUDGE
In this section, we introduce the Manifold Distance Judge (MDJ) strategy. It uses the manifold detection method to learn the manifold of samples and uses image preprocessing methods to expand the diversity of manifolds between samples. Manifold Distance Judge essentially an orchestration defense strategythat achieves the optimal defense effect according to the characteristics of different defense methods.
5.1 BASIC IDEA OF OUR METHODS
Manifold distance judge (MDJ) is detector that can detect whether the input sample is adversarial. The detector uses the combination of manifold distance detection method and the image preprocess- ing changes to represent the subtle changes of the input samples in the high-dimensional space in the diw-diimnsional space, achieving the purpose of identifying adversarial samples.
The detector is inspired by manifold hypothesis orchestration strategy and can exist independently as a detection system. This allows our method to combined with other defenses, promoted, and enhance detection performance. In the selection of detection methods, the focus of improving
5

---

Under review as a conference paper at ICLR 2022
performance lies in how to change the input mapping position in the high-dimensional space in deep learning and how to represent the change of position distance.
5.2 DETECTOR BASED ON PREPROCESSING
To solve the problem of how change the mapping position of the picture in the high- nigh-dimensional space, we get inspiration from the preprocessing process of the image. The transformation of the image can intuitively change the feature representation of the sample, that is, change the mapping position of the image from high-dimensional space.
For legitimate samples, this transformation does not change the mapping position much. But for adversarial samples, different transformations change the spatial mapping of the picture a lot, which verified the subsequent experimental process. We list the preprocessing methods used as fol- lowing:
Random Zoom: (Guo et al. 2017) considered cropping and rescaling of an image as one of their defenses, which is effectively zooming in portion of the image, an approach that was defeated by (Athalye et al. 20). We reuse this as one of our defenses, where the distance from each edge of the image cropped by U[10, 50), independently for each edge.
Swirl: We introduce a simple defense which is to apply a weak swirl to the image, rotating the pixels around a randomly selected point in the image. The radius of intensity is randomly selected from U110, 200], and strength from UOO.1, 2.0]. The angle of rotation, center of rotation, and radius of effect are all randomized Raff et al. 2019).
Median Filter: We use simple median filter, and follow the same approach as the Guassian blur. The radius of the blur kernel is chosen from r < U[2, 5], with 50% chance all channels are forced to use the same radius(Raff et al. 2019).
JPEG Noise: Using lossy JPEG compression introduce artifacts was introduced by (Goodfellow et al. 2016). Their work looked at how different values of the JPEG compression level (a range from to 100) reduced the impact of adversarial attacks for different values of 6 S 16. However, was subsequently defeated, having 0% effectiveness (Athalye et al. 201 8a). When using this approach, we randomize by selecting the compression level from U[55, 95].
5.2.1 DETECTOR BASED ON PROBABILITY DIVERGENCE
To indicate the change of position distancewe propose manifold distance detection method, which utilize the output probability of softmax layer of DNN model. The probability can measure the distance of the sample to the decision border. Most neural network classifiers implement the softmax function at the last layer.
exp(4)
softmax(1); Σラ= cxp(()
The output of softmax is a probability mass function over the classes. The input to softmax is vector I. called logit. Let rank(L,,ibbe the index of the element that is ranked the ith largest among all the elements in L. given a normal example whose logit is I,, the goal of the attacker is to perturb the example to get new logit L2 such that rank(L, & rank(L2,11
Use f(x) to represent the output the last layer (softmax) of the neural network and x as the input of the neural network. For the categories that have been determined, the neural network has cor- responding output probability for each category. We believe that these probabilities represent the comparison between the model's features for the input and the model's 's learned classification fea- tures. The size of the probability output represents the shape of the input x in the low-dimensional manifold mapping and the distance from this type of feature in the low-dimensional manifold map- ping.
We use simple method,
dr p(maz) p( seconddaz))
to reflect the transformation of the distance of the picture manifold. Using only a single probability means that the effect is not significant for adversarial attacks, especially for adversarial samples that

---

Under review as a conference paper at ICLR 2022
have greatly crossed the boundary, their spatial mapping is almost the same that of legitimate samples. is the basic idea of the manifold distance detection method.
6 EXPERIMENT
In this section, we explain the experiment results for single method and the combination method after orchestration, and the comparison with Feature Squeezing.
6.1 RESULT- OF SINGLE METHOD
Table Single defense method in transfer attacks
Attacks
Math
JPEG Noise
Random Zoom
Swidl
Modian Hilter
MI-FGMM
0.91
0.33
0.61
0.21
0.76
Black box
DEFGSM
0.86
0.5
0.68
0.41
0.77
Grey box
MI-FGSM
0.74
0.67
0.74
0.3
DI-GGSM
0.65
0.44
0.47
0.67
0.86
Table 2: Single defense method in gradient optimization model attacks
Allacks
Math
JPEG Nosse
Random Zoom
Swirl
0.97
1.00
0.63
CW1.a
Next
0.35
0.89
0.95
0.37
0.94
1.00
0.99
0.72
CWL2
Nex
0.76
0.91
0.96
0,71
White box
0.78
1.00
1.00
0.74
CWL.;
Next
024
0.94
0.7
0.45
FGSM
0.75
0.49
0.90
0.30
DeepFool
0.52
0.23
BIM
0.01
0.48
0.79
0.09
Table and Table 2 shows the detection rates for successful adversarial examples for each attack method. It can be obtained from TABLE that the mainfold detection method performs better under black-box and gray-box attacks, and the ability to reach a detection model above 0.65 thresholds is weak. In contrast, in the white-box scenario, the attack algorithm generated powerful ability to counter the sample interference model, which also led to the failure the mainfold detection method in some attack methods in TABLE2. For instance, the detection rate of BIM attacks is only 0.01. The reason is that BIM, as a variant FGSM, has efficacious interference to the model, which makes it impossible to distinguish samples under the current threshold. Second, in CW attack, the attack only changes a limited number of pixels. Morcover, in CW attack, it only changes a limited number of pixels, the Next class the targeted attack has a greater impact the detection model, resulting in an increase in the detection rate of the least-likely class as the targeted attack. This is consistent with the observation result of the mainfold detection method in TABLE2, that is, the mainfold detection method can improve the accuracy of model detection to varying degrees.
The detection effect of the preprocessing method is the opposite of that of the math method, and it has higher detection rate in the white-box scene. As the attack algorithm learns more about the model. the more the features of the images can be changed by the preprocessing method, which in turn affects the judgment of the model. The observations in TABLEI and TABLE2 confirm our conclusion. Secondly, for different sample preprocessing methods, in white-bor box attacks, the detection rates of JPEG Noise and Random Zoom are significantly higher than those of Swirl. The reason is that these two processing methods can blur or enlarge the image as an integrated, and change the features more than the Swirl method. However, the JPEG Noise method has no significant effect on FGSM and BIM attacks. the gray-box scenario, the Swirl preprocessing method has a superior detection rate under MI-FGSM attacks. It illustrates that in different scenarios, each attack method has the most effective preprocessing way for it, and every preprocessing method can change the manifold characteristies of the image to certain extent.

---

Under review as a conference paper at ICLR 2022
6.2 RESULT-PERFORMANT OF THE COMBINATION METHOD
Table 3: Performance of the combination method in transfer attacks
Black box
Grey bux
Attacks
MI-FGSM
DI-FGSM
MI-FGSM
DI-FGSM
Comb
0.85
0.93
0.70
0.79
JPEG-Noisse
0.95
Comb
0.94
0.95
0.85
0.84
Comb
0.78
0.91
0.82
1.00
Random Zoom
Comb 2
0.88
1.00
0.96
0,86
Comb
0.97
1.00
0.96
0.86
Comb
0.88
0.95
0.86
Swirl
Comb
0.88
0.95
0.85
0.68
Comb
0.97
0.95
0.85
0.72
0.86
0.70
Median Hilter
Comb
0.94
1.00
0.93
1.00
Comb 1.00
1.00
0.93
1.00
Table 4: Performance of the combination method in gradient optimization model attacks
Attacks
CWL.
CWL2
CWL,
FGSM
DospPool
BIM
LL
Next
LL
Next
LL
Next
Comb
0.66
0.66
0.45
0.47
0.47
0.53
0.75
0.48
JPEG-Noise
Comh 2
1.00
0.94
1.00
1.00
1.00
0.97
0.77
0.93
0.71
Comb
1.00
0.97
1.00
1.00
1.00
0.99
0.78
0.71
Comb
0.57
0.55
0.45
0.55
0.46
0.46
0.58
0.51
0.57
Random Zoom
Comb 2
1.00
0.98
0.99
0.96
1.00
0.99
0.90
0.94
0.1
1.00
0.99
0.95
0.55
Comb
0,86
0.61
0.80
0.57
0.83
0.58
0.62
0.53
0.12
Swirl
Comb 0.50
0.59
0.95
0.83
0.93
0.72
0.61
0.50
0108
Comh
0.93
0.06
0.98
0.33
0.93
0.72
0.67
0.61
0.15
TABLE3 and TABLE4 show the detection result of using the orchestration detective strategy to successfully attack samples. Combl, Comb2, and Comb3 respectively represent the three kinds of Manifold Distance Judge's orchestration strategies.
Comb/ Combl is only to set up the threshold after performing the same image input transformation listed on the top of the table and use manifold detection method to make judgments. Comb2 Comb2 combines two single defense methods simply. In Comb2, we first performed image transformation on the samples and set up the threshold of manifold detection. After completing the image trans- former, we checked the labels of the images to find out the examples which labels have changed before and after the transformation, as well as the examples whose output of softmax is higher than the threshold for filtering, achieve the effect of classifying adversarial samples. Comb3 Comb3 firstly sets up a threshold, and perform manifold detection method on input samples. We filter sam- ples above the threshold, perform image transformation on samples below the threshold, judge the change of the sample label, and use manifold detection method to detect again.
For these three orchestration strategies, Combl is equivalent random combination method. Compared to Comb2, Comb3 not only has an extra level of detection process but also the two-level detection method uses the idea of orchestration.
The first column in TABLE3 and TABLE4 indicates different preprocessing methods. We can see that for each preprocessing method, the detection rates of Comb1, Comb2, and Comb3 are rising significantly. Meanwhile, in white-box attack scenario, reasonably orchestrated Comb3 defense strategy, such as the Random Zoom method used in the preprocessing stage, can achieve a detection rate of more than 95% for all attacks. In the black-box attack scenario, the Comb3 combined defense strategy uses methods, such as Random Zoom and Median Filter in the preprocessing stage, to achieve detection rate of 85% for all attacks. It further confirms conclusion that the Manifold Distance Judge method designed by applying orchestration can effectively improve the accuracy of the model.
8

---

Under review as a conference paper at ICLR 2022
In summary, Manifold Distance Judge is meaningful, because after the preprocessing transforma- tion, the distance between the legitimate sample and the adversarial sample on the manifold changes differently. In the orchestration defense strategy. compared the performance of Combl, Comb2, and Comb3, and found that an orchestration strategy that carefully detects changes in feature dis- tances, such as Comb3, obtain the best detection effect.
6.3 THE COMPARISON
We compare our results with Feature Squeezing Xu al. 2017) Table 5. Comb represent the Com3 in the previous Section4. We configured the Feature Squeez- ing detectors on ImageNet following the same experi- ment set-up d in Feature Squeezing paper and reported the detection performance with our target models and the detection dataset. It shows that our recall value on the same white box attack significantly higher than fea- ture squeezing, especially on CW attacks, can detect all pairs of samples, and the best classification rate of feature squeezing is only 0.87.
Table 5: Contrast with Feature Squeez-
ing
Recall
Attacks
Coubb
Squeezing
CWLo
LL
1,00
0.83
Next
1.00
0.85
LL
1.00
CWL2
Next
1.00
0,87
LL
1.00
0.82
CWL,
Next
0.99
0.77
FGSM
0.95
0.33
DeepFool BIM
0.97
0,72
The experimental effect proves that our orchestration defense strategy is better than other combined defense methods, which further verifies the contribution point 3.
0.95
0.41
CONCLUSION
From TABLEI TABLE4, we can evaluate the classification efficiency of the orchestrated com- bined defense model. The experimental results illustrated that the performance of single detection method was inferior or equivalent to that of an ensemble combination detection methods. How- ever, the performance of the orchestration detection method was far better than that, which shows that the defense performance of this method has been improved significantly. Meanwhile, there were different combinations in the orchestration detection methods. The different input transforma- tion methods will affect the final detection effect, and the choice of math method will also affect the final detection effect. From their perspective, mation chose the Median Filter, and the math method selected the difference method instead of the maximum value, which highlights the significance of the orchestration.
optimized the arrangement. The input transfor-
REFERENCES
Eddieet et al. Aamari. Estimating the reach of manifold. Electronic journal of statistics, 18: 1359-1399, 2019.
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of curity: Circumventing defenses to adversarial examples. In International conference on machine learning, pp. 274-283, PMLR, 2018a.
Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. Synthesizing robust adversarial examples. In International conference on machine learning, pp. 284-293. PMLR, 2018b.
Arjun Nitin Bhagoji, Daniel Cullina, Chawin Sitawarin, and Prateek Mittal. Enhancing robustness of machine learning systems via data transformations. In 2018 52nd Annual Conference on Infor- mation Sciences and Systems (CISS), pp. 1-5. IEEE, 2018.
Nicholas Carlini and David Wagner. Defensive distillation is not robust to adversarial examples. arXiv preprint arXiv: 0666000.111,1 2016.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 20/7 ieee symposium on security and privacy (sp), pp. 39-57. IEEE, 2017.
Yair Carmon, Aditi Raghumathan, Ludwig Schmidt, Percy Liang, and John C Duchi. Unlabeled data improves adversarial robustness. arXiv preprint arXiv: 13736, 2019.

---

Under review as a conference paper at ICLR 2022
Anirban Chakraborty, Manaar Alam, Vishal Dey, Anupam Chattopadhyay, and Debdeep Mukhopad- hyay. Adversarial attacks and defences: A survey, arXiv preprint 1rXiv: 101...00000.... 2018.
Ilya Sutskever Joan Bruna Dumitru Erhan lan J. Goodfellow Christian Szegedy. Wojciech Zaremba and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learn ing Representations (ICIR), 18: 18:1-10, 2014.
Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Yinpeng Dong, Fangzhou Liao, Tianyu Pang. Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boost- ing adversarial attacks with momentum. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9185-9193, 2018.
Charles Fefferman, Sanjoy Mitter, and Hariharan Narayanan. Testing the manifold hypothesis. Journal of the American Mathematical Society, 29(4):983-1049 2016.
Sergei Ivanov Matti Lassas Fefferman, Charles and Hariharan Narayanan. Fitting a manifold of large reach to noisy data. arXiv preprint 100.5549,,0 18:770- 2019.
Volker Fischer, Mummadi Chaithanya Kumar, Jan Hendrik Metzen, and Thomas Brox. Adversarial examples for semantic image segmentation. arXiv preprint arXiv: 100000/0300000 101, 2017.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1.
MIT Press, 2016.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens Van Der Maaten. Countering adversarial images using input transformations. arXiv preprint arXiv:1711.0011 2017.
Warren Hc, James Wei, Xinyun Chen, Nicholas Carlini, and Dawn Song. Adversarial example defense: Ensembles of weak defenses are not strong. In //th {USENIX) workshop on offensive technologies (1 (WOOT) 17), 2017.
Jonathon Shlens Ian J. Goodfellow and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations (ICLR), 18:770-778, 2015.
Shaoqing Ren Kaiming He, Xiangyu Zhang and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 18: 770-778, 778, 2016.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning scale. arXiv preprint 1rXiv:1611.01236 2016.
Alexey Kurakin, Ian Goodfellow. Samy Bengio, Yinpeng Dong, Fangzhou Liao, Ming Liang. Tianyu Pang, Jun Zhu, Xiaolin Hu, Cihang Xie, et al. Adversarial attacks and defences competition. In The NIPS'T 17 Competition: Building Intelligent Systems, pp. 195-231. Springer, 2018.
Dongyu Meng and Hao Chen. Magnet: two- pronged defense against adversarial examples. In ACM Conference on Computer and Communications Security (CCS), 18, 2017.
Seyed-Mohsen Moosavi-Dezfoolt Alhussein Fawzi, and Pascal Frossard. Deepfool: simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference on com- puter vision and pattern recognition, pp. 2574-2582. 2582, 2016.
Luis Gonizlzz,zzázz, Battista Biggio, Ambra Demontis, Andrea Paudice, Vasin Wongrassamee, Emil C Lupu, and Fabio Roli. Towards poisoning of deep learning algorithms with back crek-gradient optimization. Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pp. 27-3273 2017.
10

---

Under review as a conference paper at ICLR 2022
Nicolas Paperot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Ananthram Swami. The limitations deep learning in adversarial settings. 2016 IEEE European sympo- sium security and privacy (EuroS&P), pp. 372-387. IEEE, 2016a.
Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation defense to adversarial perturbations against deep neural networks. In 2016 IEEE symposium on security and privacy (SP), pp. 582-597. IEEE, 2016b.
Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation defense to adversarial perturbations against deep neural networks. In 2016 IEEE symposium on security and privacy (SP), pp. 582-597, IEEE, 2016c.
Edward Raff, Jared Sylvester, Steven Forsyth, and Mark McLean. Barrage of random transforms for adversarially robust defense. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6528-6537, 2019.
Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks neural networks. arXiv preprint 1804.00800000 00799, 2018
Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jianyu Wang, Zhou Ren, and Alan L. Yuille. Improving transferability of adversarial examples with input diversity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2730-2739, 2019.
Weilin Xu, David Evans, and Yanjun Qi. Feature squeezing: Detecting adversarial examples in deep
neural networks. arXiv preprint arXiv:1704.0113 01.00000115555 2017.
Kaiming He & Sun (2016) Christian Szegedy & Fergus (2014) Fefferman et al. (2016) Aamari
[ Feffermma Narayanan (2019) Ian Goodfellow & Szegedy (2013) Meng & Chen (2017)
Xu et al. (2017) (2017) & Wagner ( Munoz- et al. (201 Shaaahi et (2.18) Good-
fellow et al. (2014]]uuaki) Kurakin et al. (2018) MMosavvi-Dezfooli et al. (2016) Papernot et (2016a)
Bhagoji et al. (2018) Kurakin et al. (2016) Dong et (2018) Xie al. (2019) Carmon et al.
(2019)
Papernot et al. (2016b) Chakraborty et al. (2018) Papernot al. (2016c) Carlini & Wagner (016
He et al. (2017) )) Guo et al. (2017) Athalye al. (2018b) Raff et al. (2019) Athhaly et al.
2018a
Fischer et al. (2017) Goodfellow et al. (2016) Deng ct al. (2009)
A EXPERIMENT
A.1 EXPERIMENT SETUP
Threat model Depending on the attacker's knowledge of target model, there three scenarios:
Black-box attack:
Based on the alternative model, the attacker does not know the parameters of target model..
White-box attack:
Based on the gradient optimization model, the attacker knows the parameter of
target model.
Gray-box attack: Based on the gradient optimization model, except for the parameters, the attacker knows everything else about the target model.
Table 6: Success rate for each attack on ImageNet
Configration
Attack
L
La
Lo
FGSM
BIM
DeepFioll
Success Rate
0.99
0.99
0.90
0.97
1.00
1.00
0.99
1.00
0.89
We evaluate the manifold distance judge the attacks described and summarized in Table 6. For each targeted attack, we try two different targets: the Next class (t= L mod #classes), and the least-likely class (LL), == min (ÿ). Here is the target class, is the index of the ground-truth class and Φ is the prediction vector of an input image. This gives eleven total attacks: the four untargeted attacks (FGSM, BIM, MI-FGSM and DI-FGSM), and two versions each of the three targeted attacks
11

---

Under review as a conference paper at ICLR 2022
(JSMA, CW... co. CW2, and CW)). We use the implementations of FGSM, and BIM provided by the Cleverhans library. For MI-FGSM, DI-FGSM and the three CW attacks, we use the implementations from the original authors.
We chose different kinds of preprocessing methods and regarded the output of the Softmax layer of the model manifestation of the manifold distance. In the selection of preprocessing methods, we have selected several preprocessing methods that are completely different for image processing and compared them to verify the effectiveness of our method to the greatest extent.
Dataset and Target model We use the ImageNet dataset for the image classification task, and we set up pre-trained model with the state-of-the-ar performance. Meanwhile, we use the output of the last logits layer of the model as the input of softmax to obtain the required probability to represent the distance between manifolds.
In the ImageNet dataset, the target model of white-box attacks is set to the MobileNets model, because MobileNets are widely used on mobile phones and their small and efficient design makes it easier to conduct experiments. For black box and gray box attacks, the Inception model was chosen because has excellent classification accuracy and is widely used in the field of deep learning image classification.
Detection model We choose Inception V4 as the gray-box attack detection model, and the ResNet1 model as the black-box because gray-box attacks between similar structural models are achievable. Another reason is the Inception V4 and the Inception V3 model have similar model structures, while the 15 model is different. This model selection method can simulate gray box and black box attacks to the greatest extent.
Training As mentioned before, in the image detection of the deep learning model, there differences between the features of the normal image and the features of the adversarial sample. This is reflected in the probability output of the Softmax layer of the model, and the subsequent experimental part is represented by mathematic. Based on this feature, before each detection model starts detect, 100 images that can be correctly identified are randomly selected in ImageNet, and they are input into the detection model to select the appropriate anti-sample filter index to select the threshold.
Validation Next, we use the chosen threshold value to measure the detection rate on three groups: successful adversarial examples (SAEs), failed adversarial examples (FAEs), and legitimate exam- ples (for false positive rate). Except when noted explicitly, "detection rate" means the detection rate on successful adversarial examples. It is important to distinguish failed adversarial examples from legitimate examples here since detecting failed adversarial examples is useful for detecting attacks early, whereas an alarm on a legitimate example is always undesirable and is counted as false positive.
A.2 THE COMPARISON
Table 7 shows the detection effects of our different orchestration strategies successful attack sam- ples under the attack based on gradient optimization. Single represents the single method processing effect of the preprocessing method on the far right of the table. In Table 7, single represents a single method, and ensemble and comb represent an orchestration strategy. It can be seen from the differ- ence between the recall and IS indicators that the effect of the orchestration method is significant due to the single method. Compared with single defense method, in this experimental result, the recall and IS values of combination methods increase due to the change of the sample label of the attacking sample after the image transformation, indicating that more adversarial samples are detected.
Different orchestration strategies will produce different effects. On the one hand, for different de- tection methods, the effect of detecting the change of the characteristic distance of the sample is different. On the other hand, from the significance of the arrangement, whether the latter detection method can effectively use the conclusion of the previous level, that is, according the pretreatment method of the previous stage to select the method of the latter stage.
12

---

Under review as a conference paper at ICLR 2022
Table 7: Performance of different orchestration strategies gradient optimization model Attacks
Transformers
recall Comb2
Single
Comb2
Comb3
Single
Comb3
Single
Combz
Combb
CWLo
1.00
1,00
1,00
1.00
1.00
Next
0.94
0.86
0.81
0.95
0.98
1.00
0.90
0.95
1.00
0.93
0.83
0.81
0.99
0.99
1.00
0.98
0.97
1.00
CWL2
Next
Random Zoom
LL.
0.93
0.86
0.80
1.00
1.00
1.00
1.00
1.00
1.00
CWL,
0.99
0.97
FGSM
0.84
0.82
0.78
0.90
0.90
0.95
0.63
0.76
0.85
DeepHool
0.80
0.84
0.94
0.97
0.09
0.83
0.91
BIM
0.83
0.31
0.80
0.79
0.91
0395
0.61
0.78
U.86
CWL.o
LL. 0.80
0.80
0.80
0.63
0.90
0.93
0.44
0.75
0.81
0.19
CWL2
LT. 0.83
0.82
0.33
0.72
0.95
0.98
0.55
0.86
0.94
Next
0.81
0.71
0.93
0.53
0.66
0.81
Swirl
CWL,
LL.
0.83
0.81
0.82
0.74
0.93
0.95
0.56
0.81
0.86
Next
0.70
0.72
0.46
0.72
0.72
0.27
0.42
0.43
PGSM
DeepPuol
0.60
0.60
0.64
0.23
0.50
0.61
0.11
0.17
0.27
0.15
0.95
0.83
0.82
1.00
1.00
1.00
1.00
1.00
1.00
CWL.o
Next
0.90
0.81
0.81
0.39
0.94
0.97
0.38
0.84
0.91
CWL2
0.85
0.83
1.00
1.00
1.00
1.00
1.00
Next
0.91
0.83
0.81
0.91
1.00
1.00
83.00
1.00
1.00
Noise
0.82
1.00
CWL:
Next
0.93
0.83
0.82
0.94
0.97
0.99
0.88
0.92
0.97
FGSM
0.71
0.72
0.49
0.77
0.78
0.27
0.48
0.50
DeepPool
0.84
0.83
0.79
0.77
0.93
0395
0.60
0.82
0.85
BIM
0.69
0.70
0.87
0.48
0.71
0.71
0.26
0.40
0.37
13


Reviews:
[{"review_id": "nctP5GBePlb", "reviewer": "Reviewer_MGG8", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed method is based on the famous manifold hypothesis, that is the real-world high dimensional data generally lie in the lower dimensional manifold. \n\n1: I think the discussion in Section 3 is trivial and well-known in the deep learning community, hence it is unnecessary to spend many contexts to discuss this.\n\n2: May I ask about those experiments, how are the threat models defined? Against each adversarial attack, how is the attack defined? Could the authors provide some details on this?", "summary_of_the_paper": "This paper studies the manifold distance detection between adversarial examples and clean data. The proposed Manifold Distance Judge has shown effectiveness in detecting adversarial examples under various adversarial attack settings.", "summary_of_the_review": "Overall, I do not think the paper provides a significant contribution to the community. The proposed method is trivial and similar kinds of pre-processing methods have already been considered.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "lfyBLZv2yzW", "reviewer": "Reviewer_wmbn", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Strength]\n1.\tAt least the experiments were done with various attack & defense methods. However, as mentioned in the comments, a proper set of competitors should be other cutting-edge manifold defense methods and the authors should perform additional experiments against an adversary stronger than the vanilla implementation.\n\n[Weakness]\n1.\tThe paper contains severe writing issues such as grammatical errors, abuses of mathematical symbols, unclear sentences, etc. \n2.\tThe paper needs more literature survey, especially about the existing defense methods using the manifold assumption.\n3.\tThe paper does not have enough (either theoretical or experimental) progress to get accepted, compared to previous methods using the manifold assumption.\n\n[Comments]\n1.\tFirst of all, use some spell/grammar checker (or ask someone else to proofread) to fix basic grammatical errors. \n2.\tSection 3 is very unclear in general.\nFirst, I cannot understand the reason why the Section is needed at all. Manifold-based defense against adversarial example is not a new approach and reviewers know well about the manifold assumption in the adversarial machine learning setting. Section 3 does not introduce anything new more than those reviewers’ understanding, and the reasonings are too crude to be called an “analysis”.\nSecond, the writing is not cohesive enough. Each paragraph is saying some topic, however, the connections between the paragraphs are not very clear, making Section 3 more confusing. Even in a single paragraph, the logical reasonings between sentences are sometimes not provided at all.\nThird, some of those contents are added for no reason. For example, Figure 1 exists for no reason whereas the figure is not referred at all in the paper. The propositions mentioned in Section 3.3 are vaguely written and not used at all. By having these unnecessary parts, the writing looks to be verbose and overstating.\n3.\tIn Section 5, the defense method should be written with more formality. Based on the description given in the paper “dx = p(max)-p(secondmax)”, it is very unclear what each term means. Each probability (the authors did not even say that they are probabilities) must correspond to the output from a softmax layer, but which model provides such a softmax layer output, the target classifier, or is there another classifier prepared for it? How are the described transformations used to get the divergence value? What does the detector do with the divergence? (All of these details should be described in Section 5.) Section 6.2 mentions some thresholding strategies, how did the detector work in Section 6.1, though? When thresholding is used, what is the threshold value used and what is the rationale of the choice of the threshold value? There are so many missing details to understand the method.\n4.\tSection 7 looks to be a conclusion for experiments. This should be moved to Section 6 and Section 7 should be an overall conclusion of the paper.\n5.\tThe suggested method is neither creative nor novel, compared to the existing methods utilizing the distance from manifolds. As pointed out, the defense based on the manifold assumption is not a new approach. [1][3][4][6](These papers are only a few representative examples. There are many other papers on this type of defense.) Moreover, the idea of using probability divergence is already proposed by previous work [1] and an effective attack for such detection already exists. [2] (Of course, this paper proposes another probability divergence, but there is no support that this method could be significantly better than the previous work.)\n6.\tThe experiment should be done more extensively. It looks like that some transformations were brought from the Raff et al. paper [5] which tested the defense against the adversary as strong as possible. Specifically, Raff et al. considered potential improvements of existing attacks to attack their work then tested the defense performance against the improved attack. However, the paper only uses vanilla implementation in the Cleverhans library (or by the original authors). The authors should have shown that the proposed method is robust against a stronger adversary because adversaries who are aware of the method will not use a simple version of the attack. (At least, those adversaries will try using the attack suggested by Raff et al.)\n\n[References]\n\n[1] (Meng & Chen) MagNet: a Two-Pronged Defense Against Adversarial Examples\n\n[2] (Carlini & Wagner) MagNet and “Efficient Defenses Against Adversarial Attacks” are Not Robust to Adversarial Examples\n\n[3] (Samangouei et al.) Defense-GAN: Protecting Classifiers Against Adversarial Attacks Using Generative Models\n\n[4] (Jiang et al.) To Trust or Not to Trust a Classifier\n\n[5] (Raff et al.) Barrage of Random Transforms for Adversarially Robust Defense\n\n[6] (Dubey et al.) Defense Against Adversarial Images using Web-Scale Nearest-Neighbor Search", "summary_of_the_paper": "Based on the manifold assumption in the adversarial machine learning setting, this paper proposes a detection strategy using a set of different image transformations and the classifier’s output probability divergence. Some experiments were performed to show the detection performance against vanilla implementations of well-known attack methods. ", "summary_of_the_review": "I believe that the paper quality is far below the acceptance threshold. First, this paper does not contain any significant progress in the manifold-based defense method. Second, the average quality of paper writing is extremely poor to understand the paper properly.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ffq5hjGwAr1", "reviewer": "Reviewer_Cbtd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes a manifold distance based detection based against adversarial samples. In addition, it also proposes manifold distance judge for the adversarial defense. The focus of the paper is to make low-dimensional manifolds better express high-dimensional data; the experimental results seems that the proposed method has good performance. However, I have the following concerns:\n1.\tThe paper introduces too much intuitive idea, but it lacks technical steps;\n2.\tTechnical contributions should be better analyzed. Otherwise, it is hardly to find the contributions of the paper;\n3.\tThe basic idea seems interesting, but the manifold distance method has already be utilized in many extant works;\n4.\tWhy the authors only select Random zoom, Swirl, Median filter, and JPEG noise as the preprocessing methods? It is difficult to find the relation between the proposed detection method with the preprocessing methods;\n5.\tIn the experiment, comparing with Feature squeezing seems inadequate. In addition, the experimental results should be better organized;\n6.\tThere are some typos and grammar errors, such as “Table 1 and Table 2 show”, etc.\n", "summary_of_the_paper": "This paper proposes a manifold distance based detection based against adversarial samples. In addition, it also proposes manifold distance judge for the adversarial defense. The focus of the paper is to make low-dimensional manifolds better express high-dimensional data; the experimental results seems that the proposed method has good performance, but there are still lots of space to improve.", "summary_of_the_review": "This paper proposes some novel idea to defend adversarial attacks, but the paper should be better organized to present the technical contributions.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "-PiU1Y9O31L", "reviewer": "Reviewer_fqVb", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n- The paper reports high detection rates of adversarial examples generated using a variety of attacks in the black-, grey- and white-box settings.\n**Weaknesses**\n- The quality of writing and explanation in this paper is rather poor, to the point that it actively impedes the reader's ability to understand the methods being proposed and the experiments that are conducted. My summary of the paper is an educated guess based on what I could parse from the language, but I may have misunderstood due to issues with the writing and presentation. I encourage the authors to improve the writing and presentation, specifically by giving a summary of the detection method used in the form of a algorithm block in the text.\n- From the description of the white-box attacks in the paper, it would appear that the proposed detection method has not been validated with adaptive attacks. An indication of this comes from the detection performance against the Basic Iterative Method in Table 2. I suspect that with a more careful evaluation, the proposed detection method would not be successful. The fact that detection methods are not effective at dealing with adaptive attacks has been known for some time (see *Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods* by Carlini and Wagner). This paper could be drastically improved with a detailed analysis of an adaptive attack against the proposed method.\n- The paper only considers a single source of information (the so-called manifold) to determine if an example is adversarial or not. It does not consider the use of embedding layers within the network etc. as other possible sources of information for detection.\n- Crucial experimental details such as the parametrization of the attacks used are missing. Further, in some cases such as for the FGSM attack, the detection rate is lower for grey-box attacks than for more powerful white-box attacks, which is incorrect and points to experimental error.", "summary_of_the_paper": "This paper proposes using the difference between the highest and second-highest softmax outputs from a model to detect adversarial examples. It also considers the use of different preprocessing methods to modify the output probabilities such that the adversarial examples can be well-differentiated from benign ones.", "summary_of_the_review": "This paper suffers from poor quality of writing, an unclear contribution to the already vast literature on the detection of adversarial examples and limited engagement with well-known recommendations from the research community on the evaluation of defenses/detection methods for adversarial examples.\n++++++++++++++++++++++\nNo response so I retain my score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a manifold distance-based detection based against adversarial samples, i.e., using the difference between the highest and second-highest softmax outputs from a model to detect adversarial examples. All the reviewers gave negative scores. The main concerns lie in 1) poor quality of writing; 2) contributions of the paper are not clearly stated; and 3) limited engagement with well-known recommendations from the research community on the evaluation of defenses/detection methods for adversarial examples. No rebuttals are provided. Thus, I cannot recommend accepting the paper to ICLR.
Decision: Reject