Title: Evaluation of Defensive Methods for DNNs against Multiple Adversarial Evasion Models
Authors: Xinyun Chen
Bo Li
Yevgeniy Vorobeychik
Venue: ICLR 2017
Keywords: Deep learning

Abstract: Due to deep cascades of nonlinear units, deep neural networks (DNNs) can automatically learn non-local generalization priors from data and have achieved high performance in various applications.
However, such properties have also opened a door for adversaries to generate the so-called adversarial examples to fool DNNs. Specifically, adversaries can inject small perturbations to the input data and therefore decrease the performance of deep neural networks significantly.
Even worse, these adversarial examples have the transferability to attack a black-box model based on finite queries without knowledge of the target model. 
Therefore, we aim to empirically compare different defensive strategies against various adversary models and analyze the cross-model efficiency for these robust learners. We conclude that the adversarial retraining framework also has the transferability, which can defend adversarial examples without requiring prior knowledge of the adversary models.
We compare the general adversarial retraining framework with the state-of-the-art robust deep neural networks, such as distillation, autoencoder stacked with classifier (AEC), and our improved version, IAEC, to evaluate their robustness as well as the vulnerability in terms of the distortion required to mislead the learner.
Our experimental results show that the adversarial retraining framework can defend most of the adversarial examples notably and consistently without adding additional
vulnerabilities or performance penalty to the original model.

TL;DR: robust adversarial retraining

Review discussion:
Adversarial examples for one network may be adversarial examples for different networks (trained with different initial conditions or even different network architectures). Have you examined how each defense method performs when an adversarial example is calculated on a different model then where it is evaluated on?
Thank you for your question.
Indeed, we have performed precisely the analysis you suggest; the results are shown in Tables 1 & 2 for the MNIST and CIFAR-10 datasets, respectively. In these tables, each column corresponds to a model used to generate adversarial examples, and each row represents a specific method for defending against attacks.
Thus, we evaluate all defensive approaches against all attack methods, with each cell in the table reporting the performance of the “defense” algorithm. The "no adversary" column in these tables shows the original test data without being contaminated for comparison.
Hi,
I am curious if you have checked the perturbation magnitude of the attacker using cg or adam.
For example, in the result of table 3 and table 4, while fgs fails, does it take cg and adam a huge perturbation to change the prediction label? Are the generated adversarial examples still imperceptible to human?
Best,
Thank you for your question.
Precisely, we have performed the analysis you are interested in. In Table 5 and 6 we show how much perturbation is needed for cg and adam to evade different defensive methods (rows) on MNIST and CIFAR-10, respectively. Here we normalize the values of pixels to [-0.5,0.5]. The magnitude of noise added for cg and adam are actually slightly smaller than fgs_0.5 (distortion=0.013 for MNIST, 0.0075 for CIFAR-10) but still succeed to attack as shown in Table 3 and 4.
The generated adversarial examples are not imperceptible after being attacked by cg and adam in this case, which is shown in Figure 3, 4 (d) & (e) for MNIST and CIFAR-10.
In Table 3, how is the size of the perturbation for cg and adam in the additional attack, compared to the size in the training of the robustified network? Sorry for the previous vague question.
Also, all pixel values for all the experiments are in the range of [-0.5, 0.5]?
Thank you for your question!
In Table 5, the perturbations for LeNet-5 is the noise needed in the training of the original robust network. The perturbations for different defensive algorithms from row 2-7 are the perturbations of cg and adam in the additional attack.
Yes, all pixel values for all the experiments are in the range of [-0.5, 0.5].
What is n in 1/n \sqrt(\sum ...) at the end of the introduction of Section 3?
Why is n outside the square root rather then inside?
Thank you for your question!
n is the the number of pixels within one image. We are trying to calculating the distortion per pixel per channel.
So inside the root squared sign we calculate the sum of distortions for all the pixels and then divide it by n to get the average distortion.
I reviewed the manuscript as of December 6th.
The authors perform a systematic investigation of various retraining methods for making a classification network robust to adversarial examples. The authors achieve lower error rates using their RAD and IAEC methods perform better then previously introduced distillation methods for retraining networks to be robust to adversarial examples. This method suggests a promising direction for building a defense for adversarial examples.
Major Comments:
I find the paper to not be lacking in exposition and clarity. The paper has a laundry list of related results (page 2) but no clear message. I *think* one larger point is the superior performance of their retraining techniques but it is not clear how well these techniques perform compared to other retraining techniques, nor are the details of the retraining techniques clear. The paper requires more discussion and a clear exposition about the methods the authors introduced (i.e. RAD, IAEC). What follow are some more detailed comments along this theme of improving the exposition and clarity:
- The authors should provide more details about how they constructed the auto-encoder in the IAEC method (diagram?). The same needs to be said for the RAD method. The authors point to a previous workshop submission (https://arxiv.org/abs/1604.02606) but the authors need more discussion about what this method entails and how it compares to other retraining methods (e.g. [1,2]) since this is the first peer-review of this work.
- Section 3.2 indicates that the authors are concerned with 'cross-model' efficiency but it is not clear from the text what this means. Are the author exploring the phenomenon of retraining off one algorithm and then evaluating adversarial images derived on another? Or, are the authors examining how examples derived from one instance of a trained model may fool or trick a second instance of a model? The latter point is quite important because this points towards examples and retraining procedures that can generalize across the class of all models.
- How do RAD compare with basic retraining methods described in [1, 2]? Since the main contribution of this paper seems to be evaluating the efficacy of RAD, AEC and IAEC, I would suggest that the authors provide more discussion and exposition.
- Why are the authors measuring 'recall' (https://en.wikipedia.org/wiki/Precision_and_recall) in Section 3.1? What does recall mean in this context? I would expect the authors to measure something more like the error rate of the classifier after employing the retraining procedure. This needs to be clarified in the manuscript.
[1] https://arxiv.org/abs/1412.6572
[2] https://arxiv.org/abs/1611.01236
Thank you for your review and suggestions!
We will add more details about the experimental settings.
The “cross-model” here means the first point as you mentioned, i.e., add instances crafted using one adversarial example generation algorithm and then evaluate adversarial images derived by another.
Actually, we aim to evaluate the ability of defender here to check if the defender can defend against various adversarial models if he applies a different algorithm to generate the retraining instances and improve the classifier. This provides information about how retraining framework can generalize across different adversarial models.
However, the second point mentioned by the reviewer is from the attacker’s perspective and hope to check if the attacker can generalize their attacks against different defensive models. This is one interesting point but not the scope of this paper here, and in fact such work on transferability of attacks have been discussed in Introduction as reference [Papernot et al., 2016a;b].
We will also add the discussion of work [1]. We didn’t discuss work [2] when we wrote the paper, since it is another submission for this conference at the same time.
Also, work [2] aims to solve the scalability issue of adversarial training, which is not the main topic of our paper. We aim to test the efficacy of different adversarial example generation algorithms here, but we are happy to discuss this work in our revision.
The “recall” here means the true positive rate. Since we focus on the binary classification problem and only allow the malicious (positive) instances to evade the classifier as benign (negative) ones, we want to check how many of these modified instances are correctly classified after retraining.
Meanwhile, we have reported the classification error for all the experimental comparisons as suggested, the classification errors corresponding to experimental settings of Figure 1 and Figure 2 are reported in Table 1 and Table 2 respectively.
The paper compares several defense mechanisms against adversarial attacks: retraining, two kinds of autoencoders and distillation with the conclusion that the retraining methodology proposed by Li et al. works best of those approaches.
The paper documents a series of experiments on making models robust against adversarial examples. The methods proposed here are not all too original, RAD was proposed by Li et al, distillation was proposed in Goodfellow et al's "Explaining and harnessing adversarial examples", stacked autoencoders were proposed by Szegedy et al's "Intriguing Properties of Neural Networks". The most original part of the paper is the improved version of autoencoders proposed in this paper.
The paper establishes experimental evidence that the RAD framework provides the best defense mechanism against adversarial attacks which makes the introduction of the improved autoencoder mechanism less appealing.
Although the paper establishes interesting measurement points and therefore it has the potential for being cited as a reference, its relative lack of originality decreases its significance.
Thank you for your review and suggestions!
The “Adversarial retraining framework” has been proposed but no work has been done to evaluate its efficiency in general or provide insights about its robustness and weaknesses.
Therefore, to fill up this gap, we propose this comparison work to offer a general reference for how and when to use adversarial retraining framework and what to expect.
Our contributions here are trying to evaluate the retraining framework on adversarial examples generated using different attack methods. The evaluation of cross-model efficiency (Section 3.2) and additional attacks (Section 3.3) are also our contributions. We will revise our paper to make our points clearer.
This paper performs a series of experiments to systematically evaluate the robustness of several defense methods, including RAD, AEC and its improved version etc.. It provides interesting observations.
Overall, RAD and distillation have the best performances, but none of the methods can really resist the 'additional' attack from cg or adam. Since it is an experimental paper, my main concern is about its clarity. See the comments below for details.
Pros:
1. This paper provides a good comparison of the performances for the selected methods.
2. Section 3.3 (the 'additional' attack) is a interesting investigation.
Although the final result about the defense methods is negative, its results are still inspiring.
3.
Overall, this paper provides interesting and inspiring experimental results about the selected methods.
Cons:
1. There are several other methods in the literature that are missing from the paper.
For example the defense methods and the attack methods in the papers [1,2].
2.
Although a long list of experimental results are provided in the paper, many details are skipped.
For example, details of the experiments that generate the results in Table 5.
3. Without further explanations and analyses about the experimental results, the contribution of the paper seems limited.
4. This paper proposed an improved version of the AEC algorithm.
But its experimental results seems not promising.
Minor comments:
Page 3: Equation (3) is also non-convex.
So the non-convexity of Equation (2) should not be the motivation of Equation (3).
[1] https://arxiv.org/abs/1507.00677
[2] https://arxiv.org/abs/1511.03034
Thank you for the review and suggestions!
As for attack and defense methods evaluated in our paper, we are not intended to cover all proposed methods in our experiments;
instead, we select several representative methods. We will add more discussion about different proposed methods and provide more comparisons in our revision.
In particular, as for the references you mentioned, Miyato et al. ([1]) is already mentioned in paragraph 2 of the Introduction in our paper, and we will add the discussion of Huang et al. ([2]).
We will provide more experimental details for the results.
Meanwhile, we will polish the paper to provide further analysis of experimental results and highlight our contributions.
The improved version of AEC actually performs much better than AEC (see Table 1). Our goal of proposing IAEC is to provide the strongest version of this strategy for later comparisons with other defense methods.
We will address the minor comment in our revision.
We thank all reviewers and readers for the helpful comments! We have updated the paper with the following changes:
1) Provide more details about the experimental settings in section 4.1.
2) Add a section "Related Work" (section 2) and move the detailed discussion of related works in Introduction to this section, so that the Introduction focuses more on our contributions.
3) Update the Introduction section to highlight the differences between our work and previous works, and emphasize our contributions.
Also, we provide more explanation of the term "cross-model" in Introduction section (at the top of page 2) to avoid confusion.
4) Add the discussion of following related works: Huang et al. (https://arxiv.org/abs/1511.03034), Kurakin et al. (https://arxiv.org/abs/1611.01236), and adversarial training proposed in Goodfellow et al. (https://arxiv.org/abs/1412.6572).
New comments are welcome!

Meta-review:
role: PC
data: comment: The paper investigates several retraining approached based upon adversarial data. While the experimental evaluation looks reasonable, the actual contribution of this paper is quite small. The approaches being evaluated, for the most part, are already proposed in the literature, with the one exception being the "improved autoencoder stacked with classifier" (IAEC), which is really just a minor modification to the existing AEC approach with an additional regularization term. The results are fairly thorough, and seem to suggest that the IAEC method performs best in some cases, but this is definitely not a novel enough contribution to warrant publication at ICLR.
 
 Pros:
 + Nice empirical evaluation of several adversarial retraining methods
 
 Cons:
 - Extremely minor algorithmic advances
 - Not clear what is the significant contribution of the paper

Decision: Reject