Title: On Detecting Adversarial Perturbations
Authors: Jan Hendrik Metzen
Tim Genewein
Volker Fischer
Bastian Bischoff
Venue: ICLR 2017
Keywords: Computer vision
Deep learning
Supervised Learning

Abstract: Machine learning and  deep learning in particular has advanced tremendously on perceptual tasks in recent years. However, it remains vulnerable against adversarial perturbations of the input that have been crafted specifically to fool the system while being quasi-imperceptible to a human. In this work, we propose to augment deep neural networks with a small ``detector'' subnetwork which is trained on the binary classification task of distinguishing genuine data from data containing adversarial perturbations. Our method is orthogonal to prior work on addressing adversarial perturbations, which has mostly focused on making the classification network itself more robust.  We show empirically that adversarial perturbations can be detected surprisingly well even though they are quasi-imperceptible to humans. Moreover, while the detectors have been trained to detect only a specific adversary, they generalize to similar and weaker adversaries. In addition, we propose an adversarial attack that fools both the classifier and the detector and a novel training procedure for the detector that counteracts this attack. 

TL;DR: We present and evaluate an approach for detecting adversarial perturbations in images based on attaching a small subnetwork to a deep neural network that is trained specifically to detect adversarial perturbations.

Review discussion:
One application of an adversary-detector would be 'flag' potentially pathological images for an ML system.
Although this work demonstrated the utility of an adversary-detector for flagging such images, would it not be possible for the adversary to be constructed such that it is an adversary to *both* the original classification network as well as the adversary-detector? If so, then the gains of an adversary-detector would be minimal.
Yes, constructing such an adversary is possible;
in fact, that is exactly what we denote by "dynamic adversary". We present an algorithm for one such adversary in Section 3.3.
However, the adversary-detector can also be trained in a way such that it becomes more robust to such a dynamic adversary (see also Section 3.3). See Section 4.1.2 for an evaluation of the robustness against a dynamic adversary and Section 5 for a discussion.
We would also like to emphasize that while constructing a dynamic adversary against an adversary detector is possible, it requires access to the detector's predictions and gradients. This is a stronger assumption than granting the adversary access to the original classifier's predictions and gradients since the classifier's predictions need often be presented to a user (and thus also to an adversary). The same is typically not true for the predictions of the adversary detector as they will only be used internally.
Hi,
I am wondering how the test accuracy of the detector is calculated. For each sample in the test set, both the original image and its adversarial version are fed to the detector, or each original image has a probability of 0.5 being perturbed?
What is the range of the input data, [0,1] or [0,255]? I think it is [0, 255]. In Figure 3, what is the predictive accuracy when using a perturbation of magnitude {1,2,3,4}? wound't this magnitude be too small?
Hi!
The test accuracy of the detector is calculated as follows: for every test sample, its adversarial version is computed. The original and the corresponding adversarial examples form a joint test set (twice the size of the original test set). This test set is shuffled and the detector is evaluated on this dataset - original and corresponding adversarial example are thus processed independently.
The range of the input data is [0, 255].
The predictive accuracy for perturbations of magnitude {1, 2, 3, 4} correspond to those in Figure 2. The four points in the plot correspond to these four values, with the smallest value for epsilon corresponding to the highest predictive accuracy. The specific values for the predictive accuracy can be found below. These magnitudes for epsilon are not too small: for instance, for "Iterative (l_infty)" with epsilon=3 we obtain an accuracy of ~0.001.
Moreover, larger values of epsilon would make the task only more simple for the detector.
The actual predictive accuracies are the following:
Adversary: Fast
epsilon:1 accuracy:0.441
epsilon:2 accuracy:0.260
epsilon:3 accuracy:0.195
epsilon:4 accuracy:0.166
Adversary: Iterative (l_2)
epsilon:20 accuracy:0.512
epsilon:40 accuracy:0.129
epsilon:60 accuracy:0.027
epsilon:80 accuracy:0.007
Adversary: Iterative (l_infty)
epsilon:1 accuracy:0.264
epsilon:2 accuracy:0.018
epsilon:3 accuracy:0.001
epsilon:4 accuracy:0.000
Please note that epsilon \in {1, 2, 3, 4} is in l_infinity distance, while epsilon \in {20, 40, 60, 80} is in l_2 distance.
Is the convolution sizes of Figure 1 correct? It has two resnet sequences with 8x8 output. Is 16x16 meant there?
Yes, you are right; the output of the second "Res" block is supposed to be 16x16. Thanks for noting this! We will correct this issue in the next version.
I reviewed the manuscript on December 5th.
Summary:
The authors investigate the phenomenon of adversarial perturbations and ask whether one may build a system to independently detect an adversarial data point -- if one could detect an adversarial example, then might prevent a machine from automatically processing it.
Importantly, the authors investigate whether it is possible to build an adversarial detector which is resilient to adversarial examples built against *both* the classifier and the detector. Their results suggest that training a detector in this more difficult setting still yields gains but does not entirely resolve the problem of detecting adversarial examples.
Major comments:
The authors describe a novel approach for dealing with adversarial examples from a security standpoint -- namely, build an independent system to detect the adversary so a human might intervene in those cases.
A potential confound of this approach is that an adversary might respond by constructing adversarial examples to fool *both* the original classifier and the new detector. If that were possible, then this approach is moot since an attacker could always outwit the original system. To their credit, the authors show that building a 'dynamic' detector to detect adversarial examples but also be resilient to an adversary mitigates this potential escalation (worse case from 55% to 70% detection rate).
Even though the 'dynamic' detector  demonstrates positive gains, I am concerned about overall scores. Detecting adversarial examples at this rate would not be a reliable security procedure.
My second comment is about 'model transferability'. My definition of 'model transferability' is different then the one used in the paper. My definition means that one constructs an adversarial example on one network and measures how well the adversarial examples attack a second trained model -- where the second model has been trained with different initial conditions. (The author's definition of 'transferability' is based on seeing how well the detector generalizes across training methods). 'Model transferability' (per my definition) is quite important because it measures how general an adversarial example is across all models -- and not specific to a given trained model. Different methods have different levels of 'model transferability' (Kurakin et al, 2016) and I am concerned how well the detector they built would be able to detect adversarial examples across *all models* and not just the trained model in question.
In other words, a good detector would be able to detect adversarial examples from any network and not just one particularly trained network. This question seems largely unaddressed in this paper but perhaps I missed some subtle point in their descriptions.
Minor comments:
If there were any points in the bottom-left of the Figure 2 left, then this would be very important to see -- perhaps move the legend to highlight if the area contains no points.
- X-axis label is wrong in Figure 2 right.
Measure the transferability of the detector?
- How is \sigma labeled on Figure 5?
- Whenever an image is constructed to be an 'adversary', has the image actually been tested to see if it is adversarial?
In other words, does the adversarial image actually result in a misclassification by the original network?
We would like to thank the reviewer for his detailed comments. We would also like to respond to the raised concerns:
Regarding 'transferability' of the adversary: we believe that detectors need a different kind of transferability than adversaries. Adversaries need to generalize across models since an adversary will typically not know the exact model it is trying to fool. Detectors, on the other hand, would be trained by the same entity that would also train the model and thus the specific model would be known.
Thus, a detector need not necessarily be model-agnostic (it would nevertheless be interesting to see if training such a detector would be possible). A detector does *not* know, however, what kind of adversary it will face.
Thus, it is important that a detector works equally well for many adversaries.
Because of this setting, we consider transferability in the case of detectors to be a transferability across adversaries (and not models).
Accordingly, this is also what we investigated in the experiments.
Regarding the detectability of dynamic adversaries: we acknowledge that there is room for future improvement and 70% detectability in the worst case is not satisfying.
However, we would also like to emphasize (see also our other comment) that this worst case is actually relatively unlikely since it requires that the adversary has access to the detector's predictions and gradients. This is a stronger assumption than granting the adversary access to the original classifier's predictions and gradients since the classifier's predictions often need to be presented to a user (and, thus, also to an adversary). The same is typically not true for the predictions of the adversary detector as they will only be used internally.
Regarding the minor comments: we have addressed those in the revised version of the paper.
Regarding "has the image actually been tested to see if it is adversarial": yes, this is exactly the "predictive accuracy on adversarial images" in Figure 2 and Figure 6. As this accuracy is larger than 0, there are some images where the adversaries fail to fool the classifier.
Nevertheless, we included those images when computing the detectability as the images contain adversarial perturbations (albeit not sufficient to fool the classifier). Detectability restricted to adversarial examples which actually fool the classifier would probably be larger than the detectability reported in the Figure 2 and Figure 6.
This paper explores an important angle to adversarial examples: the detection of adversarial images and their utilization for trainig more robust networks.
This takes the competition between adversaries and models to a new level. The paper presents appealing evidence for the feasibility of robustifying networks by employing the a detector subnetwork that is trained particularly for the purpose of detecting the adversaries in a terget manner rather than just making the networks themselves robust to adversarial examples.
The jointly trained primary/detector system is evaluated in various scenarios including the cases when the adversary generator has access to the model and those where they are generated in a generic way.
The results of the paper show good improvements with the approach and present well motived thorough analyses to back the main message. The writing is clear and concise.
This paper proposes a new idea to help defending adversarial examples by training a complementary classifier to detect them. The results of the paper show that adversarial examples in fact can be easily detected.
Moreover, such detector generalizes well to other similar or weaker adversarial examples. The idea of this paper is simple but non-trivial.
While no final scheme is proposed in the paper how this idea can help in building defensive systems, it actually provides a potential new direction. Based on its novelty, I suggest an acceptance.
My main concern of this paper is about its completeness. No effective method is reported in the paper to defend the dynamic adversaries. It could be difficult to do so, but rather the paper doesn’t seem to put much effort to investigate this part. How difficult it is to defend the dynamic adversaries is an important and interesting question following the conclusions of this paper. Such investigation may essentially help improve our understanding of adversarial examples.
That being said, the novelty of this paper is still significant.
Minor comment:
The paper needs to improve its clarity. Some important details are skipped in the paper.
For example, the paper should provide more details about the dynamic adversaries and the dynamic adversary training method.
Thanks for your review! We have provided more details on dynamic adversaries and the dynamic adversary training method  in the revised paper and believe that all required details are contained. If you find that some further details are missing, could you please indicate what is missing? Thanks again!
We have uploaded a new revision of the paper in which we have tried to address the reviewer comments. Here is a more detailed changelog:
* Fixed a bug in the ImageNet experiment: we originally applied the softmax operator twice (once before and once after selecting the ten target classes). This did not affect the accuracy of the classification network but made the network harder to fool by adversaries for similar reasons as in the ``defensive distillation'' approach. We have corrected the issue in the updated version of the paper by applying softmax only after selection the ten target classes. To briefly summarize the corrected results: adversaries remain detectable with an accuracy of at least 85% (with the same exception as before, the basic iterative l2-based adversary for epsilon=400). More details are contained in the updated Section 4.2. Sorry for this error in the first revision.
* Fixed wrong resolution in Figure 1 (16x16 instead of 8x8). Thanks to AnonReviewer3 for noting this.
* Input range specified to be [0, 255] (Section 4.1.1). Thanks to AnonReviewer1 for requesting clarification on this.
* Clarified computation of adversarial detectability (footnote in Section 4.1.1).
* We discuss briefly that dynamic adversaries are based on stronger assumptions than static adversaries (footnote in Section 3.3)
* Clarified that we did use version 1 of DeepFool (Section 3.1)
* Fixed x-axis label in Figure 2 (right). Thanks to AnonReviewer2 for noting this.
* Moved legend in Figure 2 (left) to upper right corner based on suggestion of AnonReviewer2.
* Clarified choices of \sigma in Figure 5
* Adding more details about the dynamic adversary training method.

Meta-review:
role: PC
data: comment: The paper explores the automatic detection of adversarial examples by training a classifier to recognize them. This is an interesting direction, even though they are obviously concerns about training an adversary to circumvent this model. Nonetheless, the experimental results presented in the paper are of interest to the ICLR audience. Many of the initial reviewer comments appear to be appropriately addressed in the revision of the paper.

Decision: Accept (Poster)