Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: Hi,

I am wondering how the test accuracy of the detector is calculated. For each sample in the test set, both the original image and its adversarial version are fed to the detector, or each original image has a probability of 0.5 being perturbed? 

What is the range of the input data, [0,1] or [0,255]? I think it is [0, 255]. In Figure 3, what is the predictive accuracy when using a perturbation of magnitude {1,2,3,4}? wound't this magnitude be too small?
review_title: The test accuracy of the detector, and a problme about the magnitude of perturbation

final_score: rating: null
review_content: One application of an adversary-detector would be 'flag' potentially pathological images for an ML system. Although this work demonstrated the utility of an adversary-detector for flagging such images, would it not be possible for the adversary to be constructed such that it is an adversary to *both* the original classification network as well as the adversary-detector? If so, then the gains of an adversary-detector would be minimal.
review_title: Adversary to the Adversary-Detector

final_score: rating: 7: Good paper, accept
review_content: This paper explores an important angle to adversarial examples: the detection of adversarial images and their utilization for trainig more robust networks.

This takes the competition between adversaries and models to a new level. The paper presents appealing evidence for the feasibility of robustifying networks by employing the a detector subnetwork that is trained particularly for the purpose of detecting the adversaries in a terget manner rather than just making the networks themselves robust to adversarial examples.

The jointly trained primary/detector system is evaluated in various scenarios including the cases when the adversary generator has access to the model and those where they are generated in a generic way.

The results of the paper show good improvements with the approach and present well motived thorough analyses to back the main message. The writing is clear and concise.

review_title: Nicely written experimental paper, making the next step in the adversarial contest.

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a new idea to help defending adversarial examples by training a complementary classifier to detect them. The results of the paper show that adversarial examples in fact can be easily detected. Moreover, such detector generalizes well to other similar or weaker adversarial examples. The idea of this paper is simple but non-trivial. While no final scheme is proposed in the paper how this idea can help in building defensive systems, it actually provides a potential new direction. Based on its novelty, I suggest an acceptance.

My main concern of this paper is about its completeness. No effective method is reported in the paper to defend the dynamic adversaries. It could be difficult to do so, but rather the paper doesn’t seem to put much effort to investigate this part. How difficult it is to defend the dynamic adversaries is an important and interesting question following the conclusions of this paper. Such investigation may essentially help improve our understanding of adversarial examples.
That being said, the novelty of this paper is still significant.

Minor comment:
The paper needs to improve its clarity. Some important details are skipped in the paper. For example, the paper should provide more details about the dynamic adversaries and the dynamic adversary training method. 


review_title: Good paper with significant novelty

final_score: rating: null
review_content: Is the convolution sizes of Figure 1 correct? It has two resnet sequences with 8x8 output. Is 16x16 meant there?
review_title: Question on network architecture

final_score: rating: 5: Marginally below acceptance threshold
review_content: I reviewed the manuscript on December 5th.

Summary:
The authors investigate the phenomenon of adversarial perturbations and ask whether one may build a system to independently detect an adversarial data point -- if one could detect an adversarial example, then might prevent a machine from automatically processing it. Importantly, the authors investigate whether it is possible to build an adversarial detector which is resilient to adversarial examples built against *both* the classifier and the detector. Their results suggest that training a detector in this more difficult setting still yields gains but does not entirely resolve the problem of detecting adversarial examples.

Major comments:

The authors describe a novel approach for dealing with adversarial examples from a security standpoint -- namely, build an independent system to detect the adversary so a human might intervene in those cases. 

A potential confound of this approach is that an adversary might respond by constructing adversarial examples to fool *both* the original classifier and the new detector. If that were possible, then this approach is moot since an attacker could always outwit the original system. To their credit, the authors show that building a 'dynamic' detector to detect adversarial examples but also be resilient to an adversary mitigates this potential escalation (worse case from 55% to 70% detection rate). Even though the 'dynamic' detector  demonstrates positive gains, I am concerned about overall scores. Detecting adversarial examples at this rate would not be a reliable security procedure.

My second comment is about 'model transferability'. My definition of 'model transferability' is different then the one used in the paper. My definition means that one constructs an adversarial example on one network and measures how well the adversarial examples attack a second trained model -- where the second model has been trained with different initial conditions. (The author's definition of 'transferability' is based on seeing how well the detector generalizes across training methods). 'Model transferability' (per my definition) is quite important because it measures how general an adversarial example is across all models -- and not specific to a given trained model. Different methods have different levels of 'model transferability' (Kurakin et al, 2016) and I am concerned how well the detector they built would be able to detect adversarial examples across *all models* and not just the trained model in question. In other words, a good detector would be able to detect adversarial examples from any network and not just one particularly trained network. This question seems largely unaddressed in this paper but perhaps I missed some subtle point in their descriptions.

Minor comments:

If there were any points in the bottom-left of the Figure 2 left, then this would be very important to see -- perhaps move the legend to highlight if the area contains no points.

- X-axis label is wrong in Figure 2 right.

Measure the transferability of the detector?

- How is \sigma labeled on Figure 5?

- Whenever an image is constructed to be an 'adversary', has the image actually been tested to see if it is adversarial? In other words, does the adversarial image actually result in a misclassification by the original network?
review_title: Official review

Meta-review: The paper explores the automatic detection of adversarial examples by training a classifier to recognize them. This is an interesting direction, even though they are obviously concerns about training an adversary to circumvent this model. Nonetheless, the experimental results presented in the paper are of interest to the ICLR audience. Many of the initial reviewer comments appear to be appropriately addressed in the revision of the paper.
Decision: Accept (Poster)