Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: Have any of these models been trained with label smoothing? To what degree have you observed if label smoothing or other forms of noise (e.g. dropout) might mitigate the transferability of adversarial examples?

review_title: Mitigating transferability with label smoothing

final_score: rating: null
review_content: How can we reconcile the conclusions of Table 5 (that targeted adversarial example don’t transfer well) with the conclusions of Table 10 (that targeted adversarial examples do in fact transfer well to Clarifai)? Maybe it's because in Table 10 the evaluation is done by hand and is perhaps more lenient to fine-grained misclassifications whereas in Table 5 it is done automatically? (Not sure, just throwing out ideas.) Would using either a hierarchical evaluation metric or manually examining a small number of images in Table 2 perhaps help reconcile these?

Also, for the non-targeted adversarial examples submitted to Clarifai, are the errors that Clarifai makes similar to the errors that your ensemble network makes?

Very minor and unrelated: Why is threshold \tau the same for parts 2 and 3 of the equation on pg. 6?
review_title: Reconciling conclusions of Table 5 and 10

final_score: rating: null
review_content: Could the authors discuss the recent and closely related work "Robustness of classifiers: from adversarial to random noise" (NIPS'16)? Another work by the same author was cited in the proposed paper, but this one seems more relevant.
review_title: commenting a related work

final_score: rating: 7: Good paper, accept
review_content: This paper present an experimental study of the robustness of state-of-the-art CNNs to different types of "attacks" in the context of image classication. Specifically, an attack aims to fool the classification system with a specially corrupted image, i.e. making it misclassify the image as (1) any wrong class (non-targeted attack) or (2) a target class, chosen in advance by the attacker (targeted attack). For instance, the attacker could corrupt an image of an ostrich in such a way that it would be classified as a megalith. Even though the attacker's agenda is not so clear in this example, it is still interesting to study the weaknesses of current systems in view of (1) improving them in general and (2) actual risks with e.g. autonomous vehicles.

The paper is mostly experimental. In short, it compares different strategies (already published in previous papers) for all popular networks  (VGG, GoogLeNet, ResNet-50/101/152) and the two aforementionned types of attacks. The experiments are well conducted and clearly exposed. A convincing point is that attacks are also conducted on "clarifai.com" which is a black-box classification system. Some analysis and insightful explanations are also provided to help understanding why CNNs are prone to such attacks (Section 6).

To sum up, the main findings are that non-targeted attacks are easy to perform, even on a black-box system. Non-targeted attacks are more difficult to realize with existing schemes, but the authors propose a new approach for that that vastly improves over existing attacks (even though it's still far from perfect: ~20% success rate on clarifai.com versus 2% with previous schemes).

Arguably, The paper still has some weaknesses:

 - The authors are treating the 3 ResNet-based networks as different, yet they are obviously clearly correlated. See Table 7 for instance. This is naturally expected because their architecture is similar (only their depth varies). Hence, it does not sound very fair to state that "One interesting finding is that [...] the first misclassified label (non-targeted) is the same for all models except VGG-16 and GoogLeNet.", i.e., the three ResNet-based networks.

 - A subjective measure is employed to evaluate the effectiveness of the attacks on the black box system. While this is for a good reason (clarifai.com returns image labels that are different from ImageNet), it is not certain that the reported numbers are fair (even though the qualitative results look convincing).

 - The novelty of the proposed approach (optimizing an ensemble of network instead of a single network) is limited. However, this was not really the point of the paper, and it is effective, so it seems ok overall.

 - The paper is quite long. This is expected because it is an extensive evaluation study, but still. I suggest the authors prune some near-duplicate content (e.g. Section 2.3 has a high overlap with Section 1, etc.).

 - The paper would benefit from additional discussions with the recent and related work of Fawzi et al (NIPS'16) in Section 6. Indeed the work of Fawzi et al. is mostly theoretical and well aligned with the experimental findings and observations (in particular in Section 6).


To conclude, I think that this paper is somewhat useful for the community and could help to further improve existing architectures, as well as better assess their flaws and weaknesses.
review_title: interesting and insightful work on adversarial examples for deep CNNs for image classification

final_score: rating: 5: Marginally below acceptance threshold
review_content: I reviewed the manuscript as of December 7th.

Summary:
The authors investigate the transferability of adversarial examples in deep networks. The authors confirm that transferability exists even in large models but demonstrate that it is difficult to manipulate the network to adversarially perturb an image into a specifically desired label. The authors additionally demonstrate real world attacks on a vision web service and explore the geometric properties of adversarial examples.

Major Comments:
1. The paper contains a list of many results and it is not clear what single message this paper provides. As mentioned in the comments, this paper is effectively 15 pages and 9 page of results in the Appendix heavily discussed throughout the main body of the paper. Although there is no strict page limit for this conference, I do feel this pushes the spirit of a conference publication. I do not rule out this paper for acceptance based on the length but I do hold it as a negative because clarity of presentation is an important quality. If this paper is ultimately accepted, I would suggest that the authors make some effort to cut down the length even further beyond the 13 pages posted elsewhere. I have marked some sections to highlight areas that may be trimmed.

2. The section of geometric understanding is similar to results of 'Adversarial Perturbations of Deep Neural Networks' in Warde-Farley and Goodfellow (2015). See Figure 1.2. I am not clear what the authors show above-and-beyond these results. If there are additional findings, the authors should emphasize them.

3. The authors expand on observations by Goodfellow et al (2014) and Szegedy et al (2013) demonstrating that large-scale models are susceptible to adversarial perturbations (see also Kurakin et al (2016)). The authors additionally demonstrate that attempting to perform adversarial manipulation to convert an image to a particular, desired label is more difficult.

4. The authors demonstrate that they can target a real-world vision API. These results are compelling but it is not clear what these results demonstrate above-and-beyond Papernot et al (2016). 

As far I can understand, I think that the most interesting result from this paper not previously described in the literature is to note about the unique difficulty about performing adversarial manipulation to convert an image to a particular, desired label. The rest of the results appear to expand on other results that have already appeared in the literature and the authors need to better explain what these makes these results unique above-and-beyond previous work.

Areas to Trim the Paper:
- Table 1 is not necessary. Just cite other results or write the Top-1 numbers in the text.
- Condense Section 2.2.1 and cite heavily.
- Figure 2 panels may be overlaid to highlight a comparison.

review_title: Review for Liu et al

final_score: rating: null
review_content: In equation (7), why did you use f(x) rather than J(x)? Because f(x) = argmax J(x), f(x) is highly non-linear, so the gradient descent might be much more inefficient. Even tough optimizing with J rather than f is not exactly the same objective, the outcome may be very similar. Did you try this, or di I misunderstood something?
review_title: technical detail

final_score: rating: null
review_content: I am not sure if the authors have revised the paper since the first submission but the manuscript is now 16-24 pages long depending on what sections are artificially labeled as the Appendix.

Although there is no strict page limit for this conference, this seems notably longer then the suggested limit of 8 pages mentioned here:

http://www.iclr.cc/doku.php?id=iclr2017:callforpapers

As mentioned in the web page above:

> The appropriateness of using additional pages over the recommended length will be judged by reviewers. 

Given that most other NIPS and ICLR papers abide by this length constraint, I am skeptical that this paper warrants so many pages. If there is indeed so much material, then this should be 2+ papers. That said, I would have preferred to see the authors make some effort to condense their material accordingly. I suspect that it is too late for a large request on my part at this time but I would like to hear the authors justification for the paper length before I begin my formal review.

review_title: Paper length is too long

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper presents an interesting and very detailed study of targeted and non-targeted adversarial examples in CNNs.

I’m on the fence about this paper but am leaning towards acceptance. Such detailed empirical explorations are difficult and time-consuming to construct yet can serve as important stepping stones for future work. I see the length of the paper as a strength since it allows for a very in-depth look into the effectiveness and transferability of different kinds of adversarial examples.  

There are, however, some concerns:

1) While the length of the paper is a strength in my mind, the key contributions should be made much more clear. As evidenced by my comment earlier, I got confused at some point between the ensemble/non-ensemble method, and about the contribution of the Clarifai evaluation and what I should be focusing on where. I’d strongly suggest a radical revision which more clearly focuses the story: 

- First, we demonstrate that non-targeted attacks are easy while targeted attacks are hard (evidenced by a key experiment comparing the two; we refer to appendix or later sections for the extensive exploration of e.g., current Section 3)

- Thus, we propose an ensemble method that is able to handle targeted attacks much better (evidenced by experiments focusing on the comparison between ensemble and non-ensemble method, both in a controlled setting and on Clarifai)

- Also, here are all the other details and explorations. 

2) Instead of using ResNet-152, Res-Net-101 and ResNet-50 as three of the five models, it would've been better to use one ResNet architecture and the other two, say, AlexNet and Network-in-Network. This would make the ensemble results a lot more compelling.
review_title: good in-depth exploration but strongly recommend a rewrite

Meta-review: The paper is the first to demonstrate that it is possible for an adversary to change the label that a convolutional network predicts for an image to a specific value. Like Papernot et al., it presents a successful attack on Clarifai's image-recognition system. I encourage the authors to condense the paper to its key results (13 pages without / 24 pages with supplemental material is too long for a conference paper).
Decision: Accept (Poster)