Title: Delving into Transferable Adversarial Examples and Black-box Attacks
Authors: Yanpei Liu
Xinyun Chen
Chang Liu
Dawn Song
Venue: ICLR 2017
Keywords: Computer vision
Deep learning
Applications

Abstract: An intriguing property of deep neural networks is the existence of adversarial examples, which can transfer among different architectures. These transferable adversarial examples may severely hinder deep neural network-based applications. Previous works mostly study the transferability using small scale datasets. In this work, we are the first to conduct an extensive study of the transferability over large models and a large scale dataset, and we are also the first to study the transferability of targeted adversarial examples with their target labels. We study both non-targeted and targeted adversarial examples, and show that while transferable non-targeted adversarial examples are easy to find, targeted adversarial examples generated using existing approaches almost never transfer with their target labels. Therefore, we propose novel ensemble-based approaches to generating transferable adversarial examples. Using such approaches, we observe a large proportion of targeted adversarial examples that are able to transfer with their target labels for the first time. We also present some geometric studies to help understanding the transferable adversarial examples. Finally, we show that the adversarial examples generated using ensemble-based approaches can successfully attack Clarifai.com, which is a black-box image classification system.


Review discussion:
Have any of these models been trained with label smoothing? To what degree have you observed if label smoothing or other forms of noise (e.g. dropout) might mitigate the transferability of adversarial examples?
The model is retrieved from online. We believed it is not trained with label smoothing, but with some forms of regularizations such as dropout. In general, the main point of this work is to show that transferability is still an issue even when some regularizations are employed during the training. We will leave exploring other forms of regularization to future work.
How can we reconcile the conclusions of Table 5 (that targeted adversarial example don’t transfer well) with the conclusions of Table 10 (that targeted adversarial examples do in fact transfer well to Clarifai)? Maybe it's because in Table 10 the evaluation is done by hand and is perhaps more lenient to fine-grained misclassifications whereas in Table 5 it is done automatically? (Not sure, just throwing out ideas.) Would using either a hierarchical evaluation metric or manually examining a small number of images in Table 2 perhaps help reconcile these?
Also, for the non-targeted adversarial examples submitted to Clarifai, are the errors that Clarifai makes similar to the errors that your ensemble network makes?
Very minor and unrelated: Why is threshold \tau the same for parts 2 and 3 of the equation on pg. 6?
In this paper, we show two observations: (1) using existing approaches, targeted adversarial examples DO NOT transfer (which was Table 5 in our original submission, now it is Table 6); and (2) using our new ensemble-based approach, targeted examples DO transfer (now Table 7). The clarifai.com examples in Table 11 (originally Table 10) are generated using our ensembled-based approach, as described in the paper:
“our experiment shows that for targeted adversarial examples, 18% of those generated using the ensemble model can be predicted as labels close to the target label by Clarifai.com” (page 14).
It is explicit: images in Table 11 are the ones in Table 7 rather than in Table 6. We will update the column caption in Table 11 to make this point clear.
Therefore, there is no conflict between Table 6 and Table 11.
For the non-targeted adversarial examples, it is interesting to see whether our models make the same mistakes as Clarifai, but it would take time to manually inspect the results. We will spend the time to see if there are any interesting findings.
For \tau, yes, the two occurrences of \tau refer to the same threshold: we want the distortion to be at least \tau (i.e., minimizing ReLU(\tau - d)), but not to be too much greater than \tau: (i.e., minimizing ReLU(d-\tau)).
Therefore, the final distortion optimizing this objective should have a distortion very close to \tau.
I am not sure if the authors have revised the paper since the first submission but the manuscript is now 16-24 pages long depending on what sections are artificially labeled as the Appendix.
Although there is no strict page limit for this conference, this seems notably longer then the suggested limit of 8 pages mentioned here:
http://www.iclr.cc/doku.php?id=iclr2017:callforpapers
As mentioned in the web page above:
> The appropriateness of using additional pages over the recommended length will be judged by reviewers.
Given that most other NIPS and ICLR papers abide by this length constraint, I am skeptical that this paper warrants so many pages. If there is indeed so much material, then this should be 2+ papers.
That said, I would have preferred to see the authors make some effort to condense their material accordingly. I suspect that it is too late for a large request on my part at this time but I would like to hear the authors justification for the paper length before I begin my formal review.
We apologize for not marking the appendix well, and we have made the changes.
Excluding the references and the appendix, the paper is 14.5 pages long right now.
We believe this paper is addressing one important problem, and thus cannot be divided into two papers.
In fact, we try to provide an extensive analysis of the transferability phenomenon.  To this end, we include evaluations for not only existing popular methods to generate adversarial examples, but also our new proposal. The latter indeed demonstrates an intriguing property that is not shown by prior methods --- targeted transferability. We believe that such an extensive evaluation is necessary to have a full understanding of the phenomenon. This is why this paper becomes both long and dense.
Having said this, however, we can indeed move some interesting but secondary findings into the appendix to shrink the main body down to around 12-13 pages.
However, as we have discussed above, we believe most of the current materials are necessary to be presented to the readers.
Also, although the appropriateness of using additional pages is judged by reviewers, we do not think a judgment solely based on the number of pages without reviewing the content is fair to us or is consistent with the spirit of ICLR.
To sum up, we are happy to move secondary results to the appendix if the number of pages is indeed an issue.
But we have received pre-review questions from other reviewers. To avoid confusion, we would like to wait for the full reviews from all reviewers first before updating the paper, in case some reviewers may think some results to be important though we may think they are secondary.
However, we do upload a version with its main body being 12.5 at:
http://people.eecs.berkeley.edu/~liuchang/tmp/transferability_iclr_2017.pdf, if you find it’s helpful for your review.
Could the authors discuss the recent and closely related work "Robustness of classifiers: from adversarial to random noise" (NIPS'16)?
Another work by the same author was cited in the proposed paper, but this one seems more relevant.
This NIPS 2016 paper presents some important theoretical results towards understanding the minimal adversarial perturbation.
However, we do not find immediate application of the theorems to explain the transferability of adversarial examples among different models, which is our paper’s main focus.
In fact, in this NIPS 16 paper, the discussion is totally based on one single model, and it is not obvious how the derived bounds on the minimal adversarial perturbation can be transferred to another model.
But we would like to discuss about these theoretical results in the paper.
In equation (7), why did you use f(x) rather than J(x)? Because f(x) = argmax J(x), f(x) is highly non-linear, so the gradient descent might be much more inefficient. Even tough optimizing with J rather than f is not exactly the same objective, the outcome may be very similar. Did you try this, or di I misunderstood something?
This was a presentation issue. We used f to represent the label output in the first part of the paper, but the softmax output in Section 5. We have fixed this issue, and the paper is updated. We indeed optimized for J rather than f. Thanks for pointing out!
I reviewed the manuscript as of December 7th.
Summary:
The authors investigate the transferability of adversarial examples in deep networks. The authors confirm that transferability exists even in large models but demonstrate that it is difficult to manipulate the network to adversarially perturb an image into a specifically desired label. The authors additionally demonstrate real world attacks on a vision web service and explore the geometric properties of adversarial examples.
Major Comments:
1. The paper contains a list of many results and it is not clear what single message this paper provides. As mentioned in the comments, this paper is effectively 15 pages and 9 page of results in the Appendix heavily discussed throughout the main body of the paper.
Although there is no strict page limit for this conference, I do feel this pushes the spirit of a conference publication. I do not rule out this paper for acceptance based on the length but I do hold it as a negative because clarity of presentation is an important quality. If this paper is ultimately accepted, I would suggest that the authors make some effort to cut down the length even further beyond the 13 pages posted elsewhere. I have marked some sections to highlight areas that may be trimmed.
2. The section of geometric understanding is similar to results of 'Adversarial Perturbations of Deep Neural Networks' in Warde-Farley and Goodfellow (2015). See Figure 1.2. I am not clear what the authors show above-and-beyond these results. If there are additional findings, the authors should emphasize them.
3. The authors expand on observations by Goodfellow et al (2014) and Szegedy et al (2013) demonstrating that large-scale models are susceptible to adversarial perturbations (see also Kurakin et al (2016)). The authors additionally demonstrate that attempting to perform adversarial manipulation to convert an image to a particular, desired label is more difficult.
4. The authors demonstrate that they can target a real-world vision API. These results are compelling but it is not clear what these results demonstrate above-and-beyond Papernot et al (2016).
As far I can understand, I think that the most interesting result from this paper not previously described in the literature is to note about the unique difficulty about performing adversarial manipulation to convert an image to a particular, desired label. The rest of the results appear to expand on other results that have already appeared in the literature and the authors need to better explain what these makes these results unique above-and-beyond previous work.
Areas to Trim the Paper:
- Table 1 is not necessary. Just cite other results or write the Top-1 numbers in the text.
- Condense Section 2.2.1 and cite heavily.
- Figure 2 panels may be overlaid to highlight a comparison.
We thank the reviewer for the suggestions to shrink the paper. Before we give a full response, we would like to ask a quick clarification question. The reviewer mentioned the paper, 'Adversarial Perturbations of Deep Neural Networks' in Warde-Farley and Goodfellow (2015), and commented that our geometric understanding section is similar to this paper.
However, we tried our best, but could not find this paper.
Can we ask the reviewer to provide the full citation of this referred paper, so that we can adjust our paper accordingly?
Figure 1.2 - Figure 1.4
http://www-etud.iro.umontreal.ca/~wardefar/publications/adv_perturb_chapter.pdf
1. We are happy to incorporate the suggestions on trimming the paper, but we also notice that other reviewers indeed consider the length of this paper as a strength. We will work hard to maximize the information-noise ratio.
2. We asked for the full citation since we are unclear how to cite this work properly. A url alone is not enough to resolve this issue.
So we still hope the reviewer can provide the full citation, so that we can cite it properly. To our understanding, this paper is from the Perturbations, Optimization, and Statistics workshop accompanying NIPS 2016.
Thus we do not think that the appearance of this (likely workshop) article hinders the novelty of our paper.
Despite this issue, we have carefully studied the article mentioned in the review. Fig 1.2-1.4 are all drawn using models trained on CIFAR-10 dataset, which has only 10 labels. Models showed throughout our paper (including Section 6) are trained over ImageNet, which contains 1,000 classes. We want to emphasize that we do not claim the methodology to conduct the geometric analysis as our contributions, but the new findings in Section 6 are. Examining ImageNet models is the main point of our work, and it allows us to make unique findings that are not presented in any other work (including the one mentioned in the review):
(1) Gradient directions of different models in our examination are orthogonal to each other (see page 11 Table 9)
(2) Although there are 1000 classes, on a 2-D plot, there are at most 21 regions (see page 13 Table 10 for results, page 12 for text)
(3) The width of ground truth region is thinner than its height. (see page 13 Figure 5, page 13 for text)
(4) Other novel findings are presented in Section 6, and we do not want to flood this response by repeating all of them one by one.
To sum up, we believe that most of Section 6 are novel findings. We can also add a statement somewhere in Section 6 to emphasize that all these results are not presented in prior works.
3. We are not quite following reviewer’s comments here. We agree that Szegedy et al (2013) and Goodfellow et al (2014) made the observation that ImageNet models also have adversarial examples, but our paper is studying the transferability among ImageNet models, which we do not see before, except a concurrent work by Moosavi-Dezfooli et al (2016) (cited in our submission). We presume that the reviewer refers to Kurakin et al (2016) as the following paper:
Alexey Kurakin, Ian Goodfellow, Samy Bengio, Adversarial examples in the physical world, arxiv: 1607.02533
However, this paper is also not studying the transferability.
To sum up, we believe that we have made it very clear (from title, to abstract, to introduction, and throughout the paper) that the focus of this paper is on transferability of large models. We would appreciate the reviewer to explain better about this point and how we can improve.
4. Papernot et al (2016) uses Amazon’s and Google’s services as black-box model providers to perform the attack, but both the training data set and the label set is provided by the authors.
Therefore, in their paper, only the model and the training process is a black-box to the adversary.
In contrast, we attack clarifai.com that we know nothing about: we do not know the training data, the model, and the training process; even the label set is unknown to us, and our observation shows that it is different from that of ImageNet.
Furthermore, we not only perform non-targeted attacks, but also targeted attacks. All of these show that we are actually solving a much harder problem than that in Papernot et al (2016). We can make this point clearer in our paper.
This paper present an experimental study of the robustness of state-of-the-art CNNs to different types of "attacks" in the context of image classication.
Specifically, an attack aims to fool the classification system with a specially corrupted image, i.e. making it misclassify the image as (1) any wrong class (non-targeted attack) or (2) a target class, chosen in advance by the attacker (targeted attack).
For instance, the attacker could corrupt an image of an ostrich in such a way that it would be classified as a megalith.
Even though the attacker's agenda is not so clear in this example, it is still interesting to study the weaknesses of current systems in view of (1) improving them in general and (2) actual risks with e.g. autonomous vehicles.
The paper is mostly experimental.
In short, it compares different strategies (already published in previous papers) for all popular networks  (VGG, GoogLeNet, ResNet-50/101/152) and the two aforementionned types of attacks. The experiments are well conducted and clearly exposed. A convincing point is that attacks are also conducted on "clarifai.com" which is a black-box classification system. Some analysis and insightful explanations are also provided to help understanding why CNNs are prone to such attacks (Section 6).
To sum up, the main findings are that non-targeted attacks are easy to perform, even on a black-box system. Non-targeted attacks are more difficult to realize with existing schemes, but the authors propose a new approach for that that vastly improves over existing attacks (even though it's still far from perfect: ~20% success rate on clarifai.com versus 2% with previous schemes).
Arguably, The paper still has some weaknesses:
- The authors are treating the 3 ResNet-based networks as different, yet they are obviously clearly correlated. See Table 7 for instance. This is naturally expected because their architecture is similar (only their depth varies).
Hence, it does not sound very fair to state that "One interesting finding is that [...] the first misclassified label (non-targeted) is the same for all models except VGG-16 and GoogLeNet.", i.e., the three ResNet-based networks.
- A subjective measure is employed to evaluate the effectiveness of the attacks on the black box system.
While this is for a good reason (clarifai.com returns image labels that are different from ImageNet), it is not certain that the reported numbers are fair (even though the qualitative results look convincing).
- The novelty of the proposed approach (optimizing an ensemble of network instead of a single network) is limited.
However, this was not really the point of the paper, and it is effective, so it seems ok overall.
- The paper is quite long. This is expected because it is an extensive evaluation study, but still. I suggest the authors prune some near-duplicate content (e.g. Section 2.3 has a high overlap with Section 1, etc.).
- The paper would benefit from additional discussions with the recent and related work of Fawzi et al (NIPS'16) in Section 6.
Indeed the work of Fawzi et al. is mostly theoretical and well aligned with the experimental findings and observations (in particular in Section 6).
To conclude, I think that this paper is somewhat useful for the community and could help to further improve existing architectures, as well as better assess their flaws and weaknesses.
Specifically, we will revise the observations for the decision boundary in Section 6; do our best to remove redundant information; and cite Fawzi et al (2016) properly in Section 6.
The paper presents an interesting and very detailed study of targeted and non-targeted adversarial examples in CNNs.
I’m on the fence about this paper but am leaning towards acceptance. Such detailed empirical explorations are difficult and time-consuming to construct yet can serve as important stepping stones for future work. I see the length of the paper as a strength since it allows for a very in-depth look into the effectiveness and transferability of different kinds of adversarial examples.
There are, however, some concerns:
1) While the length of the paper is a strength in my mind, the key contributions should be made much more clear. As evidenced by my comment earlier, I got confused at some point between the ensemble/non-ensemble method, and about the contribution of the Clarifai evaluation and what I should be focusing on where. I’d strongly suggest a radical revision which more clearly focuses the story:
- First, we demonstrate that non-targeted attacks are easy while targeted attacks are hard (evidenced by a key experiment comparing the two; we refer to appendix or later sections for the extensive exploration of e.g., current Section 3)
- Thus, we propose an ensemble method that is able to handle targeted attacks much better (evidenced by experiments focusing on the comparison between ensemble and non-ensemble method, both in a controlled setting and on Clarifai)
- Also, here are all the other details and explorations.
2) Instead of using ResNet-152, Res-Net-101 and ResNet-50 as three of the five models, it would've been better to use one ResNet architecture and the other two, say, AlexNet and Network-in-Network. This would make the ensemble results a lot more compelling.
We thank the reviewer for the suggestions.
In fact, we have added a contribution section to highlight the most important take-aways. We will rework the story part of the paper as suggested to make the messages come across.
We choose to report the results for three ResNet models, because these results show the transferability between models using the same architecture but with different hyper-parameters. As an in-depth study, we believe both same-architecture transferability and cross-architecture transferability are meaningful to investigate.
In fact, we indeed make some interesting unexpected and expected observations:
1) using single-model approaches, transferability between ResNet models is not significantly better than other models.
For example, VGG-16’s transferability to ResNet-50 is better than both ResNet-101’s and ResNet-152’s (Table 2);
2) at the same time, we found that ResNet models are more frequently making the same mistakes than others (Table 3).
There are more findings, and we will highlight them in the next revision.
For AlexNet and NIN, we had technical issues, i.e., we cannot choose 100 images from the testset that all models make correct predictions for them. We can mitigate this issue by omitting the model with the lowest test accuracy, i.e., AlexNet, choosing a smaller test set, or choosing test images not only from the ILSVRC validation dataset but also from its training dataset.
However, any of these choices will require a re-run of all experiments. We will work on this, but given the expected time to be consumed, we are afraid that we may not have time to update the paper to reflect these results before Jan 19, 2017.
We have updated the paper with the following changes:
1) We shrink the paper in the following aspects:
a) Sec 2.2 is condensed
b) We remove the original Sec 2.3, which introduces transferability and black-box attack, since all materials have been covered in Sec 1
c) The original Table 1 is moved to the appendix Table 7
d) The original Table 2 and Table 4 are now merged as two panels of Table 1, so that only one caption needs be provided.
e) The alternative approach to generate non-targeted adversarial images is moved to the appendix
f) The paragraph discussing different models make the same mistake in original Sec 3.1 is moved to the appendix
g) The paragraph discussing that adversarial images may come from multiple intervals along the gradient direction is moved to the appendix
h) The results for random perturbation in original Sec 3.2 is moved to the appendix.
i) We move the original Table 9 in Section 6 about cosine values between pairs of gradients to the appendix Table 33
2) We provide a contribution and organization paragraph in Section 1 to highlight the the main conclusions of this work and to facilitate readers to follow the arguments in the paper to support our conclusions.
3) In related work, we highlight the difference between our work and Papernot et al (2016ab) by explaining why our black-box attack is harder.
Also, we discuss Fawzi et al (2016) at the beginning of Section 6.
4) We provide justification of why we choose three ResNet models in Section 2.3, and highlight the findings about the transferability between both homogeneous architectures (i.e., ResNet models) and heterogeneous architectures in various places (i.e., Section 3.1, Section 5 and Section 6).
We welcome new comments!

Meta-review:
role: PC
data: comment: The paper is the first to demonstrate that it is possible for an adversary to change the label that a convolutional network predicts for an image to a specific value. Like Papernot et al., it presents a successful attack on Clarifai's image-recognition system. I encourage the authors to condense the paper to its key results (13 pages without / 24 pages with supplemental material is too long for a conference paper).

Decision: Accept (Poster)