Title: Adversarial examples in the physical world
Authors: Alexey Kurakin
Ian J. Goodfellow
Samy Bengio
Venue: ICLR 2017
Keywords: Supervised Learning
Computer vision

Abstract: Most existing machine learning classifiers are highly vulnerable to adversarial examples.
An adversarial example is a sample of input data which has been modified
very slightly in a way that is intended to cause a machine learning classifier
to misclassify it.
In many cases, these modifications can be so subtle that a human observer does
not even notice the modification at all, yet the classifier still makes a mistake.
Adversarial examples pose security concerns
because they could be used to perform an attack on machine learning systems, even if the adversary has no
access to the underlying model.
Up to now, all previous work has assumed a threat model in which the adversary can
feed data directly into the machine learning classifier.
This is not always the case for systems operating in the physical world,
for example those which are using signals from cameras and other sensors as input.
This paper shows that even in such physical world scenarios, machine learning systems are vulnerable
to adversarial examples.
We demonstrate this by feeding adversarial images obtained from a cell-phone camera
to an ImageNet Inception classifier and measuring the classification accuracy of the system.
We find that a large fraction of adversarial examples are classified incorrectly
even when perceived through the camera.


Review discussion:
Based on my understanding, no matter how large \epsilon is, the method proposed in section 2.3 can not guarantee that the adversarial sample will be misclassified as y_{LL}. Actually, it can't even guarantee that the image will be misclassified, no matter how large \epsilon is. See the discussion in Proposition 2 of the paper of Huang et al. [1] I am wondering if the target label actually appears in your experiments.
Also, is this effect taken into account in the experiment settings?
On the other hand, if the motivation of the LL method is to find a strong adversarial method (as shown in Figure 2), the alpha method in [1] may be a better one.
Also, what is the actual perturbation of the basic iterative method in the results of Figure 2?
When \epsilon is large, the actual perturbation may be much smaller than \epsilon.
[1] Huang, R., Xu, B., Schuurmans, D., & Szepesvári, C. (2015). Learning with a strong adversary. CoRR, abs/1511.03034.
Yes, you are correct that this method is not guaranteed to cause a
misclassification. We do not claim that it is guaranteed to cause a
misclassification.
Indeed, we report the accuracy of the model on such
adversarial examples.
Main motivations of Iter L.L. method were following.
First of all, to find strong adversarial method which would produce non-trivial misclassification, as opposed to fast method which tend to produce less interesting misclassification (like one breed of dog confused with another breed of dog).
Second, to keep computational cost manageable.
Iter L.L. method achieved both of those goals.
In addition we observed that for large enough \epsilon this method actually was able to force class label to become least likely class.
In particular for \epsilon = 16 more than 90% of all adversarial examples were classified as least likely class y_{LL}.
For \epsilon=8 about 80% of all adversarial images were classified as y_{LL}
Adv_Alpha and Adv_Loss from [1] are shown to produce strong adversarial examples as could be seen from Figure 1 from that paper.
However these two methods are not suitable for our needs for the following reasons:
1. Both of them are using L_2 norm to construct adversarial examples.
Which means that adversarial examples would have fractional pixel values when converted to range [0, 255]. Given that we’re saving and printing adversarial images, all fractional values will be rounded up, and this would lead to decrease of quality of adversarial examples.
Section 2 of [1] mentions how to use Adv_Alpha method with L_{\infty} loss, however accuracy on such adversarial examples was not reported in [1]
2. Adv_Alpha method required to compute Jacobian matrix (i.e. compute gradient for each class) and then solve linear equation for each of class.
When dataset has a lot of classes(ImageNet has 1000 distinct classes) this become prohibitively expensive from the computational point of view. On the other iterative least likely class from our paper need to compute gradient only up to 20 times.
> Also, what is the actual perturbation of the basic iterative method in the results of Figure 2?
When \epsilon is large, the actual perturbation may be much smaller than \epsilon.
For epsilon from 2 to 16 (which we have used in our experiments) L_{\infty} norm of perturbation was equal or close to actual value of the epsilon.
For examples, for \epsilon=8, L_{\infty} norm of adversarial perturbation was 8 for all images from validation set. For \eps=16 more than 98% of generated adversarial perturbations had L_{\infty} norm equal to 16.
If you visually look at adversarial images produced by Fast and Iter L.L methods with same epsilon you might have an impression that perturbation of Iter L.L. adversarial images is smaller compared to Fast adversarial images.
This is explained by the fact that Fast method modifies all pixels of the image by the same absolute value (but with different sign).
At the same time iterative methods are modifying pixels by different value. And only one pixel has to be modified by epsilon for L_{\infy} norm to be equal epsilon.
The submission says, "The most similar work to this paper is Sharif et al. (2016), which appeared publicly after our work but had been submitted to a conference earlier." Can you provide more details about this timeline?
Other than the three-point description of differences, how does this work differ from the earlier conference submission?
> Can you provide more details about this timeline?
Our work appeared on arXiv on 8 July 2016, see timestamp of the first version at https://arxiv.org/abs/1607.02533
Sharif's work was submitted to the CCS2016 conference, which had submission deadline 23 May 2016 and acceptance notification on 22 July 2016 (see https://www.sigsac.org/ccs/CCS2016/call-for-papers/ ).
It seems that they made their work publicly available only in the beginning of November ( https://www.reddit.com/r/hackernews/comments/5auig2/real_and_stealthy_attacks_on_stateoftheart_face/ ). At that time we have learned about it and decided that we should add a citation.
> Other than the three-point description of differences, how does this work differ from the earlier conference submission?
The two works are somewhat different in spirit.
The goal of our work was to test whether existing forms of adversarial examples have an effect when displayed in the physical world. The goal of the work by Sharif et al was to develop a practical attack against a face recognition system.
Sharif et al developed an approach to printing out paper glasses frames that could be attached to real glasses. The design on the paper glasses is chosen to fool a face recognition system. In this case, the goal was primarily to demonstrate that this specific exploit was possible. The number of classes in the machine learning system is small (it can recognize the co-authors of the paper and a few celebrities) and some new development of adversarial example construction techniques was necessary to make the attack viable using only glasses frames.
In our work, the goal is to measure how well existing adversarial example techniques transfer to the real world. This meant that our focus was on evaluating large numbers of adversarial examples from large numbers of categories (the 1,000 imagenet categories) rather than on developing new adversarial example construction techniques.
Description.
The paper investigates whether adversarial examples survive different geometric and photometric image transformations,
including a complex transformation where the image is printed on the paper and captured again by a cell-phone camera.
The paper considers three different methods to generate adversarial examples — images with added small amount of noise that changes the output of a classification neural network.  In the quantitative experiments the paper assumes available access to the neural network and its parameters. Qualitative results are shown for a set-up where the network used to generate adversarial images is different from the test network.
Strong  points.
- adversarial examples are an interesting phenomenon that is worth detailed investigation.
- the paper is well written and presented.
- Results showing (and quantifying) that adversarial examples can survive a complex image transformation such as printing and re-capturing are interesting.
- Experiments are well done and solid.
Weak points:
- Probably the main negative point is the amount of novelty and contribution. The paper essentially presents a set of experiments evaluating whether adversarial examples survive different image transformations.
Apart from that there is no other main contribution / novelty.
While the experiments are solid and well-done, this seems borderline.
Detailed evaluation.
Originality:
- the main contribution of this work is the experimental evaluation showing (and quantifying) how adversarial examples behave under various image transformations.
Quality:
- The shown experiments are solid and well done.
Clarity:
- The paper is well written and clear.
Significance:
- The findings and shown experiments are interesting, but I not sure if the scale and amount of contribution is significant enough for the main conference track.
Overall:
Experimental paper. Well written. Solid experiments. Not sure if contribution is significant enough.
Thanks for the review.
We would like to address the significance of the paper.
We agree that we did not propose radically novel methods or explanations of the adversarial example problem.
At the same time the observations made in this paper are significant for practical ML applications because they show how real world machine learning systems are susceptible to adversarial attacks similar to “digital-only” machine learning systems.
Thus the novelty and significance of this paper is in uncovering a new risk for practical real-world machine learning systems. Or in other words it could be considered as a security paper with a new vulnerability disclosed.
In some sense, the Sharif et al. work "scooped" this paper, but as the authors indicate, the spirit of the work remains somewhat different. Sharif's approach was constrained in an interesting way (usable surface area limited to front portion of glasses frames) and also a bit gimmicky (focused on fooling a small scale face ID system to select among a set of celebrities). The present work is less sensational and more methodical in its study of physical manifestations of adversarial patterns for standard benchmark objects. I think the paper is at least a little above the bar since it poses an interesting question and carries out an informative empirical study.
Thanks for the review.
It's worth noting that the two papers mutually "scooped" each other.
While Sharif et al submitted to a conference earlier, our work has been publicly available longer and has more citations.
According to Google scholar on Jan 17, 2017, our article has been cited 9 times ( https://scholar.google.com/scholar?cites=263531058904899909 ), while that of Sharif et al has been cited 4 times ( https://scholar.google.com/scholar?cites=10006479087737690195 ). One of these 4 citations to Sharif et al is the citation we added in November.
The paper is well motivated and well written. The setting of the experiments is to investigate a particular case.
While the results of experiments are interesting, such investigation is not likely to systematically improve our understanding of the adversarial example phenomenon.
Overall, the contribution of the paper seems incremental.
Pros:
1. This paper proposes the iterative LL method, which is efficient in both computation and success rate in generating adversarial examples. This method could be useful when the number of classes in the dataset is huge.
2. Some observations of the experiments are interesting.
For example, overall photo transformation does not affect much the accuracy on clean image, but could destroy some adversarial methods.
Cons:
1. As noticed by the authors, some similar works exist in the literature. According to the authors, what differs this work from other existing works is that this paper tend to fool NN by making very small perturbations of the input.
But based on the experiments and the demonstration (the real pictures), it is arguable that the perturbations in the experiments are still small.
2. Some hypotheses proposed in the paper based on one-shot experiments seems too rushy.
3. As mentioned above, the results of this paper seems not really improving the understanding of the adversarial example phenomenon.
Thanks for the review.
We would like to address some of the cons you mentions.
We acknowledge that there are existing papers which describe various adversarial attacks (including white-box and black box attacks) on various machine learning systems (including neural networks).
However we would like to emphasize that the main difference of our work and prior work is the fact that the attacker in prior work has direct access to inputs of the classifier thus can fine-tune all input values, while in our work we have shown that adversarial attacks are possible even when data is perceived through the sensors (like a camera).
Overall this observation is important for practical machine learning systems operating in real world.
As we mentioned in the paper, the closest work to ours is Sharif et al.
However their work has somewhat different scope and also became publicly available after our paper.
Regarding whether the size of perturbation is small or not.
We agree that there is no good quantitative threshold assessing whether a perturbation is small or not and this is a very subjective measure. In this sense our primary criterion was whether the perturbation was large enough to interfere with human recognition of the image.
Also we have run additional experiments with eps=2 and 4 (results are added to Tables 1, 2 and 3 in the paper). These experiments are showing the same trend as experiments with higher epsilon - adversarial images may remain misclassified after “photo transformation”.
In particular, for eps=2 and 4 about half of “Fast” adversarial examples remain misclassified after “photo transformation” (see destruction rate in Table 3).
Arguably adversarial examples with eps=4 and 2 are small enough to be hard to notice (examples of adversarial image with eps=4 can be found in Figure 5).
So hopefully the fact that physical adversarial examples exist even for such small epsilon should address your concern.
Regarding “Some hypotheses proposed in the paper based on one-shot experiments seems too rushy,” we’d be happy to provide further support for any particular hypothesis you think is invalid.
Regarding “the results of this paper seems not really improving the understanding of the adversarial example phenomenon”: this paper is an empirical paper and not a theoretical paper, but we’ve found important empirical observations that theory should aim to explain, such as the reduced transferability of iterative adversarial examples.
In addition the paper helps to advance research on adversarial examples by showing that they also exist in the physical world.

Meta-review:
role: PC
data: comment: This paper studies an interesting aspect of adversarial training with important practical applications: its robustness against transformations that correspond to physical world constraints. The paper demonstrates how to construct adversarial examples such that they can be used to attack a machine-learning device through a physical interface such as a camera device. 
 
 The reviewers agreed that this is a well-written paper which clearly describes its contributions and offers a detailed experimental section. The authors took into account the reviewer's concerns and the rebuttal phase offered interesting discussions. 
 
 In light of the reviews, the main critique of this work is its lack of significance, relative to existing works in adversarial examples. The authors, however, did a good job during the rebuttal phase to highlight the empirical nature of the work and the potential practical significance of their findings in the design of ML models. The AC concludes that the potential practical applications, while not significant enough to be part of the conference proceedings, are worthy to be disseminated in the workshop. I therefore recommend submitting this work to the workshop track.

Decision: Invite to Workshop Track