Title: Synthesizing Robust Adversarial Examples
Authors: Anish Athalye
Logan Engstrom
Andrew Ilyas
Kevin Kwok
Venue: ICLR 2018
Keywords: adversarial examples

Abstract: Neural network-based classifiers parallel or exceed human-level accuracy on many common tasks and are used in practical systems. Yet, neural networks are susceptible to adversarial examples, carefully perturbed inputs that cause networks to misbehave in arbitrarily chosen ways. When generated with standard methods, these examples do not consistently fool a classifier in the physical world due to a combination of viewpoint shifts, camera noise, and other natural transformations. Adversarial examples generated using standard techniques require complete control over direct input to the classifier, which is impossible in many real-world systems.

We introduce the first method for constructing real-world 3D objects that consistently fool a neural network across a wide distribution of angles and viewpoints. We present a general-purpose algorithm for generating adversarial examples that are robust across any chosen distribution of transformations. We demonstrate its application in two dimensions, producing adversarial images that are robust to noise, distortion, and affine transformation. Finally, we apply the algorithm to produce arbitrary physical 3D-printed adversarial objects, demonstrating that our approach works end-to-end in the real world. Our results show that adversarial examples are a practical concern for real-world systems.


TL;DR: We introduce a new method for synthesizing adversarial examples robust in the physical world and use it to fabricate the first 3D adversarial objects.

Review discussion:
Summary: This work proposes a way to create 3D objects to fool the classification of their pictures from different view points by a neural network.
Rather than optimizing the log-likelihood of a single example, the optimization if performed over a the expectation of a set of transformations of sample images. Using an inception v3 net, they create adversarial attacks on a subset of the imagenet validation set transformed by translations, lightening conditions, rotations, and scalings among others, and observe a drop of the classifier accuracy performance from 70% to less than 1%. They also create two 3D printed objects which most pictures taken from random viewpoints are fooling the network in its class prediction.
Main comments:
- The idea of building 3D adversarial objects is novel so the study is interesting.
However, the paper is incomplete, with a very low number of references, only 2 conference papers if we assume the list is up to date.
See for instance Cisse et al. Houdini: fooling Deep Structured Prediction Models, NIPS 2017 for a recent list of related work in this research area.
- The presentation of the results is not very clear. See specific comments below.
- It would be nice to include insights to improve neural nets to become less sensitive to these attacks.
Minor comments:
Fig1 : a bug with color seems to have been fixed
Model section: be consistent with the notations. Bold everywhere or nowhere
Results: The tables are difficult to read and should be clarified:
What does the l2 metric stands for ?
How about min, max ?
Accuracy -> classification accuracy
Models -> 3D models
Describe each metric (Adversarial, Miss-classified, Correct)
Thank you for your review. In the latest revision of our paper we greatly expand on the related work section, both discussing in more detail our current list, and introducing other related works from the field which we explain and differentiate from our own. We hope that this gives the reader a more complete view of the field, and further indicates the novelty of our work.
The focus of this work was to demonstrate that it is possible to construct transformation-tolerant adversarial examples, even in the physical world; defenses against adversarial examples are beyond the scope of this paper. We hesitate to present intuitions for defenses without rigorous experimentation, because as researchers like Carlini have shown, developing defenses is challenging, and many proposed ideas for defenses are easily defeated [1].
We have addressed all of the minor comments including:
* Fixing the color bug
* Removing the selected bolding from the model section
* Elaborated and defined the l2 metric, and removed min/max in favor of mean/stdev
* Models -> 3D models and accuracy -> Classification accuracy
* Added a paragraph defining the terms “adversarial,” “misclassified,” and “correct” as we use them
[1]: https://arxiv.org/abs/1705.07263
The authors present a method to enable robust generation of adversarial visual
inputs for image classification.
They develop on the theme that 'real-world' transformations typically provide a
countermeasure against adversarial attacks in the visual domain, to show that
contextualising the adversarial exemplar generation by those very
transformations can still enable effective adversarial example generation.
They adapt an existing method for deriving adversarial examples to act under a
projection space (effectively a latent-variable model) which is defined through
a transformations distribution.
They demonstrate the effectiveness of their approach in the 2D and 3D
(simulated and real) domains.
The paper is clear to follow and the objective employed appears to be sound. I
like the idea of using 3D generation, and particularly, 3D printing, as a means
of generating adversarial examples -- there is definite novelty in that
particular exploration for adversarial examples.
I did however have some concerns:
1. What precisely is the distribution of transformations used for each
experiment? Is it a PCFG? Are the different components quantised such that
they are discrete rvs, or are there still continuous rvs? (For example, is
lighting discretised to particular locations or taken to be (say) a 3D
Gaussian?) And on a related note, how were the number of sampled
transformations chosen?
Knowing the distribution (and the extent of it's support) can help situate
the effectiveness of the number of samples taken to derive the adversarial
input.
2.
While choosing the distance metric in transformed space, LAB is used, but
for the experimental results, l_2 is measured in RGB space -- showing the
RGB distance is perhaps not all that useful given it's not actually being
used in the objective. I would perhaps suggest showing LAB, maybe in
addition to RGB if required.
3. Quantitative analysis: I would suggest reporting confidence intervals;
perhaps just the 1st standard deviation over the accuracies for the true and
'adversarial' labels -- the min and max don't help too much in understanding
what effect the monte-carlo approximation of the objective has on things.
Moreover, the min and max are only reported for the 2D and rendered 3D
experiments -- it's missing for the 3D printing experiment.
4. Experiment power: While the experimental setup seems well thought out and
structured, the sample size (i.e, the number of entities considered) seems a
bit too small to draw any real conclusions from. There are 5 exemplar
objects for the 3D rendering experiment and only 2 for the 3D printing one.
While I understand that 3D printing is perhaps not all that scalable to be
able to rattle off many models, the 3D rendering experiment surely can be
extended to include more models? Were the turtle and baseball models chosen
randomly, or chosen for some particular reason? Similar questions for the 5
models in the 3D rendering experiment.
5. 3D printing experiment transformations: While the 2D and 3D rendering
experiments explicitly state that the sampled transformations were random,
the 3D printing one says "over a variety of viewpoints". Were these
viewpoints chosen randomly?
Most of these concerns are potentially quirks in the exposition rather than any
issues with the experiments conducted themselves. For now, I think the
submission is good for a weak accept –- if the authors address my concerns, and/or
correct my potential misunderstanding of the issues, I'd be happy to upgrade my
review to an accept.
Thank you for your review. We have made several clarifications in the exposition that we believe address your concerns and improve the paper.
In particular:
1. The parameters of the distribution used in generating examples was given in the Appendix, but the method by which they are sampled was not made clear; we now explicitly state in the evaluation section that the parameters are sampled as independent uniformly distributed continuous random variables (except for Gaussian noise, which is sampled as a Gaussian continuous RV). There was no fixed number of transformations chosen during the synthesis of the adversarial example: the transformations are independently sampled at each gradient descent step. We have updated the text in the approach section to clarify this.
2. Yes, we agree: we minimized LAB, not RGB, and Euclidean distances make more sense in a perceptually uniform color space like LAB. We have switched to reporting LAB distances.
3.
While we gave the distribution of adversariality across examples in a graph the appendix, we did not explicitly state the standard deviation/confidence intervals. This has been resolved in the latest version. We have also removed the min and max metrics from the evaluation section, and have added the standard deviation over the accuracies for the true and ‘adversarial’ labels as suggested. We report mean/standard deviation for 2D and rendered 3D experiments and not the 3D printing experiment because we report the statistics for each 3D objects separately.
4. In the case of the 3D printing experiment, we were limited by printer capability and shipping feasibility for this revision, but would be happy to include a few more in the camera ready version. We also included 5 more models in the 3D rendering experiment, making a total of 200 adversarial examples (10 models, 20 randomly chosen targets for each model). The turtle and baseball models were chosen because they could be easily adapted for the 3D printing process. The adversarial targets for the turtle and baseball (as well as all our other experiments) were randomly chosen across all the eligible ImageNet classes. Models for the 3D simulation experiment were chosen based on the first 10 realistic, textured 3D models we could find in OBJ format.
5. We have added a footnote to address this concern;
although the viewpoints were not selected or cherry-picked in any capacity, we opt to not call them “random” because in contrast to the 2D and 3D virtual examples, the viewpoints were not (and realistically could not have been) uniformly sampled from some concrete distribution of viewpoints;
instead the objects were repeatedly moved and rotated on a table with humans walking around them and taking pictures from “a variety of viewpoints.”
The paper proposes a method to synthesize adversarial examples that remain robust to different 2D and 3D perturbations. The paper shows this is effective by transferring the examples to 3D objects that are color 3D-printed and show some nice results.
The experimental results and video showing that the perturbation is effective for different camera angles, lighting conditions and background is quite impressive. This work convincingly shows that adversarial examples are a real-world problem for production deep-learning systems rather than something that is only academically interesting.
However, the authors claim that standard techniques require complete control and careful setups (e.g. in the camera case) is quite misleading, especially with regards to the work by Kurakin et. al. This paper also seems to have some problems of its own (for example the turtle is at relatively the same distance from the camera in all the examples, I expect the perturbation wouldn't work well if it was far enough away that the camera could not resolve the HD texture of the turtle).
One interesting point this work raises is whether the algorithm is essentially learning universal perturbations (Moosavi-Dezfooli et. al). If that's the case then complicated transformation sampling and 3D mapping setup would be unnecessary. This may already be the case since the training set already consists of multiple lighting, rotation and camera type transformations so I would expect universal perturbations to already produce similar results in the real-world.
Minor comments:
Section 1.1: "a affine" -> "an affine"
Typo in section 3.4: "of a of a"
It's interesting in figure 9 that the crossword puzzle appears in the image of the lighthouse.
Moosavi-Dezfooli, S. M., Fawzi, A., Fawzi, O., & Frossard, P. Universal adversarial perturbations. CVPR 2017.
Thank you for the review and detailed comments. We are glad you enjoyed the paper. We have made revisions to the related work section, including a clearer description of Kurakin et al. and a more thorough discussion of other works (including the suggested “Universal Perturbations” paper by Moosavi-Dezfooli et al). We have additionally fixed all the minor issues you pointed out.
We thank the anonymous reviewers for helping us improve the paper. In response to their feedback, we have made the following revisions to our paper (in addition to the fixing of a few spelling mistakes/typos):
* Related work: We have updated our description of Kurakin et al. and added a comparison with Universal Adversarial Perturbations (Moosavi-Dezfooli et al.) and adversarial eyeglasses (Sharif et al.)
* Evaluation: We have included an additional 5 models in our Robust 3D Adversarial Examples evaluation. We have additionally further explained our evaluation metrics, improved and defined previously confusing terminology, and added standard deviations to further elucidate the distributions of adversariality and classification accuracy.

Meta-review:
role: PC
data: comment: This paper studies the problem of synthesizing adversarial examples that will succeed at fooling a classification system under unknown viewpoint, lighting, etc conditions. For that purpose, the authors propose a data-augmentation technique (called "EOT") that makes adversarial examples robust against a predetermined family of transformations.

Reviewers were mixed in their assessment of this work, on the one hand highlighting the potential practical applications, but on the other hand warning about weak comparisons with existing literature, as well as lack of discussion about how to improve the robustness of the deep neural net against that form of attacks.
The AC thus believes this paper will greatly benefit from a further round of iteration/review, and therefore recommends rejection at this time. 

Decision: Reject