Title: Spatially Transformed Adversarial Examples
Authors: Chaowei Xiao
Jun-Yan Zhu
Bo Li
Warren He
Mingyan Liu
Dawn Song
Venue: ICLR 2018
Keywords: adversarial examples
spatial transformation

Abstract: Recent studies show that widely used Deep neural networks (DNNs) are vulnerable to the carefully crafted adversarial examples.
Many advanced algorithms have been proposed to generate adversarial examples by leveraging the L_p distance for penalizing perturbations.
Different defense methods have also been explored to defend against such adversarial attacks. 
While the effectiveness of L_p distance as a metric of perceptual quality remains an active research area, in this paper we will instead focus on a different type of perturbation, namely spatial transformation, as opposed to manipulating the pixel values directly as in prior works.
Perturbations generated through spatial transformation could result in large L_p distance measures, but our extensive experiments show that such spatially transformed adversarial examples are perceptually realistic and more difficult to defend against with existing defense systems. This potentially provides a new direction in adversarial example generation and the design of corresponding defenses.
We visualize the spatial transformation based perturbation for different examples and show that our technique
can produce realistic adversarial examples with smooth image deformation.
Finally, we visualize the attention of deep networks with different types of adversarial examples to better understand how these examples are interpreted.

TL;DR: We propose a new approach for generating adversarial examples based on spatial transformation, which produces perceptually realistic examples compared to existing attacks. 

Review discussion:
This paper explores a new way of generating adversarial examples by slightly morphing the image to get misclassified by the model. Most other adversarial example generation methods tend to rely on generating high frequency noise patterns based by optimizing the perturbation on an individual pixel level. The new approach relies on gently changing the overall image by computing a flow an spatially transforming the image according to that flow. An important advantage of that approach is that the new attack is harder to protect against than to previous attacks according to the pixel based optimization methods.
The paper describes a novel model method that might become a new important line of attack. And the paper clearly demonstrates the advantages of this attack on three different data sets.
A minor nitpick: the "optimization based attack (Opt)" was first employed in the original "Intriguing Properties..." 2013 paper using box-LBFGS as the method of choice predating FGSM.
We thank the reviewer for the constructive suggestions! We have updated the method Opt to C&W throughout the paper.
This paper creates adversarial images by imposing a flow field on an image such that the new spatially transformed image fools the classifier. They minimize a total variation loss in addition to the adversarial loss to create perceptually plausible adversarial images, this is claimed to be better than the normal L2 loss functions.
Experiments were done on MNIST, CIFAR-10, and ImageNet, which is very useful to see that the attack works with high dimensional images.
However, some numbers on ImageNet would be helpful as the high resolution of it make it potentially different than the low-resolution MNIST and CIFAR.
It is a bit concerning to see some parts of Fig. 2. Some of Fig. 2 (especially (b)) became so dotted that it no longer seems an adversarial that a human eye cannot detect. And model B in the appendix looks pretty much like a normal model. It might needs some experiments, either human studies, or to test it against an adversarial detector, to ensure that the resulting adversarials are still indeed adversarials to the human eye.
Another good thing to run would be to try the 3x3 average pooling restoration mechanism in the following paper:
Xin Li, Fuxin Li. Adversarial Examples Detection in Deep Networks with Convolutional Filter Statistics . ICCV 2017.
to see whether this new type of adversarial example can still be restored by a 3x3 average pooling the image (I suspect that this is harder to restore by such a simple method than the previous FGSM or OPT-type, but we need some numbers).
I also don't think FGSM and OPT are this bad in Fig. 4. Are the authors sure that if more regularization are used these 2 methods no longer fool the corresponding classifiers?
I like the experiment showing the attention heat maps for different attacks. This experiment shows that the spatial transforming attack (stAdv) changes the attention of the classifier for each target class, and is robust to adversarially trained Inception v3 unlike other attacks like FGSM and CW.
I would likely upgrade to a 7 if those concerns are addressed.
After rebuttal: I am happy with the additional experiments and would like to upgrade to an accept.
We thank the reviewer for the thoughtful comments and suggestions.
Human study:
We have added a human study in Section 4.3.
In particular, we follow the same perceptual study protocol used in prior image synthesis work [Zhang et al. 2016; Isola et al. 2017]. In our study, the participants are asked to choose the more visually realistic image between (1) an adversarial example generated by stAdv and (2) its original image. The user study shows that the generated adversarial examples can fool human participants 47% of the time (perfectly realistic results would achieve 50%). This experiment shows that our adversarial examples are almost indistinguishable from natural images. Please see section 4.3 for more details.
3x3 mean blur defense
We included the suggested related work and added an analysis of the 3x3 average pooling restoration mechanism [Li et al. 16’]. See section 4.4 and Table 5 in Appendix B for the discussion and results.
In summary, the restoration is not as effective on stAdv examples. The classification accuracy on restored stAdv examples is around 50% (Table 5), compared to restored C&W examples (around 80%) and FGSM examples (around 70%)  [Carlini et al. 2017, Li et al. 2016].
In addition, stAdv achieves near 100% attack success rate in a perfect knowledge adaptive attack [Carlini et al. 2017].
Comparison with C&W and FGSM (Figure 4)
In our revised version, we have updated Figure 4 to show adversarial examples for FGSM and C&W with a strong adversarial budget as: L_infinity perturbation limit of 0.3 on MNIST and 8 on CIFAR-10. We apply the same setting for the later evaluations against defenses
References:
[Carlini et al. 2017] Carlini, Nicholas, and David Wagner. "Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods." arXiv preprint arXiv:1705.07263 (2017).
[Li et al. 2016] Li, Xin, and Fuxin Li. "Adversarial examples detection in deep networks with convolutional filter statistics." arXiv preprint arXiv:1612.07767 (2016).
[Zhang et al. 2016] Zhang, Richard, Phillip Isola, and Alexei A. Efros. "Colorful image colorization." European Conference on Computer Vision. Springer International Publishing, 2016.
[Isola et al. 2017]Isola, Phillip, et al. "Image-to-image translation with conditional adversarial networks." arXiv preprint arXiv:1611.07004 (2016).
This paper proposes a new way to create adversarial examples.
Instead of changing pixel values they perform spatial transformations.
The authors obtain a flow field that is optimized to fool a target classifier. A regularization term controlled by a parameter tau is ensuring very small visual difference between the adversarial and the original image.
The used spatial transformations are differentiable with respect to the flow field (as was already known from previous work on spatial transformations) it is easy to perform gradient descent to optimize the flow that fools classifiers for targeted and untargeted attacks.
The obtained adversarial examples seem almost imperceivable (at least for ImageNet).
This is a new direction of attacks that opens a whole new dimension of things to consider.
It is hard to evaluate this paper since it opens a new direction but the authors do a good job using numerous datasets, CAM attention visualization and also additional materials with high-res attacks.
This is a very creative new and important idea in the space of adversarial attacks.
Edit: After reading the other reviews , the replies to the reviews and the revision of the paper with the human study on perception, I increase my score to 9. This is definitely in the top 15% of ICLR accepted papers, in my opinion.
Also a remark: As far as I understand, a lot of people writing comments here have a misconception about what this paper is trying to do: This is not about improving attack rates, or comparing with other attacks for different epsilons, etc.
This is a new *dimension* of attacks. It shows that limiting l_inf of l_2 is not sufficient and we have to think of human perception to get the right attack model.
Therefore, it is opening a new direction of research and hence it is important scholarship. It is asking a new question, which is frequently more important than improving performance on previous benchmarks.
We thank the reviewer for the helpful comments. To further improve our work, we have added a user study to our updated version to evaluate the perceptual realism for the generated instances.
In particular, we follow the same perceptual study protocol used in prior image synthesis work [Zhang et al. 2016; Isola et al. 2017]. In our study, the participants are asked to choose the more visually realistic image between (1) an adversarial example generated by stAdv and (2) its original image. The user study shows that the generated adversarial examples can fool human participants 47% of the time (perfectly realistic results would achieve 50%). This experiment shows that our adversarial examples are almost indistinguishable from natural images. Please see section 4.3 for more details.
Changes made in our revised version are listed as below:
- Added human perceptual study of our algorithm in section 4.3.
- Analyzed the efficiency of mean blur defense strategy against our algorithm in section 4.4.
- Added a detailed description of experiment setting for C&W and FGSM method in section 4.4.
- Added more adversarial examples for ImageNet-Compatible set, MNIST, and CIFAR-10 in Appendix C.
- Updated figure 4 with a strong adversarial budget on MNIST and CIFAR-10.
- Fixed some grammatical errors.
- Changed the name “Opt” to “C&W”.
We would like to thank the reviewers again for the useful feedbacks and suggestions.

Meta-review:
role: PC
data: comment: All reviewers gave "accept" ratings.
it seems that everyone thinks this is interesting work.

The paper generated a large number of anonymous comments and these were addressed by the authors. 

Decision: Accept (Poster)