Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper creates adversarial images by imposing a flow field on an image such that the new spatially transformed image fools the classifier. They minimize a total variation loss in addition to the adversarial loss to create perceptually plausible adversarial images, this is claimed to be better than the normal L2 loss functions.

Experiments were done on MNIST, CIFAR-10, and ImageNet, which is very useful to see that the attack works with high dimensional images. However, some numbers on ImageNet would be helpful as the high resolution of it make it potentially different than the low-resolution MNIST and CIFAR.

It is a bit concerning to see some parts of Fig. 2. Some of Fig. 2 (especially (b)) became so dotted that it no longer seems an adversarial that a human eye cannot detect. And model B in the appendix looks pretty much like a normal model. It might needs some experiments, either human studies, or to test it against an adversarial detector, to ensure that the resulting adversarials are still indeed adversarials to the human eye. Another good thing to run would be to try the 3x3 average pooling restoration mechanism in the following paper:

Xin Li, Fuxin Li. Adversarial Examples Detection in Deep Networks with Convolutional Filter Statistics . ICCV 2017.

to see whether this new type of adversarial example can still be restored by a 3x3 average pooling the image (I suspect that this is harder to restore by such a simple method than the previous FGSM or OPT-type, but we need some numbers).

I also don't think FGSM and OPT are this bad in Fig. 4. Are the authors sure that if more regularization are used these 2 methods no longer fool the corresponding classifiers?

I like the experiment showing the attention heat maps for different attacks. This experiment shows that the spatial transforming attack (stAdv) changes the attention of the classifier for each target class, and is robust to adversarially trained Inception v3 unlike other attacks like FGSM and CW. 

I would likely upgrade to a 7 if those concerns are addressed.

After rebuttal: I am happy with the additional experiments and would like to upgrade to an accept.
review_title: Interesting solid paper, could use a few more experiments

final_score: rating: 7: Good paper, accept
review_content: This paper explores a new way of generating adversarial examples by slightly morphing the image to get misclassified by the model. Most other adversarial example generation methods tend to rely on generating high frequency noise patterns based by optimizing the perturbation on an individual pixel level. The new approach relies on gently changing the overall image by computing a flow an spatially transforming the image according to that flow. An important advantage of that approach is that the new attack is harder to protect against than to previous attacks according to the pixel based optimization methods.

The paper describes a novel model method that might become a new important line of attack. And the paper clearly demonstrates the advantages of this attack on three different data sets.

A minor nitpick: the "optimization based attack (Opt)" was first employed in the original "Intriguing Properties..." 2013 paper using box-LBFGS as the method of choice predating FGSM.
review_title: An interesting way of creating better adversarial examples.

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: This paper proposes a new way to create adversarial examples. Instead of changing pixel values they perform spatial transformations. 

The authors obtain a flow field that is optimized to fool a target classifier. A regularization term controlled by a parameter tau is ensuring very small visual difference between the adversarial and the original image. 

The used spatial transformations are differentiable with respect to the flow field (as was already known from previous work on spatial transformations) it is easy to perform gradient descent to optimize the flow that fools classifiers for targeted and untargeted attacks. 

The obtained adversarial examples seem almost imperceivable (at least for ImageNet). 
This is a new direction of attacks that opens a whole new dimension of things to consider. 

It is hard to evaluate this paper since it opens a new direction but the authors do a good job using numerous datasets, CAM attention visualization and also additional materials with high-res attacks. 

This is a very creative new and important idea in the space of adversarial attacks. 

Edit: After reading the other reviews , the replies to the reviews and the revision of the paper with the human study on perception, I increase my score to 9. This is definitely in the top 15% of ICLR accepted papers, in my opinion.   

Also a remark: As far as I understand, a lot of people writing comments here have a misconception about what this paper is trying to do: This is not about improving attack rates, or comparing with other attacks for different epsilons, etc. 
This is a new *dimension* of attacks. It shows that limiting l_inf of l_2 is not sufficient and we have to think of human perception to get the right attack model. Therefore, it is opening a new direction of research and hence it is important scholarship. It is asking a new question, which is frequently more important than improving performance on previous benchmarks. 


review_title: Moving the pixels to get adversarial attacks is also possible

Meta-review: All reviewers gave "accept" ratings.
it seems that everyone thinks this is interesting work.

The paper generated a large number of anonymous comments and these were addressed by the authors. 
Decision: Accept (Poster)