Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: 
I read the rebuttal and thank the authors for the thoughtful responses and revisions. The updated Figure 2 and Section 4.4. addresses my primary concerns. Upwardly revising my review.

====================

The authors describe a method for detecting adversarial examples by measuring the likelihood in terms of a generative model of an image. Furthermore, the authors prescribe a method for cleaning or 'santizing' an adversarial image through employing a generative model. The authors demonstrate some success in restoring images that have been adversarially perturbed with this technique.

The idea of using a generative model (PixelCNN) to assess whether a given image has been adversarially perturbed is a very interesting and understandable finding that may contribute quite nicely to the adversarial literature. One limitation of this method, however, is our ability to build successful generative models for high resolution images. However, I would be curious to know if the authors tried their method on high resolution images, regardless?

Major comments:
1) Cross validation. Figure 2a is quite interesting and compelling. It is not clear from the figure if the 'clean' (nor the other data for that matter) is from the *training* or  *testing* data for the PixelCNN model. I would *hope* that this is from the *testing* data indicating that these are the likelihood on unseen images?

That said, it would be interesting to see the *training* data on this plot as well to see if there are any systematic shifts that might make the distribution of adversarial examples less discernible.

2) Adversary to PixelCNN. It is not clear why a PixelCNN may not be adversarially attacked, nor if such a model would be able to guard against an adversarial attack. I am not sure how well viable of strategy this may be but it is worth understanding or addressing to determine how viable this method for guarding actually is.

3) Restorative effects of PixelDefend. I would like to see individual examples of (a) adversarial perturbation for a given image and (b) PixelDefend perturbation for that adversarial image. In particular, I would like to see how close (a) is the negative of (b). This would give me more confidence that this techniques is successfully guarding against the original attack.

I am willing to adjust my rating upward if the authors are able to address some of the points above in a substantive manner. 


review_title: Review of PixelDefend

final_score: rating: 7: Good paper, accept
review_content: The paper describes the creative application of a density estimation model to clean up adversarial examples before applying and image model (for classification, in this setup). The basic idea is that the image is first moved back to the probable region of images before applying the classifier. For images, the successful PiexlCNN model is used as a density estimator and is applied to clean up the image before the classification is attempted.

The proposed method is very intuitive, but might be expensive if a naive implementation of PixelCNN is used for the cleaning. The approach is novel. It is useful that the density estimator model does not have to rely on the labels. Also, it might even be trained on a different dataset potentially.

The con is that the proposed methodology still does not solve the problem of adversarial examples completely.

Minor nitpick: In section 2.1, it is suggested that DeepFool was the first optimization based attack to minimize the perturbation wrt the original image. In fact the much earler (2013) "Intriguing Propoerties ... " paper relied on the same formulation (minimizing perturbation under several constraints: changed detection and pixel intensities are being in the given range).
review_title: A convincing new way to defend image models against adversarial examples.

final_score: rating: 7: Good paper, accept
review_content: The authors propose to use a generative model of images to detect and defend against adverarial examples. White-box attacks against standard models for image recognition (Resnet and VGG) are considered, and a generative model (a PixelCNN) is trained on the same data as the classifiers. The authors first show that adversarial examples created by the white-box attacks correspond to low likelihood region (according to the pixelCNN), which first gives a classification rule for detecting adversarial examples.

Then, to turn the genrative model into a defensive algorithm, the authors propose to preprocess test images by approximately maximizing the likelihood under similar constraints as the attacker of images, to "project" adversarial examples back to high-density regions (as estimated by the generative model). As a heuristic method, the authors propose to greedily maximize the likelihood of the incoming images pixel-by-pixel, which is possible because of the specific form of the PixelCNN likelihood in the context of l-infty attacks. An "adaptive" version of the algorithm, in which the preprocessing is used only when the likelihood of an example is below a certain threshold, is also proposed.

Experiments are carried out on Fashion MNIST and CIFAR-10. At a high level, the message is that projecting the image into a high density region is sufficient to correct for a significant portions of the mistakes made on adversarial examples. The main result is that this approach based on generative models seems to work even on against the strongest attacks.

Overall, the idea proposed in the paper, using a generative model to detect and filter out spurious patterns that can appear in adversarial examples, is rather intuitive. The experimental result that adversarial examples can somehow be corrected by a generative model is also interesting. The design choice of PixelCNN, which allows for a greedy optimization seems reasonable in that setting.

Whereas the paper is an interesting step forward, the paper still doesn't provide definitive arguments in favor of using such approaches in practice. There is a significant loss in accuracy on clean examples (2% on CIFAR-10 for a resnet), and more generally against weaker opponents such as the fast gradient sign. Thus, in reality, the experiments show that the pipeline generative model + classifier is robust against the strongest white box methods for this classifier, but on the other hand these methods do not transfer well to new models. This somewhat weakens the result, since robustness against these methods that do not transfer well is achieved by changing the model. 

review_title: interesting experimental results, but no definitive argument

Meta-review: The paper studies the use of PixelCNN density models for the detection of adversarial images, which tend to lie in low-probability parts of image space. The work is novel, relevant to the ICLR community, and appears to be technically sound.

A downside of the paper is its limited empirical evaluation: there evidence suggesting that defenses against adversarial examples that work well on MNIST/CIFAR do not necessarily transfer well to much higher-dimensional datasets, for instance, ImageNet. The paper could, therefore, would benefit from empirical evaluations of the defense on a dataset like ImageNet.
Decision: Accept (Poster)