Venue: ICLR 2018 conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The authors describe a new defense mechanism against adversarial attacks on classifiers (e.g., FGSM). They propose utilizing Generative Adversarial Networks (GAN), which are usually used for training generative models for an unknown distribution, but have a natural adversarial interpretation. In particular, a GAN consists of a generator NN G which maps a random vector z to an example x, and a discriminator NN D which seeks to discriminate between an examples produced by G and examples drawn from the true distribution. The GAN is trained to minimize the max min loss of D on this discrimination task, thereby producing a G (in the limit) whose outputs are indistinguishable from the true distribution by the best discriminator. 

Utilizing a trained GAN, the authors propose the following defense at inference time. Given a sample x (which has been adversarially perturbed), first project x onto the range of G by solving the minimization problem z* = argmin_z ||G(z) - x||_2. This is done by SGD. Then apply any classifier trained on the true distribution on the resulting x* = G(z*). 

In the case of existing black-box attacks, the authors argue (convincingly) that the method is both flexible and empirically effective. In particular, the defense can be applied in conjunction with any classifier (including already hardened classifiers), and does not assume any specific attack model. Nevertheless, it appears to be effective against FGSM attacks, and competitive with adversarial training specifically to defend against FGSM. 

The authors provide less-convincing evidence that the defense is effective against white-box attacks. In particular, the method is shown to be robust against FGSM, RAND+FGSM, and CW white-box attacks. However, it is not clear to me that the method is invulnerable to novel white-box attacks. In particular, it seems that the attacker can design an x which projects onto some desired x* (using some other method entirely), which then fools the classifier downstream.

Nevertheless, the method is shown to be an effective tool for hardening any classifier against existing black-box attacks 
(which is arguably of great practical value). It is novel and should generate further research with respect to understanding its vulnerabilities more completely. 

Minor Comments:
The sentence starting “Unless otherwise specified…” at the top of page 7 is confusing given the actual contents of Tables 1 and 2, which are clarified only by looking at Table 5 in the appendix. This should be fixed.  

review_title: A novel idea with room for future work. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents Defense-GAN: a GAN that used at test time to map the input generate an image (G(z)) close (in MSE(G(z), x)) to the input image (x), by applying several steps of gradient descent of this MSE. The GAN is a WGAN trained on the train set (only to keep the generator). The goal of the whole approach is to be robust to adversarial examples, without having to change the (downstream task) classifier, only swapping in the G(z) for the x.

+ The paper is easy to follow.
+ It seems (but I am not an expert in adversarial examples) to cite the relevant litterature (that I know of) and compare to reasonably established attacks and defenses.
+ Simple/directly applicable approach that seems to work experimentally, but
- A missing baseline is to take the nearest neighbour of the (perturbed) x from the training set.
- Only MNIST-sized images, and MNIST-like (60k train set, 10 labels) datasets: MNIST and F-MNIST.
- Between 0.043sec and 0.825 sec to reconstruct an MNIST-sized image.
? MagNet results were very often worse than no defense in Table 4, could you comment on that?
- In white-box attacks, it seems to me like L steps of gradient descent on MSE(G(z), x) should be directly extended to L steps of (at least) FGSM-based attacks, at least as a control.
review_title: Interesting but hard to conclude decisively from the current experiments

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents a method to cope with adversarial examples in classification tasks, leveraging a generative model of the inputs.  Given an accurate generative model of the input, this approach first projects the input onto the manifold learned by the generative model (the idea being that inputs on this manifold reflect the non-adversarial input distribution).  This projected input is then used to produce the classification probabilities.  The authors test their method on various adversarially constructed inputs (with varying degrees of noise). 

Questions/Comments:

- I am interested in unpacking the improvement of Defense-GAN over the MagNet auto-encoder based method.  Is the MagNet auto-encoder suffering lower accuracy because the projection of an adversarial image is based on an encoding function that is learned only on true data?  If the decoder from the MagNet approach were treated purely as a generative model, and the same optimization-based projection approach (proposed in this work) was followed, would the results be comparable?  

- Is there anything special about the GAN approach, versus other generative approaches? 

- In the black-box vs. white-box scenarios, can the attacker know the GAN parameters?  Is that what is meant by the "defense network" (in experiments bullet 2)?

- How computationally expensive is this approach take compared to MagNet or other adversarial approaches? 

Quality: The method appears to be technically correct.

Clarity: This paper clearly written; both method and experiments are presented well. 

Originality: I am not familiar enough with adversarial learning to assess the novelty of this approach. 

Significance: I believe the main contribution of this method is the optimization-based approach to project onto a generative model's manifold.  I think this kernel has the potential to be explored further (e.g. computational speed-up, projection metrics).
review_title: review

Meta-review: The paper studied defenses against adversarial examples by training a GAN and, at inference time, finding the GAN-generated sample that is nearest to the (adversarial) input example. Next, it classifies the generated example rather than the input example. This defense is interesting and novel. The CelebA experiments the authors added in their revision suggest that the defense can be effective on high-resolution RGB images.
Decision: Accept (Poster)