Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: I thank the authors for the thoughtful response and rebuttal. The authors have substantially updated their manuscript and improved the presentation.

Re: Speed. I brought up this point because this was a bulleted item in the Introduction in the earlier version of the manuscript. In the revised manuscript, this bullet point is now removed. I will take this point to be moot.

Re: High resolution. The authors point to recent GAN literature that provides some first results with high resolution GANs but I do not see quantitative evidence in the high resolution setting for this paper. (Figure 4 provides qualitative examples from ImageNet but no quantitative assessment.)

Because the authors improved the manuscript, I upwardly revised my score to 'Ok but not good enough - rejection'. I am not able to accept this paper because of the latter point.
==========================

The authors present an interesting new method for generating adversarial examples. Namely, the author train a generative adversarial network (GAN) to adversarial examples for a target network. The authors demonstrate that the network works well in the semi-white box and black box settings.

The authors wrote a clear paper with great references and clear descriptions.

My primary concern is that this work has limited practical benefit in a realistic setting. Addressing each and every concern is quite important:

1) Speed. The authors suggest that training a GAN provides a speed benefit with respect to other attack techniques. The FGSM method (Goodfellow et al, 2015) is basically 1 inference operation and 1 backward operation. The GAN is 1 forward operation. Granted this results in a small difference in timing 0.06s versus 0.01s, however it would seem that avoiding a backward pass is a somewhat small speed gain.
 
Furthermore, I would want to question the practical usage of having an 'even faster' method for generating adversarial examples. What is the reason that we need to run adversarial attacks 'even faster'? I am not aware of any use-cases, but if there are some, the authors should describe the rationales at length in their paper.

2) High spatial resolution images. Previous methods, e.g. FGSM, may work on arbitrarily sized images. At best, GANs generate reasonable images that are lower resolutions (e.g. < 128x128). Building GAN's that operate above-and-beyond moderate spatial resolution is an open research topic. The best GAN models for generating high resolution images are  difficult to train and it is not clear if they would work in this setting. Furthermore, images with even higher resolutions, e.g. 512x512, which is quite common in ImageNet, are difficult to synthesizes using current techniques.

3) Controlling the amount of distortion. A feature of previous optimization based methods is that a user may specify the amount of perturbation (epsilon). This is a key feature if not requirement in an adversarial perturbation because a user might want to examine the performance of a given model as a function of epsilon. Performing such an analysis with this model is challenging (i.e. retraining a GAN) and it is not clear if a given image generated by a GAN will always achieve a given epsilon perturbation/

On a more minor note, the authors suggest that generating a *diversity* of adversarial images is of practical import. I do not see the utility of being able to generate a diversity of adversarial images. The authors need to provide more justification for this motivation.
review_title: Review for 'Generating Adversarial Examples with Adversarial Networks'

final_score: rating: 7: Good paper, accept
review_content: The paper proposes a way of generating adversarial examples that fool classification systems.
They formulate it for a blackbox and a semi-blackbox setting (semi being, needed for training their own network, but not to generate new samples).

The model is a residual gan formulation, where the generator generates an image mask M, and (Input + M) is the adversarial example.
The paper is generally easy to understand and clear in their results.
I am not awfully familiar with the literature on adversarial examples to know if other GAN variants exist. From this paper's literature survey, they dont exist. 
So this paper is innovative in two parts:
- it applies GANs to adversarial example generation
- the method is a simple feed-forward network, so it is very fast to compute

The experiments are pretty robust, and they show that their method is better than the proposed baselines.
I am not sure if these are complete baselines or if the baselines need to cover other methods (again, not fully familiar with all literature here).

review_title: adversarial adversial example generation, wins MadryLab's mnist challenge

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper describes AdvGAN, a conditional GAN plus adversarial loss. AdvGAN is able to generate adversarial samples by running a forward pass on generator. The authors evaluate AdvGAN on semi-white box and black box setting.

AdvGAN is a simple and neat solution to for generating adversary samples. The author also reports state-of-art results.

Comment:

1. For MNIST samples, we can easily find the generated sample is a mixture of two digitals. Eg, for digital 7 there is a light gray 3 overlap. I am wondering this method is trying to mixture several samples into one to generate adversary samples. For real color samples, it is harder to figure out the mixture.
2. Based on mixture assumption, I suggest the author add one more comparison to other method, which is relative change from original image, to see whether AdvGAN is the most efficient model to generate the adversary sample (makes minimal change to original image).




review_title: Review

Meta-review: The paper presents AdvGAN: a GAN that is trained to generate adversarial examples against a convolutional network. The motivation for this method is unclear: the proposed attack does not outperform simpler attack methods such as Carlini-Wagner attack. In white-box settings, a clear downside for the attacker is that it needs to re-train its GAN everytime the defender changes its convolutional network.

More importantly, the work appears preliminary. In particular, the lack of extensive quantitative experiments on ImageNet makes it difficult to compare the proposed approach to alternative attacks methods such as (I-)FGSM, DeepFool, and Carlini-Wagner. The fact that AdvGAN performs well on MNIST is nice, but MNIST should be considered for what it is: a toy dataset. If AdvGANs are, as the authors state in their rebuttal, fast and good at generating high-resolution images, then it should be straightforward to perform comprehensive experiments with AdvGANs on ImageNet (rather than focusing on a small number of images on a single target, as the authors did in their revision)?
Decision: Reject