Title: Generating Adversarial Examples with Adversarial Networks
Authors: Chaowei Xiao
Bo Li
Jun-Yan Zhu
Warren He
Mingyan Liu
Dawn Song
Venue: ICLR 2018
Keywords: adversarial examples
generative adversarial network
black-box attack

Abstract: Deep neural networks (DNNs) have been found to be vulnerable to adversarial examples resulting from adding small-magnitude perturbations to inputs. Such adversarial examples can mislead DNNs to produce adversary-selected results.
Different attack strategies have been proposed to generate adversarial examples, but how to produce them with high perceptual quality and more efficiently requires more research efforts. 
In this paper, we propose AdvGAN to generate adversarial examples with generative adversarial networks (GANs), which can learn and approximate the distribution of original instances. 
For AdvGAN, once the generator is trained, it can generate adversarial perturbations efficiently for any instance, so as to potentially accelerate adversarial training as defenses.  
We apply AdvGAN in both semi-whitebox and black-box attack settings. In semi-whitebox attacks, there is no need to access the original target model after the generator is trained, in contrast to traditional white-box attacks. In black-box attacks, we dynamically train a distilled model for the black-box model and optimize the generator accordingly.
Adversarial examples generated by AdvGAN on different target models have high attack success rate under state-of-the-art defenses compared to other attacks. Our attack  has placed the first with 92.76% accuracy on a public MNIST black-box attack challenge. 

TL;DR: We propose to generate adversarial example based on generative adversarial networks in a semi-whitebox and black-box settings.

Review discussion:
I thank the authors for the thoughtful response and rebuttal. The authors have substantially updated their manuscript and improved the presentation.
Re: Speed. I brought up this point because this was a bulleted item in the Introduction in the earlier version of the manuscript. In the revised manuscript, this bullet point is now removed. I will take this point to be moot.
Re: High resolution. The authors point to recent GAN literature that provides some first results with high resolution GANs but I do not see quantitative evidence in the high resolution setting for this paper. (Figure 4 provides qualitative examples from ImageNet but no quantitative assessment.)
Because the authors improved the manuscript, I upwardly revised my score to 'Ok but not good enough - rejection'. I am not able to accept this paper because of the latter point.
The authors present an interesting new method for generating adversarial examples.
Namely, the author train a generative adversarial network (GAN) to adversarial examples for a target network. The authors demonstrate that the network works well in the semi-white box and black box settings.
The authors wrote a clear paper with great references and clear descriptions.
My primary concern is that this work has limited practical benefit in a realistic setting. Addressing each and every concern is quite important:
1) Speed. The authors suggest that training a GAN provides a speed benefit with respect to other attack techniques. The FGSM method (Goodfellow et al, 2015) is basically 1 inference operation and 1 backward operation. The GAN is 1 forward operation.
Granted this results in a small difference in timing 0.06s versus 0.01s, however it would seem that avoiding a backward pass is a somewhat small speed gain.
Furthermore, I would want to question the practical usage of having an 'even faster' method for generating adversarial examples. What is the reason that we need to run adversarial attacks 'even faster'? I am not aware of any use-cases, but if there are some, the authors should describe the rationales at length in their paper.
2) High spatial resolution images. Previous methods, e.g. FGSM, may work on arbitrarily sized images. At best, GANs generate reasonable images that are lower resolutions (e.g. < 128x128). Building GAN's that operate above-and-beyond moderate spatial resolution is an open research topic. The best GAN models for generating high resolution images are  difficult to train and it is not clear if they would work in this setting.
Furthermore, images with even higher resolutions, e.g. 512x512, which is quite common in ImageNet, are difficult to synthesizes using current techniques.
3) Controlling the amount of distortion. A feature of previous optimization based methods is that a user may specify the amount of perturbation (epsilon). This is a key feature if not requirement in an adversarial perturbation because a user might want to examine the performance of a given model as a function of epsilon. Performing such an analysis with this model is challenging (i.e. retraining a GAN) and it is not clear if a given image generated by a GAN will always achieve a given epsilon perturbation/
On a more minor note, the authors suggest that generating a *diversity* of adversarial images is of practical import. I do not see the utility of being able to generate a diversity of adversarial images. The authors need to provide more justification for this motivation.
We thank the reviewer for the thoughtful comments and suggestions.
Speed. Speed is just one advantage of our method and is not the main motivation of our method. We agree with the reviewer that FGSM is already fast enough for most applications.
However, our proposed model is much more effective in the fooling rate against both white-box and blackbox settings than the FGSM method and is still faster than the FGSM method. In wide resnet setting on CIFAR, FGSM takes 0.39s to generate 100 examples (1 forward and 1 backward pass through the classifier), while AdvGAN takes 0.16s to generate 100 examples (1 forward pass through the generator). In our experiments, the classifier was a wide resnet with 46.16M parameters, while the generator had 0.24M parameters. The speed difference is even larger with deeper classifiers.
Moreover, in AdvGAN, the forward operation does not use classifier’s network, but uses generator’s network.
Overall, we claim to have developed a faster and more effective alternative method to generating adversarial examples, but improving the speed is just a byproduct for us and generating more photorealistic and effective adversarial examples in both semi white-box and blackbox settings is the main goal.
High spatial resolution images. Early GANs have had this problem.
However, we claim that AdvGAN still works at high spatial resolution (and it is not unique in doing so). Here are three techniques we applied in AdvGAN.
(i) Previous work in high resolution: Our method is built on image-to-image translation and conditional GANs  (e.g. [pix2pix], [CycleGAN]) rather than unconditional GANs (e.g. [vanilla GANs], [DCGAN]). Many conditional GAN methods have been shown to be able to produce photorealistic results at relatively high resolution (256x256 and 512x512 from [pix2pix] and [CycleGAN]).  The recent pix2pixHD paper on arXiv from NVIDIA [Wang et al. 2017] can even produce 2k photo-realistic images. Even recent unconditional GANs like progressive GANs [Karras et al. 2017] are able to produce 1k images.
(ii) Retaining details from original image: Our goal is to produce the perturbation rather than the final image: output = input + G(input). Details and textures are copied from the input image.
(iii) Resolution-independent architecture: Our model is fully convolutional and can be applied to input images with arbitrary sizes, similar to [pix2pix] and [CycleGAN].
Controlling the amount of distortion. We added more detailed description in the updated paper about how we control the amount of perturbation. Basically, we use parameter c within the hinge loss as shown in eq. (3) to allow users to specify the perturbation amount (epsilon). Note that AdvGAN can explicitly control the amount of perturbation since in the MNIST challenge, it is strictly required that the perturbation is bounded within 0.3 in terms of L-infinity.
So the competition results also show that we are able to bound the perturbation accurately so as to win the challenge.
Why are we interested in the diversity of adversarial examples? We have seen that ensemble adversarial training works better than adversarial training against FGS + rand [Florian et al. 2017]. This indicates that more diverse adversarial examples are needed to perform adversarial training as a defense.
In addition, exploring other diverse adversarial examples can help us better understand the space of adversarial examples. For these reasons, we are interested in how to produce diverse adversarial examples, but indeed, we have not made it the main goal of AdvGAN.
Reference
[Pix2pix] Isola, Phillip, et al. "Image-to-image translation with conditional adversarial networks." arXiv preprint arXiv:1611.07004 (2016).
[CycleGAN] Zhu, Jun-Yan, et al. "Unpaired image-to-image translation using cycle-consistent adversarial networks." arXiv preprint arXiv:1703.10593 (2017).
[Vanilla GAN] Goodfellow, Ian, et al. "Generative adversarial nets." Advances in neural information processing systems. 2014.
[DCGAN] Radford, Alec, Luke Metz, and Soumith Chintala. "Unsupervised representation learning with deep convolutional generative adversarial networks." arXiv preprint arXiv:1511.06434 (2015).
[Wang et al. 2017] Wang, Ting-Chun, et al. "High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs." arXiv preprint arXiv:1711.11585 (2017).
[Karras et al. 2017] Karras, Tero, et al. "Progressive growing of gans for improved quality, stability, and variation." arXiv preprint arXiv:1710.10196 (2017).
[Florian et al. 2017] Tramèr, Florian, et al. "Ensemble Adversarial Training: Attacks and Defenses." arXiv preprint arXiv:1705.07204 (2017).
Thanks for the valuable feedback.
Regarding the reviewer’s request about high resolution images, we have added (1) a quantitative experiment on attack success rates, (2) a user study on perceptual realism of the examples, and (3) additional qualitative examples, which demonstrate that AdvGAN can effectively generate high resolution adversarial examples. The details are as follows.
We generate 100 high resolution (299x299) adversarial examples under an L_infinity bound of 0.01 (pixel values are in range [0,1]). This competition provided a dataset compatible with ImageNet. We observe that the attack success rate of AdvGAN is 100%. Section 4.5 details the experiment settings.
In order to evaluate the perceptual realism of high resolution adversarial examples generated by AdvGAN,  we have added a human study in Section 4.5. In our study, participants chose AdvGAN’s adversarial examples as more realistic over the original images in 49.4% of the trials (matching the realism of the original images results in around 50%). This experiment shows that these high resolution AdvGAN adversarial examples are about as realistic as benign images.
In addition to the quantitative experiment and the user study, we include some high resolution adversarial examples in Figure 8.
adversarial training needs adversarial examples.
Hence fastly generating high-quality adversarial examples may accelerate the adversarial training.
This paper describes AdvGAN, a conditional GAN plus adversarial loss. AdvGAN is able to generate adversarial samples by running a forward pass on generator. The authors evaluate AdvGAN on semi-white box and black box setting.
AdvGAN is a simple and neat solution to for generating adversary samples. The author also reports state-of-art results.
Comment:
1. For MNIST samples, we can easily find the generated sample is a mixture of two digitals. Eg, for digital 7 there is a light gray 3 overlap. I am wondering this method is trying to mixture several samples into one to generate adversary samples. For real color samples, it is harder to figure out the mixture.
2. Based on mixture assumption, I suggest the author add one more comparison to other method, which is relative change from original image, to see whether AdvGAN is the most efficient model to generate the adversary sample (makes minimal change to original image).
We thank the reviewer for the thoughtful comments and suggestions. We plot the perturbation in Figure 3 (c) (d) and 4 (b) in the updated version. From these plots, we can see that AdvGAN’s perturbations (amplified by 10×) do not resemble images from CIFAR-10/ImageNet.
For fair comparison against other attacks, we limit the perturbation to 0.3 L_infinity distance for MNIST and 8 for CIFAR-10.
We have compared the attack success rate of adversarial examples by different methods under defenses in Tables 3 and 4 and show that AdvGAN can often achieve high attack success rate under the same perturbation budget compared to other methods.
We have also added another comparison on MNIST and CIFAR-10 with FGSM and optimization methods, showing the relative change from original image in Table 7 in the appendix.
From the table, we can see that AdvGAN adds comparable perturbation with CW and less perturbation compared with FGSM. As AdvGAN aims to generate photo realistic images with bounded perturbation instead of minimizing the perturbation as CW does, the perturbation added by AdvGAN is slightly higher compared to CW.
The paper proposes a way of generating adversarial examples that fool classification systems.
They formulate it for a blackbox and a semi-blackbox setting (semi being, needed for training their own network, but not to generate new samples).
The model is a residual gan formulation, where the generator generates an image mask M, and (Input + M) is the adversarial example.
The paper is generally easy to understand and clear in their results.
I am not awfully familiar with the literature on adversarial examples to know if other GAN variants exist. From this paper's literature survey, they dont exist.
So this paper is innovative in two parts:
- it applies GANs to adversarial example generation
- the method is a simple feed-forward network, so it is very fast to compute
The experiments are pretty robust, and they show that their method is better than the proposed baselines.
I am not sure if these are complete baselines or if the baselines need to cover other methods (again, not fully familiar with all literature here).
We thank the reviewer for the thoughtful comments and suggestions. As mentioned by the reviewer, for baselines comparison, the proposed AdvGAN is currently the best attack method  in Madry et al.’s MNIST Adversarial Examples Challenge (https://github.com/MadryLab/mnist_challenge), which includes many state-of-the-art attack methods.
In our updated version, we have also added another comparison on MNIST and CIFAR-10 with FGSM and optimization methods, showing the perturbation amount in Table 7 in the appendix.
Changes made in our revised version are listed as below：
- Added perturbation plot in Figure 3(c)(d) and Fig 4(b) for CIFAR-10 and ImageNet, respectively.
- Added a comparison between AdvGAN, FGSM, and optimization methods, comparing the relative changes from original images to adversarial examples (Table 7 in the appendix).
- Added a detailed description in section 3.2 about how to control distortion amount generated by AdvGAN.
- Added suggested references and updated section 2 to include more comprehensive analysis for related work.
- Updated the wording throughout the paper to make it more clear.
- Added human perceptual study and quantitative results for AdvGAN on high resolution images in Section 4.5
- Added additional generated adversarial examples by AdvGAN on high resolution images in Figure 8 in Appendix.
We would like to thank the reviewers again for the useful feedbacks and suggestions.

Meta-review:
role: PC
data: comment: The paper presents AdvGAN: a GAN that is trained to generate adversarial examples against a convolutional network. The motivation for this method is unclear: the proposed attack does not outperform simpler attack methods such as Carlini-Wagner attack. In white-box settings, a clear downside for the attacker is that it needs to re-train its GAN everytime the defender changes its convolutional network.

More importantly, the work appears preliminary. In particular, the lack of extensive quantitative experiments on ImageNet makes it difficult to compare the proposed approach to alternative attacks methods such as (I-)FGSM, DeepFool, and Carlini-Wagner. The fact that AdvGAN performs well on MNIST is nice, but MNIST should be considered for what it is: a toy dataset. If AdvGANs are, as the authors state in their rebuttal, fast and good at generating high-resolution images, then it should be straightforward to perform comprehensive experiments with AdvGANs on ImageNet (rather than focusing on a small number of images on a single target, as the authors did in their revision)?

Decision: Reject