Title: GENERATING ADVERSARIAL EXAMPLES WITH AD-VERSARIAL NETWORKS

Abstract: Deep neural networks (DNNs) have been found to be vulnerable to adversarial examples resulting from adding small-magnitude perturbations to inputs. Such adversarial examples can mislead DNNs to produce adversary-selected results. Different attack strategies have been proposed to generate adversarial examples, but how to produce them with high perceptual quality and more efficiently requires more research efforts. In this paper, we propose AdvGAN to generate adversarial examples with generative adversarial networks (GANs), which can learn and approximate the distribution of original instances. For AdvGAN, once the generator is trained, it can generate adversarial perturbations efficiently for any instance, so as to potentially accelerate adversarial training as defenses. We apply AdvGAN in both semi-whitebox and black-box attack settings. In semi-whitebox attacks, there is no need to access the original target model after the generator is trained, in contrast to traditional white-box attacks. In black-box attacks, we dynamically train a distilled model for the black-box model and optimize the generator accordingly. Adversarial examples generated by AdvGAN on different target models have high attack success rate under state-of-the-art defenses compared to other attacks. Our attack has placed the first with 92.76% accuracy on a public MNIST black-box attack challenge (M ądry et al., 2017b).

1. INTRODUCTION:

Deep Neural Networks (DNNs) have achieved great successes in a variety of applications ranging from image recognition (Krizhevsky et al., 2012; He et al., 2016) to speech processing (Hinton et al., 2012) and from robotics training (Levine et al., 2016) to medical diagnostics (Ciresan et al., 2012) . However, recent work has demonstrated that DNNs are vulnerable to adversarial perturbations (Szegedy et al., 2014; Goodfellow et al., 2015) . An adversary can add small-magnitude perturbations to inputs and generate adversarial examples to mislead DNNs. Such maliciously perturbed instances can cause the learning system to misclassify them into either a maliciously-chosen target class (in a targeted attack) or classes that are different from the ground truth (in an untargeted attack). Different algorithms have been proposed for generating such adversarial examples, such as the fast gradient sign method (FGSM) (Goodfellow et al., 2015) and optimization-based methods (Opt.) (Carlini & Wagner, 2017a; Liu et al., 2017) .

Most of the the current attack algorithms (Carlini & Wagner, 2017a; Liu et al., 2017) rely on optimization schemes with simple pixel space metrics, such as L ∞ distance from a benign image, to encourage visual realism. To generate more perceptually realistic adversarial examples, in this paper, we propose to train a feed-forward network to generate perturbations such that the resulting example must be realistic according to a discriminator network. We apply generative adversarial networks (GANs) (Goodfellow et al., 2014) to produce adversarial examples in both the semi-whitebox and black-box settings. As conditional GANs are capable of producing high-quality images (Isola et al., 2017) , we apply a similar paradigm to produce perceptually realistic adversarial instances. We name our method AdvGAN.

Note that in the previous white-box attacks, such as FGSM and optimization methods, the adversary needs to have white-box access to the architecture and parameters of the model all the time. However, by deploying AdvGAN, once the feed-forward network is trained, it can instantly pro-duce adversarial perturbations for any input instances without requiring access to the model itself anymore. We name this attack setting semi-whitebox.

To evaluate the effectiveness of our attack strategy AdvGAN, we first generate adversarial instances based on AdvGAN and other attack strategies on different target models. We then apply the stateof-the-art defenses to defend against these generated adversarial examples (Goodfellow et al., 2015; Tramèr et al., 2017a; M ądry et al., 2017a) . We evaluate these attack strategies in both semi-whitebox and black-box settings. We show that adversarial examples generated by AdvGAN can achieve a high attack success rate, potentially due to the fact that these adversarial instances appear closer to real instances compared to other recent attack strategies.

Our contributions are listed as follows.

• Different from the previous optimization-based methods, we train a conditional adversarial network to directly produce adversarial examples, which are both perceptually realistic and achieve state-of-the-art attack success rate against different target models. 

2. RELATED WORK:

Here we review recent work on adversarial examples and generative adversarial networks.

Adversarial Examples A number of attack strategies to generate adversarial examples have been proposed in the white-box setting, where the adversary has full access to the classifier (Szegedy et al., 2014; Goodfellow et al., 2015; Carlini & Wagner, 2017a; Moosavi-Dezfooli et al., 2015; Papernot et al., 2016; Biggio et al., 2013; Kurakin et al., 2016) . Goodfellow et al. propose the fast gradient sign method (FGSM), which applies a first-order approximation of the loss function to construct adversarial samples. Formally, given an instance x, an adversary generates adversarial example x A = x + η with L ∞ constraints in the untargeted attack setting as η = • sign(∇ x f (x, y)), where f (•) is the cross-entropy loss used to train the neural network f , and y represents the ground truth of x. Optimization based methods have also been proposed to optimize adversarial perturbation for targeted attacks while satisfying certain constraints (Carlini & Wagner, 2017a; Liu et al., 2017) . Its goal is to minimize the objective function as ||η||+λ f (x A , y), where ||•|| is an appropriately chosen norm function. However, the optimization process is slow and can only optimize perturbation for one specific instance each time. In contrast, our feed-forward network can produce perturbation for any instance. It achieves higher attack success rate against different defenses and performs much faster than the current attack algorithms. Independently from our work, feed-forward networks have been applied to generate adversarial perturbation (Baluja & Fischer, 2017) . However, Baluja & Fischer combine the re-ranking loss and an L 2 norm loss, aiming to constrain the generated adversarial instance to be close to the original one in terms of L 2 ; while we apply a deep neural network as a discriminator to help distinguish the instance with other real images to encourage the perceptual quality of the generated adversarial examples.

Black-box Attacks Current learning systems usually do not allow white-box accesses against the model for security reasons. Therefore, there is a great need for black-box attacks analysis. Most of the black-box attack strategies are based on the transferability phenomenon (Papernot et al., 2017) , where an adversary can train a local model first and generate adversarial examples against it, hoping the same adversarial examples will also be able to attack the other models. Many learning systems allow query accesses to the model. However, there is little work that can leverage query-based access to target models to construct adversarial samples and move beyond transferability. Papernot Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) have achieved visually appealing results in both image generation (Radford et al., 2015; Gulrajani et al., 2017; Berthelot et al., 2017) and manipulation (Zhu et al., 2016) settings. Recently, image-to-image conditional GANs have further improved the quality of synthesis results (Isola et al., 2017; Zhu et al., 2017) . We adopt a similar adversarial loss and image-to-image network architecture to learn the mapping from an original image to a perturbed output such that the perturbed image cannot be distinguished from real images in the original class. Different from prior work, we aim to produce output results that are not only visually realistic but also able to mislead target learning models.

3. GENERATING ADVERSARIAL EXAMPLES WITH ADVERSARIAL NETWORKS:

3.1 PROBLEM DEFINITION Let X ⊆ R n be the feature space, with n the number of features. Suppose that (x i , y i ) is the ith instance within the training set, which is comprised of feature vectors x i ∈ X , generated according to some unknown distribution x i ∼ P data , and y i ∈ Y the corresponding true class labels. The learning system aims to learn a classifier f : X → Y from the domain X to the set of classification outputs Y, where |Y| denotes the number of possible classification outputs. Given an instance x, the goal of an adversary is to generate adversarial example x A , which is classified as f (x A ) = y (untargeted attack), where y denotes the true label; or f (x A ) = t (targeted attack) where t is the target class. x A should also be close to the original instance x in terms of L 2 or other distance metric.

3.2. ADVGAN FRAMEWORK:

Figure 1 illustrates the overall architecture of AdvGAN, which mainly consists of three parts: a generator G, a discriminator D, and the target neural network f . Here the generator G takes the original instance x as its input and generates a perturbation G(x). Then x + G(x) will be sent to the discriminator D, which is used to distinguish the generated data and the original instance x. The goal of D is to encourage that the generated instance is indistinguishable with the data from its original class. To fulfill the goal of fooling a learning model, we first perform the white-box attack, where the target model is f in this case. f takes x + G(x) as its input and outputs its loss L adv , which represents the distance between the prediction and the target class t (targeted attack), or the opposite of the distance between the prediction and the ground truth class (untargeted attack).

The adversarial loss (Goodfellow et al., 2014) can be written as:1 

L GAN = E x log D(x) + E x log(1 -D(x + G(x))).

(1)

Here, the discriminator D aims to distinguish the perturbed data x + G(x) from the original data x.2 Note that the real data is sampled from the true class, so as to encourage that the generated instances are close to data from the original class.

The loss for fooling the target model f in a targeted attack is:

EQUATION

)

where t is the target class and f denotes the loss function (e.g., cross-entropy loss) used to train the original model f . The L f adv loss encourages the perturbed image to be misclassified as target class t. Here we can also perform the untargeted attack by maximizing the distance between the prediction and the ground truth, but we will focus on the targeted attack in the rest of the paper.

To bound the magnitude of the perturbation, which is a common practice in prior work (Carlini & Wagner, 2017a; Liu et al., 2017; Bartlett & Wegkamp, 2008) , we add a soft hinge loss on the L 2 norm as

EQUATION

where c denotes a user-specified bound. This can also stabilize the GAN's training, as shown in Isola et al. (2017) . Finally, our full objective can be expressed as

EQUATION

where α and β control the relative importance of each objective. Note that L GAN here is used to encourage the perturbed data to appear similar to the original data x, while L f adv is leveraged to generate adversarial examples, optimizing for the high attack success rate. We obtain our G and D by solving the minmax game arg min G max D L.

3.3. BLACK-BOX ATTACKS WITH ADVERSARIAL NETWORKS:

Static Distillation For black-box attack, we assume adversaries have no prior knowledge of training data or the model itself. In our experiments in Section 4, we randomly draw data that is disjoint from the training data of the black-box model to distill it, since we assume the adversaries have no prior knowledge about the training data or the model. To achieve black-box attacks, we first build a distilled network f based on the output of the black-box model b (Hinton et al., 2015) . Once we obtain the distilled network f , we carry out the same attack strategy as described in the white-box setting (see Equation ( 4)). Here, we minimize the following network distillation objective: Here we propose an alternative minimization approach to dynamically make queries and train the distilled model f and our generator G jointly. We perform the following two steps in each iteration. During iteration i: 4) and train the generator and discriminator based on a previously distilled model f i-1 . We initialize the weights

EQUATION

G i as G i-1 . G i , D i = arg min G max D L fi-1 adv + αL GAN + βL hinge (6)

2. Update f i given a fixed generator G i : First, we use f i-1 to initialize f i . Then, given the generated adversarial examples x + G i (x) from G i , the distilled model f i will be updated based on the set of new query results for the generated adversarial examples against the black-box model, as well as the original training images.

EQUATION

where we use both the original images x and the newly generated adversarial examples x + sG i (x) to update f .

In the experiment section, we compare the performance of both the static and dynamic distillation approaches and observe that simultaneously updating G and f produces higher attack performance. See Table 2 for more details.

4. EXPERIMENTAL RESULTS:

In this section, we first evaluate AdvGAN for both semi-whitebox and black-box settings on MNIST (LeCun & Cortes, 1998) and CIFAR-10 (Krizhevsky et al., 2014) . We also perform a semi-whitebox attack on the ImageNet dataset (Deng et al., 2009) . We then apply AdvGAN to generate adversarial examples on different target models and test the attack success rate for them under the state-of-theart defenses and show that our method can achieve higher attack success rates compared to other existing attack strategies. We generate all adversarial examples for different attack methods based on the under L ∞ bound of 0.3 on MNIST and 8 on CIFAR-10, for a fair comparison.

In general, as shown in Implementation Details Our code and models will be available upon publication. We adopt a similar architecture from image-to-image translation literature (Isola et al., 2017; Zhu et al., 2017) .

In particular, we use the architecture of generator G from Johnson et al. (2016) , and our discriminator D's architecture is similar to model C for MNIST and ResNet-32 for CIFAR-10. We apply the loss in Carlini & Wagner (2017c) as our loss

L f adv = max(max i =t f (x A ) i -f (x A ) t , κ

), where t is the target class, and f represents the target network in the semi-whitebox setting and the distilled model in the black-box setting. We set the confidence κ = 0 for both Opt. and AdvGAN. We use Adam as our solver (Kingma & Ba, 2014) , with a batch size of 128 and a learning rate of 0.001. For GANs training, we use the least squares objective proposed by LSGAN (Mao et al., 2016) , as it has been shown to produce better results with more stable training. On the diagonal, the original images are shown.

4.2. ADVGAN IN BLACK-BOX SETTING:

In this section, we evaluate the performance of AdvGAN for the black-box attack. Our black-box attack here is based on the dynamic distillation strategy. We construct a local model to distill model f , and we select the architecture of Model C as our local model. Note that we randomly select a subset of instances disjoint from the training data of AdvGAN to train the local model; that is, we assume the adversaries do not have any prior knowledge of the training data or the model itself.

With the dynamic distillation strategy, the adversarial examples generated by AdvGAN achieve an attack success rate, above 90% for MNIST and 80% for CIFAR-10, compared to 30% and 10% with the static distillation approach, as shown in Table 2 .

We apply AdvGAN to generate adversarial examples for the same instance targeting different classes on MNIST and randomly select some instances to show in Figure 2 (d)-(f) . By comparing with the pristine instances on the diagonal, we can see that these adversarial instances can achieve high perceptual quality as the original digits. Specifically, the original digit is somewhat highlighted by adversarial perturbations, which implies a type of perceptually realistic manipulation. Figure 3 (b) shows similar results for adversarial examples generated on CIFAR-10. These adversarial instances appear photo-realistic compared with the original ones on the diagonal. We show additional results in Appendix C.

4.3. ATTACK EFFECTIVENESS UNDER DEFENSES:

Facing different types of attack strategies, various defenses have been provided. Among them, different types of adversarial training methods are the most effective. Goodfellow et al. (2015) first propose adversarial training as an effective way to improve the robustness of DNNs, and Tramèr Threat Model As shown in the literature, most of the current defense strategies are not robust when attacking against them (Carlini & Wagner, 2017b; He et al., 2017) . Here we consider a weaker threat model, where the adversary is not aware of the defenses and directly tries to attack the original learning model, which is also the first threat model analyzed in Carlini & Wagner (2017b) . In this case, if an adversary can still successfully attack the model, it implies the robustness of the attack strategy. Under this setting, we first apply different attack methods to generate adversarial examples ) (Goodfellow et al., 2015) , ensemble adversarial training (Ensemble) (Tramèr et al., 2017b) , and iterative training (Iter. Adv.) (M ądry et al., 2017a) . 5 We evaluate the effectiveness of these attacks against these defended models. In Table 3 , we show that the attack success rate of adversarial examples generated by AdvGAN on different models is higher than those of the fast gradient sign method (FGSM) and optimization methods (Opt.) (Carlini & Wagner, 2017a In addition, we apply AdvGAN to the MNIST challenge (M ądry et al., 2017b) . Among all the methods, for white-box attack we achieve 88.93% accuracy on the published local model as shown in Table 5 . For the reported black-box attack, we achieved the accuracy as 92.76%, outperforming all other state-of-the-art attack strategies. To understand the adversarial perturbation pattern better, we plot out corresponding perturbations (amplified by a factor of 10) for CIFAR-10 in Figure 3 (c) and (d) and ImageNet in Figure 4 (b) .

From the visualization of perturbation, it shows that the perturbations do not resemble anything in particular about the original image or the target class. Although training AdvGAN exposes it to realistic instances, the perturbations it generates do not simply interpolate towards an example of the target class. In Figure 8 in the appendix, we show the original images on the left with the correct label, and we show adversarial examples generated by AdvGAN on the right with the target label.

4.5. HIGH RESOLUTION ADVERSARIAL EXAMPLES ANALYSIS:

Human Perceptual Study. We validate the realism of AdvGAN's adversarial examples with a user study on Amazon Mechanical Turk (AMT). We use 100 pairs of original images and adversarial examples (generated as described above) and ask workers to choose which image of a pair is more visually realistic.

Our study follows a protocol from Zhang et al. (2016) and Isola et al. (2017) , where a worker is shown a pair of images for 2 seconds, then the worker has unlimited time to choose. We limit each worker to at most 20 of these tasks. We collected 500 choices, about 5 per pair of images, from 50 workers on AMT.

The AdvGAN examples were chosen as more realistic than the original image in 49.4% ± 1.96% of the tasks (random guessing would result in about 50%). This result show that these high-resolution

AdvGAN adversarial examples are about as realistic as benign images. 

5. CONCLUSION:



:

For simplicity, we denote the Ex ≡ E x∼P data (x)

Note that we only use the generator to produce the perturbation G(x).

https://github.com/tensorflow/models/blob/master/research/ResNet/ResNet_model.py

https://github.com/MadryLab/cifar10_challenge/blob/master/model.py

Each ensemble adversarially trained model is trained using (i) pristine training data, (ii) FGSM adversarial examples generated for the current model under training, and (iii) FGSM adversarial examples generated for naturally trained models of two architectures different from the model under training.

https://github.com/tensorflow/cleverhans/tree/master/examples/nips17_ adversarial_competition/dataset

https://github.com/jcjohnson/fast-neural-style.

Figures and Tables:

Figure 1: Overview of AdvGAN

where f (x) and b(x) denote the output from the distilled model and black-box model respectively for the given training image x, and H denotes the commonly used cross-entropy loss. By optimizing the objective over all the training images, we can obtain a model f which behaves very close to the black-box model b. We then carry out the attack on the distilled network.Note that unlike training the discriminator D, where we only use the real data from the original class to encourage that the generated instance is close to its original class, here we train the distilled model with data from all classes.Dynamic DistillationOnly training the distilled model with all the pristine training data is not enough, since it is unclear how close the black-box and distilled model perform on the generated adversarial examples, which have not appeared in the training set before.

Figure 2: Adversarial examples generated from the same original image to different targets by Adv-GAN on MNIST with semi-whitebox attack, (a), (b), and (c), and black-box attack, (c), (d), and (e). On the diagonal, the original images are shown.

Figure 3: Adversarial examples generated by AdvGAN on CIFAR-10 for (a) semi-whitebox attack and (b) black-box attack. Image from each class is perturbed to other different classes. On the diagonal, the original images are shown. The corresponding perturbations (amplified by 10×) are shown in (c) and (d).

Figure 4: Adversarial examples (a) generated by AdvGAN on ImageNet in the semi-whitebox setting, which are classified as (from left to right) poodle, ambulance, basketball, and electric guitar. Corresponding perturbations are visualized in (b).

Figure 8: Examples from an ImageNet-compatible set. Left: original image; right: adversarial image generated by AdvGAN against Inception_v3.

• We show that AdvGAN can attack black-box models by training a distilled model. We propose to dynamically train the distilled model with query information and achieve high black-box attack success rate and targeted black-box attack, which is difficult to achieve for transferability-based black-box attacks. • We use the state-of-the-art defense methods to defend against adversarial examples and show that AdvGAN achieves much higher attack success rate under current defenses. • We apply AdvGAN on M ądry et al.'s MNIST challenge (2017a) and achieve 88.93% accu-

Table content: <table/>

Comparison with the state-of-the-art attack methods. Run time is measured for generating 1,000 adversarial instances during test time. Opt. represents the optimization based method, and Trans. denotes black-box attacks based on transferability. Update G i given a fixed network f i-1 : We follow the white-box setting (see Equation

Table content: <table><tr><td/><td colspan="3">FGSM Opt. Trans. AdvGAN</td></tr><tr><td>Run time</td><td>0.06s &gt;3h</td><td>-</td><td>&lt;0.01s</td></tr><tr><td>Targeted Attack</td><td/><td>Ens.</td><td/></tr><tr><td>Black-box Attack</td><td/><td/><td/></tr><tr><td>1.</td><td/><td/><td/></tr></table>

AdvGAN has several advantages over other white-box and blackbox attacks. For instance, regarding computation efficiency, AdvGAN performs much faster than others even including the efficient FGSM, although AdvGAN needs extra training time to train the generator. All these strategies can perform targeted attack except transferability based attack, although the ensemble strategy can help to improve. Besides, FGSM and optimization methods can only perform white-box attack, while AdvGAN is able to attack in semi-whitebox setting.

Table content: <table/>

Accuracy of different models on pristine data, and the attack success rate of adversarial examples generated against different models by AdvGAN on MNIST and CIFAR-10. p: pristine test data; w: semi-whitebox attack; b-D: black-box attack with dynamic distillation strategy; b-S: black-box attack with static distillation strategy. Models Used in the Experiments For MNIST, in all of our experiments, we generate adversarial examples for three models whose architectures are shown in Appendix A. Models A and B are used in Tramèr et al. (2017b), which represent different architectures. Model C is the target network architecture used in(Carlini & Wagner, 2017a)  for evaluating optimization based strategy. For CIFAR-10, we select ResNet-32 and Wide ResNet-34(He et al., 2016;Zagoruyko & Komodakis, 2016) for our experiments. Specifically, we use a 32-layer ResNet implemented in TensorFlow 3 and Wide ResNet derived from the variant of "w32-10 wide."4 We show the classification accuracy of pristine MNIST and CIFAR-10 test data (p) and attack success rate of adversarial examples generated by AdvGAN on different models in Table2.In addition, we analyze the attack success rate based on different loss functions on MNIST. Under the same bounded perturbations (0.3), if we replace the full loss function in (4) with L = ||G(x)|| 2 + L f adv , which is similar to the objective used inBaluja & Fischer (2017), the attack success rate becomes 86.2%. If we replace the loss function with L = L hinge + L f adv , the attack success rate is 91.1%, compared to that of AdvGAN, 98.3%.

Table content: <table><tr><td/><td/><td>MNIST</td><td/><td/><td>CIFAR-10</td></tr><tr><td>Model</td><td>A</td><td>B</td><td>C</td><td colspan="2">ResNet-32 Wide ResNet-34</td></tr><tr><td>Accuracy (p)</td><td colspan="3">98.97% 99.17% 99.09%</td><td>92.41%</td><td>95.01%</td></tr><tr><td>Attack Success Rate (w)</td><td>97.9%</td><td>97.1%</td><td>98.3%</td><td>94.71%</td><td>99.30%</td></tr><tr><td>Attack Success Rate (b-D)</td><td>93.4%</td><td>90.1%</td><td>94.02%</td><td>78.47 %</td><td>81.81%</td></tr><tr><td>Attack Success Rate (b-S)</td><td>30.7%</td><td>66.63%</td><td>87.3%</td><td>10.3%</td><td>13.3%</td></tr></table>

). -box Attack For AdvGAN, we use model B as the black-box model and train a distilled model to perform black-box attack against model B and report the attack success rate in Table 4. For the black-box attack comparison purpose, transferability based attack is applied for FGSM and optimization-based methods (Opt.). We use FGSM and optimization-based methods (Opt.) to attack model A on MNIST, and we use these adversarial examples to test on model B and report the corresponding classification accuracy. We can see that the adversarial examples generated by the black-box AdvGAN consistently achieve much higher attack success rate compared with other attack methods. For CIFAR-10, we use ResNet as black-box model and train a distilled model to perform black-box attack against ResNet. To evaluate black-box attack for optimization method and FGSM, we use adversarial examples generated by attacking Wide ResNet and test them on ResNet to report black-box attack results for these two methods.

Table content: <table><tr><td>Black</td></tr></table>

Attack success rate of adversarial examples generated by AdvGAN in semi-whitebox setting, and other white-box attacks under defenses on MNIST and CIFAR-10.

Table content: <table><tr><td>Data</td><td>Model</td><td>Defense</td><td>FGSM</td><td>Opt.</td><td>AdvGAN</td></tr><tr><td/><td/><td>Adv.</td><td>4.3%</td><td>4.6%</td><td>8.0%</td></tr><tr><td/><td>A</td><td>Ensemble</td><td>1.6%</td><td>4.2%</td><td>6.3%</td></tr><tr><td/><td/><td>Iter.Adv.</td><td>4.4%</td><td>2.96%</td><td>5.6%</td></tr><tr><td/><td/><td>Adv.</td><td>6.0%</td><td>4.5%</td><td>7.2%</td></tr><tr><td>MNIST</td><td>B</td><td>Ensemble</td><td>2.7%</td><td>3.18%</td><td>5.8%</td></tr><tr><td/><td/><td>Iter.Adv.</td><td>9.0%</td><td>3.0%</td><td>6.6%</td></tr><tr><td/><td/><td>Adv.</td><td>2.7%</td><td>2.95%</td><td>18.7%</td></tr><tr><td/><td>C</td><td>Ensemble</td><td>1.6%</td><td>2.2%</td><td>13.5%</td></tr><tr><td/><td/><td>Iter.Adv.</td><td>1.6%</td><td>1.9%</td><td>12.6%</td></tr><tr><td/><td/><td>Adv.</td><td colspan="2">13.10% 11.9%</td><td>16.03%</td></tr><tr><td/><td>ResNet</td><td colspan="3">Ensemble. 10.00% 10.3%</td><td>14.32%</td></tr><tr><td>CIFAR</td><td/><td>Iter.Adv Adv.</td><td>22.8% 5.04%</td><td>21.4% 7.61%</td><td>29.47% 14.26%</td></tr><tr><td/><td>Wide ResNet</td><td>Ensemble</td><td>4.65%</td><td>8.43%</td><td>13.94 %</td></tr><tr><td/><td/><td>Iter.Adv.</td><td colspan="2">14.9% 13.90%</td><td>20.75%</td></tr></table>

Attack success rate of adversarial examples generated by different black-box adversarial strategies under defenses on MNIST and CIFAR-10

Table content: <table><tr><td/><td/><td>MNIST</td><td/><td/><td>CIFAR-10</td><td/></tr><tr><td>Defense</td><td colspan="4">FGSM Opt. AdvGAN FGSM</td><td>Opt.</td><td>AdvGAN</td></tr><tr><td>Adv.</td><td>3.1%</td><td>3.5%</td><td>11.5%</td><td colspan="2">13.58% 10.8%</td><td>15.96%</td></tr><tr><td>Ensemble</td><td>2.5%</td><td>3.4%</td><td>10.3%</td><td>10.49%</td><td>9.6%</td><td>12.47%</td></tr><tr><td>Iterative Adv.</td><td>2.4%</td><td>2.5%</td><td>12.2%</td><td colspan="2">22.96% 21.70%</td><td>24.28%</td></tr></table>

Accuracy of the MadryLab public model under different attacks in white-box setting. The AdvGAN here achieved the best performance.

Table content: <table><tr><td>Method</td><td colspan="2">Accuracy (xent loss) Accuracy (cw loss)</td></tr><tr><td>FGSM</td><td>95.23%</td><td>96.29%</td></tr><tr><td>PGD</td><td>93.66%</td><td>93.79%</td></tr><tr><td>Opt</td><td>-</td><td>91.69%</td></tr><tr><td>AdvGAN</td><td>-</td><td>88.93%</td></tr><tr><td colspan="2">4.4 ADVERSARIAL PERTURBATION ANALYSIS.</td><td/></tr></table>

In the following experiments, we select toy poodle as our target label for all images. We select 100 benign images from the DEV set of the NIPS 2017 targeted adversarial attack competition.6 This competition provided a dataset compatible with ImageNet. We generate adversarial examples (299×299 pixels) under an L ∞ perturbation bound of 0.01 (pixels values are in the range ∈ [0, 1]) for the Inception_v3 model, whose input size is 299×299. The details of architectures for generator and discriminator we used are listed in Appendix D.

Table content: <table><tr><td/><td colspan="5">Table 6: Parameters of generated high resolution adversarial examples</td></tr><tr><td>Dateset</td><td>Model</td><td colspan="4">Target Label Resolution L ∞ bound Attack Success Rate</td></tr><tr><td colspan="2">ImageNet Inception_v3</td><td>toy poodle</td><td>299 × 299</td><td>0.01</td><td>100%</td></tr></table>

In this paper, we propose AdvGAN to generate adversarial examples using generative adversarial networks (GANs). In our AdvGAN framework, once trained, the feed-forward generator can produce adversarial perturbations efficiently. It can also perform both semi-whitebox and black-box attacks with high attack success rate. In addition, when we apply AdvGAN to generate adversarial instances on different models without knowledge of the defenses in place, the generated adversarial examples can attack the state-of-the-art defenses with higher attack success rate than examples generated by the competing methods. This property makes AdvGAN a promising candidate for improving adversarial training defense methods. The generated adversarial examples produced by AdvGAN preserve high perceptual quality due to GANs' distribution approximation property.

Table content: <table/>

Peer review (iclr2018):
 I thank the authors for the thoughtful response and rebuttal. The authors have substantially updated their manuscript and improved the presentation. Re: Speed. I brought up this point because this was a bulleted item in the Introduction in the earlier version of the manuscript. In the revised manuscript, this bullet point is now removed. I will take this point to be moot. Re: High resolution. The authors point to recent GAN literature that provides some first results with high resolution GANs but I do not see quantitative evidence in the high resolution setting for this paper. (Figure 4 provides qualitative examples from ImageNet but no quantitative assessment.) Because the authors improved the manuscript, I upwardly revised my score to *Ok but not good enough - rejection*. I am not able to accept this paper because of the latter point. ========================== The authors present an interesting new method for generating adversarial examples. Namely, the author train a generative adversarial network (GAN) to adversarial examples for a target network. The authors demonstrate that the network works well in the semi-white box and black box settings. The authors wrote a clear paper with great references and clear descriptions. My primary concern is that this work has limited practical benefit in a realistic setting. Addressing each and every concern is quite important: 1) Speed. The authors suggest that training a GAN provides a speed benefit with respect to other attack techniques. The FGSM method (Goodfellow et al, 2015) is basically 1 inference operation and 1 backward operation. The GAN is 1 forward operation. Granted this results in a small difference in timing 0.06s versus 0.01s, however it would seem that avoiding a backward pass is a somewhat small speed gain. Furthermore, I would want to question the practical usage of having an *even faster* method for generating adversarial examples. What is the reason that we need to run adversarial attacks *even faster*? I am not aware of any use-cases, but if there are some, the authors should describe the rationales at length in their paper. 2) High spatial resolution images. Previous methods, e.g. FGSM, may work on arbitrarily sized images. At best, GANs generate reasonable images that are lower resolutions (e.g. < 128x128). Building GAN*s that operate above-and-beyond moderate spatial resolution is an open research topic. The best GAN models for generating high resolution images are difficult to train and it is not clear if they would work in this setting. Furthermore, images with even higher resolutions, e.g. 512x512, which is quite common in ImageNet, are difficult to synthesizes using current techniques. 3) Controlling the amount of distortion. A feature of previous optimization based methods is that a user may specify the amount of perturbation (epsilon). This is a key feature if not requirement in an adversarial perturbation because a user might want to examine the performance of a given model as a function of epsilon. Performing such an analysis with this model is challenging (i.e. retraining a GAN) and it is not clear if a given image generated by a GAN will always achieve a given epsilon perturbation/ On a more minor note, the authors suggest that generating a *diversity* of adversarial images is of practical import. I do not see the utility of being able to generate a diversity of adversarial images. The authors need to provide more justification for this motivation.