AT-GAN: An Adversarial Generative Model for Non-constrained Adversarial Examples
Authors: Xiaosen Wang
Kun He
Chuanbiao Song
Liwei Wang
John E. Hopcroft
Venue:  (2021)
Keywords: adversarial examples
adversarial attack
generation-based attack
adversarial generative model
non-constrained adversarial examples

Abstract: With the rapid development of adversarial machine learning, numerous adversarial attack methods have been proposed. Typical attacks are based on a search in the neighborhood of input image to generate a perturbed adversarial example. Since 2017, generative models are adopted for adversarial attacks, and most of them focus on generating adversarial perturbations from input noise or input image. Thus the output is restricted by input for these works. A recent work targets unrestricted adversarial example using generative model but their method is based on a search in the neighborhood of input noise, so actually their output is still constrained by input. In this work, we propose AT-GAN (Adversarial Transfer on Generative Adversarial Net) to train an adversarial generative model that can directly produce adversarial examples. Different from  previous works, we aim to learn the distribution of adversarial examples so as to generate semantically meaningful adversaries. AT-GAN achieves this goal by first learning a generative model for real data, followed by transfer learning to obtain the desired generative model. Once trained and transferred, AT-GAN could generate adversarial examples directly and quickly for any input noise, denoted as non-constrained adversarial examples. Extensive experiments and visualizations show that AT-GAN can efficiently generate diverse adversarial examples that are realistic to human perception, and yields higher attack success rates against adversarially trained models.

 

Under review as a conference paper at ICLR 2021
AT-GAN: AN ADVERSARIAL GENERATIVE MODEL FOR NON-CONSTRAINED ADVERSARIAL EXAMPLES
Anonymous authors Paper under double-blind review
ABSTRACT
With the rapid development of adversarial machine learning, numerous adversarial attack methods have been proposed. Typical attacks are based on a search in the neighborhood of input image to generate a perturbed adversarial example. Since 2017, generative models are adopted for adversarial attacks, and most of them focus on generating adversarial perturbations from input noise input image. Thus the output is restricted by input for these works. A recent work targets "unrestricted adversarial example" using generative model but their method is based on search in the neighborhood of input noise, so actually their output is still constrained by input. In this work, propose AT-GAN (Adversarial Transfer Generative Adversarial Net) to train an adversarial generative model that can directly produce adversarial examples. Different from previous works, we aim learn the distribu- tion of adversarial examples so as to generate semantically meaningful adversaries. AT-GAN achieves this goal by first learning a generative model for real data, fol- lowed by transfer learning to obtain the desired generative model. Once trained and transferred, AT-GAN could generate adversarial examples directly and quickly for any input noise, denoted as non-constrained adversarial examples. Extensive experiments and visualizations show that AT-GAN can efficiently generate diverse adversarial examples that are realistic to human perception, and yields higher attack success rates against adversarialyy trained models.
INTRODUCTION
In recent years. Deep Neural Networks (DNNs) have been found vulnerable to adversarial exam- ples (Szegedy et al. 2014), which are well-crafted samples with tiny perturbations imperceptible to humans but can fool the learning models. Despite the great success of the deep learning empow- ered applications, many of them are safety-critical, for example under the scenario of self-driving cars (Eykholt et al. 2018 Cao et al. 2019), raising serious concerns in academy and industry.
Numerous works of adversarial examples have been developed on adversarial attacks (Goodfellow 2015 Carlini & Wagner, 2017 Madry et al.) 2018). adversarial defenses (Goodfellow et al. Kurakin et | [217]] Song et al. 2019) and exploring the property of adversarial examples (He 2018 Shamir et al. 2019). For adversarial attacks, most studies focus on the perturbation-baxee adversarial examples constrained by input images, which is also the generally accepted conception of adversarial examples. Generative models are also adopted recently generate adversarial pertur- bations from an input noise (Reddy Mopuri et al. 2018 Omid et al. 2018)) or from a given image (Xiao et al. 2018 Bai et al. 2020), and such perturbations are added the original image craft adversarial examples. Song et al. (2018) propose to search neighborhood noise around the input noise of Generative Adversarial Net (GAN) (Goodfellow et al. 2014) such that the output is an adversarial example, which they denoted as unrestricted adversarial example as there is no original image in their method. However, their output is still constrained by the input noise, and the search time-consuming.
et al.
2013
al.
|
In this work, we propose an adversarial generative model called AT-GAN (Adversarial Transfer on Generative Adversarial Net), which aims to learn the distribution of adversarial examples. Unlike previous works that constrain the adversaries in the neighborhood of input image or input noise, including the prominent work of SSng al. (2018) that searches over the neighborhood of the input noise of a pre-trained GAN in order to find a noise whose output image is misclassified by the target classifier, AT-GAN is adversarial generative model that could produce semantically meaningful

---

Under review as a conference paper at ICLR 2021
adversarial examples directly from any input noise, and we call such examples the non-constrained adversarial examples.
Specifically, first develop a normal GAN to learn the distribution of benign data so that it can produce plausible images that the classifier and a human oracle will classify in the same way. Then transfer the pre-trained GAN into adversarial GAN called AT-GAN that can fool the target classifier while being still well recognized by the human oracle. AT-GAN a conditional GAN that has learned to estimate the distribution of adversarial examples for the target classifier, so AT-GAN can directly generate adversarial examples from any random noise, leading to high diversity and efficiency.
We implement AT-GAN by adopting AC-GAN (Odena et al. 2017) and WGAN-GP (Gulrajani et al. 2017) in the pre-training stage, then do transfer learning for the adversary generation. Here we develop AT-GAN on three benchmark datasets, namely MNIST, Beshion-MNIST and CelebA, and apply typical defense methods to compare AT-GAN with existing bearch-based attacks. Empirical results show that the non-constrained adversarial examples generated by AT-GAN yield higher attack success rates, and state-of-the-ar adversarially trained models exhibit little robustness against AT- GAN, indicating the high diversity of our adversaries. In addition, AT-GAN, as generation-based adversarial attack, is more efficient than the search-based adversarial attacks.
Note that all conditional GANs that can craft realistic examples could be used for the implementation of AT-GAN. For another demonstration, we adopt StyleGAN2-ada (Karras et al.] (2020a) and develop AT-GAN on CIFAR-10 benchmark dataset using wide ResNet w32-10 (Zagoruyko & Komodakis 2016) as the target classifier. Empirical results show that AT-GAN can produce plausible adversarial images, and yield higher attack success rates on the adversarially trained models.
2 PRELIMINARIES
In this section, we provide definitions on several types of adversarial examples and adversarial attacks, and give brief overview of adversarial attacks using GAN. Other related works on typical adversarial
attacks and defenses (Goodfellow et al., 2013
Madry et al., 2018 Tramer et al.] (018). as well as
some typical GANs (Goodfellow et al., 2014 Radford et al. 2016 Odena et 2017
Arjovsky
et al.
2017
Gulrajani et al. 2017) are introduced in Appendix
2.1
DEFIITTIONS ON ADVERSARIES
Let X' be the set of all digital images under consideration for learning task, ) E R be the output label space and P= E IR' be an arbitrary probability distribution (e.g. Gaussian distribution) where m the dimension of Pa- A deep learning classifier f X' + y takes an image X' and predicts its label f()). Suppose ps and Pado are the distributions of benign images and adversarial examples, respectively. Assume we have an oracle classifier o: X - )/ which could always predict the correct label for any image x E X, we define several types of adversarial examples as follows.
For perturbation-basee adversarial examples (Szegedy et al. 2014, Goodfellow al. 2015 Moosavi Deztooli et al. 2016)) tiny perturbations are added to the input images, which are imperceptible to humans but can cause the target classifier to make wrong predictions.
Definition 1. Perturbation-basee Adversarial Examples. Given a subset (trainset testset) images T C X and a small constant >> O, the perturbation-basel adversarial examples can be defined as: Ap- {Fado 3x E T, || Zadvlll
f # o(Iadv) f(z) = o(z)).
Song et al. (2018) define new type of adversarial examples called unrestricted adversarial examples, which is not related to the subset (trainset or testset) images, by adding adversarial perturbation to the input noise of a mapping, such GAN, so that the output of the perturbed noise is an adversary to the target classifier.
Definition 2. Unrestricted Adversarial Examples. Given a mapping G from 2 ~ p. to G(z, y) ~ pe-
where pe is an approximated distribution of Pa. and small constant e 0, the unrestricted
adversarial examples can be defined as: A,, = [G( (2=","000 yy) E X|32 ~ 22,2" ~ pa. || 2n Ilp < (G(=", ya)) # oG((", ,y)) f(G(z, v=)) ((((z y.)) y.} where ys the source label.
2

---

Under review as a conference paper at ICLR 2021
In this work, we train conditional GAN to learn the distribution of adversarial examples and output the corresponding adversary directly from any imput noise. To clarify the difference with Song et al. (2018). we call our generated adversaries the non-constrained adversarial examples.
Definition 3. Non-constrained Adversarial Examples. If there is a mapping G* from Ps to (² ~ qo. where qo is an approximated distribution of Pados the non-constrained adversarial examples can be defined as An {G* (2, ys) &f (2, (2, # (z yo) where ya is the source label.
Here we need to find a mapping G*, e.g. generative model, such that for z ~ Pas Gz, y) is an image in X' and the output distribution is an approximated distribution of Podu. for example using the Kullback- divergence (Kullback & Leibler 1951). K L (P c for a small constant c.
In summary, perturbation-basse adversarial examples are based on perturbing an image E X. and unrestricted adversarial examples (Song et al.] (018)) perturbs an input noise ~ P= for an existing mapping G. Most perturbation-bace adversarial attacks and [Song et al. (2018) fall into the search-bused adversarial attack.
Definition 4. Search-based Adversarial Attack. Given an input vector v V (either benign image or random vector 2), the search-base based ad adversarial attack searches a vector e' || d'Ilp where of leads an adversarial example for the target classifier.
In contrast, non-constrained adversarial examples are more generalized so that need learn mapping G" such that for any input noise sampled from distribution pz, the output is an adversarial image. Such a mapping to be leamned is called adversarial generative model, and our method falls into the generation-base
adversarial attack.
Definition 5. Generation Adversarial Attack. Given an input vector E V (either benign im- age random vector 2), the generation-based adversarial attack generates adversarial perturbation or adversarial example directly from v, usually adopting generative models.
2.2 GENERATIVE MODELS FOR ADVERSARIAL ATTACK
Generative models have been adopted for adversarial attack in recent works (Baluja & Fischer 2017). Reddy Mopuri et al. (2018) propose Network for Adversary Generation (NAG) that models the distribution of adversarial perturbations for target classifier so that their NAG can craft adversarial perturbations from any given random noise, which will be added to the natural image to fool the target classifier. Omid et al. (2018) propose to generate universal or image-dependent adversarial perturbations using U-Net (Ronneberger et al. 2015) or ResNet Generator (He et al. (016) from any given random noise. Xiao et al. (2018) propose to train AdvGAN that takes an original image as the input and generate adversarial perturbation for the input to craft adversarial example. |Bii et al. (2020) further propose AI-GAN that adopts projected gradient descent (PGD) (Madry et al. 2018)) in the training stage to train GAN to generate target adversarial perturbation for the input image and target class. The above attack methods all fall into the generation- based adversarial attack, and their crafted examples fall into the perturbation-basee
adversarial examples. Another recent work called PS-GAN (Liu et al.) 2019) pre-processes an input seed patch (a small image) to adversarial patch that will be added to a natural image craft an adversarial example, and an attention model is used to locate the attack area on the natural image.
Different from the above methods that generate adversarial perturbations or patches, Song al. (2018) propose to search a random noise 2" around the input noise z of AC-GAN (Odena et al. (017) such that the corresponding output of AC-GAN an adversarial example for the target classifier. Their method falls into the search-based adversarial attack, and their crafted examples fall into the unrestricted adversarial examples as there is no original image in their method.
AT-GAN falls into the generation- based adversarial attack, and the crafted examples fall into the non-constrained adversarial examples. To clearly distinguish our work, we highlight the differences with most related works follows:
NAG, AdvGAN and AI-GAN vs. AT-GAN. NAG (Reddy Mopuri et al. 2018) AdvGAN (iiao
et al. 2018) and AI-GAN (Bai et al. 2020) focus crafting adversarial perturbations by GANs. NAG takes random noise as input and crafts image-agnostic adversarial perturbation. AdvGAN and AI-GAN both use natural images inputs, and generate the corresponding adversarial perturbations
3

---

Under review as a conference paper at ICLR 2021
Goal: Estimete the distribution of adversarial examples
qo (Zade Mtrue)
Pade (Hadd [Strase)
m (Fado Nove Pose (Paalo Meree)
Stage Transter By to estimate @o
||z - sawill
Ice stauelyy <
Pads (Frudi Dirue) should be close p(
mán K (pp Miree (1(pp((())))
Po (=lgtree)
p(t/)trnu)
Stage
Estimato the distribution of benign data
Figure 1:
Estimating the distribution of adversarial examples qo in two stages: 1) estimate the distribution of benign data pa- 2) transfer po to estimate qo-
for the input image. AI-GAN uses adversarial examples generated by PGD for the training. In contrast, AT-GAN does not use any natural image as the input, and generates adversarial examples directly from any random noise. Further, compared with AI-GAN, we do not use any adversarial examples for the training.
Song's vs. AT-GAN. Song's method (Song et al. 2018) searches over the neighborhood of the imput noise for the pre-trained AC-GAN in order to find noise whose output image is misclassified by the target classifier. They define such adversaries as the unrestricted adversarial examples, however, their adversaries are still constrained by the original input noise. Their method is essentially based on search, while AT-GAN is trained as an adversarial generative model, and our output is not constrained by any neighborhood.
AT-GAN: AN ADVERSARIAL GENERATIVE MODEL
Here we first introduce the estimation on the distribution of adversarial examples, then propose the AT-GAN framework, a generation-bate adversarial attack for crafting non-constrained adversarial examples. Further analysis is provided that AT-GAN could learn the adversary distribution.
3.1 ESTIMATING THE ADVERSARIAL DISTRIBUTION
In order generate non-constrained adversarial examples, we need to estimate the distribution of adversarial examples Pado (Tadv ( where ytrue is the true label. Given the parameterized estimated distribution of adversarial examples go (o(I/trre)) we can define the estimation problem as:
qe- (Zade (Mrue)) arg min K L(go (Fadul (Btrue) (ltree)l Po (ZadvlBtruel) BEn
where / indicates trainable parameters and f the parameter space.
It is hard to calculate equation I directly as Padv (Tade [Brae) is unknown. Inspired by the perturbation- based adversarial examples, as shown Figure [] we postulate that for each adversarial example Tadus there exists some benign examples æ where -adull l < c. In other words, Pado (Fadu lVtrue) close to p( (x/(ttrue) to some extent and we can obtain p( by Bayes' theorem, p(Il =
p(yerne pp)) where P(ytrue (rr), p(x: and p(Utrae) can be obtained directly from the trainset. Thus, p(terue)
we can approximately solve equation T in two stages: 1) Fit the distribution of benign data pa- 2) Transfer Py to estimate the distribution of adversarial examples qe-
Specifically, we propose adversarial generative model called AT-GAN to learn the distribution of adversarial examples. The overall architecture of AT-GAN is illustrated in Figure 2 Corresponding to the above two stages, implement AT-GAN by first training a GAN model called AC- GP, LA
which combines AC-GAN (Odena et al.) (017) and WGAN GP (Gulrajani et al.] 2017) to get gener- ator Goriginal, to learn pa (See Appendix B then transfering Gorigiaal to attack the target classifier f for the learning of go. We adopt AC-GAN and WGAN-GP for the AT-GAN implementation as they could build powerful generative model on three evaluated datasets, and Song et al. (2018) also utilize the same combination, But AT-GAN is not limited to the above GANs, and we also implement AT-GAN using StyleGAN2-ada. (Karras et al. 2020al on a different dataset.
3.2 TRANSFERRING THE GENERATOR FOR ATTACK
After the original generator Goriginat is trained, we transfer the generator G Goriginal to learn the distribution of adversarial examples in order to attack the target model. As illustrated in Figure
4

---

Under review as a conference paper at ICLR 2021
(2.
01 signori
Orginal Ganeralor
Trar slareeeeed Generator
Guiginll
O
Ganock
+
Discriminator
Target Generator
sahmas [1 1.
x
Gytece Intialized by Gungral
Target Classifer fareet
Targel Clasailier
farget
(a) AC-WGAN GP
(b) Transfor G for the Attack
(c) Generate Adversarial Examples
The architecture of AT AT-G The first training stage of AT-GAN is AC-WGAN GP. Once trained, we regard as the original model G original and transfer Gorigina! to attack target classifier to obtain Gattack- Once finished, AT-GAN can generate adversarial examples by Gattack-
Figure 2:
(b), there are three neural networks, including the original generator Goriginal- the attack generator Guttack be transferred that is initialized by the weights of Goriginal, and the classifier f to be attacked. The goal of the second stage can be described as:
G Gattack arg min |[Goriginat (2, ys) Gattack (z, Vs)llp
s. f(G(2, yt y
(2)
Gastock
where y/. denotes the target label, || llp denotes the tp norm and we focus on p in this work.
To optimize equation 2 we construct the loss function by I, and L2, where 1,1 aims to assure that yields the target label y/ that is fixed for target attack for each category:
L1 Eemp, [ (f (z, yo)), vc)].
(3)
Here .:)-)) denotes the cross entropy between the two terms and ys is sampled from V. L2 aims to assure that the adversarial generator Gattack generates realistic examples:
L = E Ez-p, [|/Goriginal Us) + p Guttack( y/) 11)p]
Here is small uniform random noise constrained by both to and loc norm. We add p to constrain Guttack (2, yo) to be in the neighborhood of Goriginal (2, ys) rather than be exactly the same as Goriginat (2,Wa),
The objective function for transferring Goriginal to Gattack can be formulated as (a BL2). where a and {9 are hyper-parameters to control the training process. Note that in the case that = and ( + 00, the objective function is similar to that of the perturbation based attacks (Goodfellow iet al. 2015) Tramèr et al. 2018; Madry et al., 2208). For the untargeted attack, La with the maximum confidence of prediction label y except for ys, max ( (y)Gattack fack (2, ys)).
replace y in
3.3 THEORETICAL ANALYSIS ON AT-GAN
This subsection provides theoretical analysis on why AT-GAN can generate as realistic and diverse non-constrained adversarial examples as real data. We will prove that under ideal condition, AT-GAN can estimate the distribution of adversarial examples, which is close to that of real data.
Suppose Pdttt the distribution of real data, Pa and Pa the distribution learned by the generator of AC-WGAN_GH and AT-GAN respectively. For the optimization of equation 4 L2 aims to constrain the image generated by Gattock in the e-neighborhood of Goriginal- We prove that under the ideal condition that Le guarantees Gattock(2 (z. y.) to be close enough to Goriginal (2, ya) for any input noise 2, the distribution of AT-GAN almost coincides the distribution of AC-WGAN GP. Formally, we state our result for the two distributions as follows.
Theorem 1. Suppose maxs.y L2 < e, we have K KL(pallp,) + 0 when e - 0.
The proof of Theorem is in Appendix C Samangouei et al. (2018) prove that the global optimum of WGAN is Pg Pdota and show that the optimum of AC WGAN GP has the same property. We formalize the property as follows.
Theorem 2. The global minimum of the virtual training of AC-WGAN GP achieved and only if Pg Pdota-
5

---

Under review as a conference paper at ICLR 2021
The proof of Theorem is in Appendix C
According to Theorem I and 2 under the ideal condition,
we conclude Pa 2 Pdata: which indicates that the distribution of non-constrained adversarial examples learned by AT-GAN is very close to that of real data as discussed in Section 3.1 so that the non-constrained adversarial instances are as realistic and diverse as the real data.
4 EXPERIMENTS
In this section, we provide two implementations of AT-GAN to validate the effectiveness and efficiency of the proposed approach. Empirical experiments demonstrate that AT-GAN yiclds higher attack success rates against adversarially trained models with higher efficiency. Besides, AT-GAN can learn distribution of adversarial examples which close to the real data distribution, and generate realistic and diverse adversarial examples.
4.1 EXPERIMENTAL SETUP
Datasets. We consider four standard datasets, namely MNIST (LeCun et al.
1989,, Fashion-MNIST
(Xiao et al. 2017). CelebA (Liu et al. 2015) on the AT-GAN implementation using AC-GAN (Odena
2017 and WGAN GP (Gulrajani al., 2017), and CIFAR-10 10 dataset (Krizhevsky al. 2009 on the AT-GAN implementation of StyleGAN2-ada (StyleGAN2 with adaptive discriminator
augmentation) (Karras et al.) 2020a). MNIST is dataset of hand written digits from MNIST is similar to MNIST with 10 categories of fashion clothes. CelebA contains more than 200, 000 celebrity faces. We group them according to female/male and focus on gender classification in Song al. (2018). CIFAR- consists of 32 32 color images in 10 classes, with 6, 000 images per class. For all datasets, we normalize the pixel values into range [0, 1].
Fashion-
Baselines. We compare AT-GAN with the cearch-based attack methods. including Song's 's (Song ct al. 2018) for unrestricted adversarial examples, as well as FGSM (Goodfellow et al. 2013), PGD (Madry et al. 2018) and R+ R+FGSM (Tramer al. 2018) for perturbation-ban based adversarial examples. Note that although the perturbation-based results are not directly comparable to ours as they are limited small perturbations on real images, they can provide good sense on the model robustness.
Models. For MNIST and Fashion- -NNIST, we adopt four models used in Tramer et al. (2018). denoted as Model A to D. For CelebA, we consider three models, i.e. CNN, VGG16 (Simonyan & Zisserman 2015) and ResNet (He et al. 2016). Details of Model A to D and CNN are described
Table
The ResNet same as Song al. (2018). For CIFAR-10, we adopt the wide ResNet
w32-10 Zagoruyko & Komodakis 2016). Details about the architectures of AT-GAN are provided in
Appendix D
Table 1: Architectures of Model A through D used for MNIST and Fashion- and CNN used for CelebA. The total number of parameters of each model is provided after the model name.
Model A (3,382,346)
Model (710.218)
Model C (1.795,082)
Model D (509,410)
CNN 066, 658)
Conv(64, RelRelu
Dropout(0.2)
Conv(128 3)+Relu
RFcc300)RReul
[Conv 3yRell]
Conv(64, 6 544Relu
Convt 64, 8 S)+Relu
Convi 64, 3)+Rclu
(ropout(0.5)
Dropout(0.3)
Dropout(022) 25)
Conv(128, R)+Relu
Dropour(0225)
[Comv(64, 3+Relu]
FQ(128)
Conv(128, 5 5)+Relu
FC(128)+Relu
FC(10) Salumax
Maxpool(2, 2) Dropout((.3)
Dropoun(0.5)
Dropour(0.55
Droopaut(0.5)
[Conv(125, 3)+Relu]
FC(10)=Softmux
FC(L0)+Softux
FC(10)-Softmax
Maxpoolt2, 2) Dropout(0.3)
HC(512) Relu
Dropout(0 (.3)
FC(10)) Softmax
Evaluation Setup. We consider normal training and existing advanced defenses, namely adversarial
training (Goodfellow et al. (2055), ensemble adversarial training (Tramèr al. (018) and iterative adversarial training (Madry et al. 2018). All experiments are conducted on single Titan X GPU and the parameerssssers used for attacks are described Appendix D
4.2 EVALUATION RESULTS
For evaluation, we report the comparisons attack success rate, attack efficiency and visualize some adversarial examples for AT-GAN and the baselines. More evaluation results on the transferability, ablation study, human evaluation, and the attack results CIFAR-10, provided in Appendix D

---

Under review as a conference paper at ICLR 2021
Table 2: Attack success rate (ASR, %) of adversarial examples generated by AT-GAN and the baseline attacks against models by normal training and various adversarial training methods. For each model, the highest ASR is highlighted in bold. Notation: Nor. Normal training, Adv. Adversarial training, Ens. Ensemble adversarial training, Iter. Iterative adversarial training.
(a) Comparison of attack success rale on MNIST.
Model A
Model B
Model C
Model D
Attack
Adv.
Ens.
Iter.
Nor.
Adv.
Ens.
Iter.
Nor.
Adv.
Ens.
Nor.
Adv.
Eos.
FGSM
68.8
2.9
8.7
4,0
88.0
9.5
18.0
9.2
70.7
4.3
7.8
4.7
89.6
23.5
34,6
26.7
PGD
100.0
92.6
85.1
5.9
100.0
42.4
98.2
36.2
100,0
76.9
96.4
9.6
91.7
96.8
99.5
81.1
R-FGSM
77.9
28.4
20.3
2.6
96.5
7.1
42.1
4.6
81.6
7.2
19.9
2.9
93.8
76.2
51.6
25.3
Song's
82.0
70.5
75.0
84.6
76.9
65.0
72.0
80.7
74.2
75.6
72.6
87.8
67.7
43.6
56.3
44.5
AT-GAN
98.7
97.5
96.7
91.4
99.5
97.7
99.3
95.6
99.3
95.8
96.9
90.0
99.9
99,9
99,5
99.7
(b) Comparison of attack success rate on Fashion-MNIST.
Model
Model
Model
Attack
Nor.
Adv.
Ens.
Iter.
Nor.
Adv.
Eas.
Iter.
Nor.
Adv.
Ens.
Iter.
Nor.
Adv.
Eas.
Iter.
18.0
9.2
23.5
34.6
PGD
100.0
92.6
85.1
5.9
100.0
42.4
98.2
36.2
100.0
76.9
96.4
9.6
91.7
96.8
99.5
81.1
R-PGSM
28.4
20.3
2.6
4.6
19.9
2.9
91.8
76.2
51.6
Sung's
82.0
70.5
75.0
84.6
76.9
65.0
72.0
80.7
74.2
75.6
72.6
87.8
67.7
43.6
56.3
44.5
AT-GAN
97.5
97.7
95.8
99.9
(c) Comparison of attack success rate on CelebA
CNN
VGG16
ResNet
Attack
Adv.
Ens.
Iter.
Adv.
Ens.
Iter.
Nor.
Adv.
Ens.
Iter.
FGSM
81.2
11.8
14.7
9.5
76.7
10.6
16.2
8.9
98.7
9.8
12.3
9.5
22.6
11.4
100.0
10.5
R+FGSM
68.7
9.5
11.3
7.9
68.4
8.7
13.2
7.3
97.5
7.9
9.5
7.8
Song's
85.7
89.8
95.7
99.2
90.6
AT-GAN
97.5
98.9
95.9
99.6
97.1
96.7
95.8
97.8
98.4
98.5
97.3
98.5
4.2.1 COMPARISON ON ATTACK SUCCESS RATE
To validate the attack effectiveness, we compare AT-GAN with the baselines under white-box setting. Since Athalyy al. (2018) show that the currently most effective defense method is adversarial training. we consider adversarially trained models as the defense models. The attack success rates are reported in Table [
On MNIST, AT-GAN achieves the highest Attack Success Rate (ASR) against the baselines on all defense models. As for normal training, AT-GAN achieves the highest ASR Model D, and the second highest ASR of over 98% on the other models. On Fashion-MNIST, AT-GAN achieves the highest ASR on average. On CelebA, AT-GAN achieves the highest ASR on almost all the models, with two exceptions under normal training but the results of AT-GAN are close to the highest.
In general, AT-GAN achieves the highest attack performance above 90% on all the defense models. As AT-GAN aims to estimate the distribution of adversarial examples, adversarial training with some specific attacks has little robustness against AT-GAN, raising new security issue for the development of more generalized adversarial training models.
4.2.2 COMPARISON ON ATTACK EFFICIENCY
There are many scenarios where one needs a large amount of adversarial examples, such adversarial training or exploring the property of adversarial examples. Thus, the efficiency of generating adversarial examples is very important, but such metric ignored in most existing works.
As an adversarial generative model, once trained, AT-GAN can generate ad adversarial examples very quickly. Here we evaluate the efficiency each attack method for Model A on MNIST. The average time of generating/searching 1000 adversarial examples is summarized in Table 3 Among the five attack methods, AT-GAN is the fastest as it could craft adversarial examples without target classifier and gradient calculation. Note that Song's needs much longer time than others it needs multiple searches and queries to generate one adversarial example. It takes about 8 minutes for transferring the generator of AT-GAN. Here we only focus the efficiency of generating adversarial examples after AT-GAN is transferred, i.e. we have already found the generator G as in such case we could generate many adversarial examples we need.
7

---

Under review as a conference paper at ICLR 2021
Table Comparison on the average example generation time, measured by generating 1000 adver- sarial instances using Model A on MNIST.
FGSM
PGD
R+FGSM
Song's s
AT-GAN
Time
0.3s
1.8s
0.4s
2.5min
0.2s
F0SM
2
1
PGD
#
5
4
Song
A G 0 2
2
2
5
5
Y
G 7
7
7
2
L
7
9
0 3 0 0
3
5
4
6 6 6
7
7
7
7
8
3
3
7
3
9
9
(a)
Adversarial examples for Model A on MNIST.
FBSM
PGD
RIFOOM
Sang
AT GAN
(b) Adversarial examples for Model A on Fashion-MNIST
Source Class: Female
Source Class: Male
FGSM
PGD
R+FGSM
Songls
25
AT GAN
(c) Adversarial examples for CNN on CelebA.
Figure 3: Adversarial examples generated by various attacks on three datasets (Zoom in for details). The red borders indicate unrealistic adversarial examples generated by Song's 's method or AT-GAN.
4.2.3 VISUALIZATION ON ADVERSARIAL EXAMPLES
Since the goal of adversarial examples fool target neural networks but not fool human oracle, in Figure 3 we illustrate some adversarial examples generated by different attacks for Modle A on MNIST and Fashion-MNIST, and CNN on CelebA.
On MNIST, AT-GAN generates slightly more realistic images than Song's, e.g. "0" and "3". On Rashion-MNIS' and CelebA, some adversarial examples generated by Song's 's method are not as realistic as AT-GAN to human perception, for example "t-shirt/top (0) ", "sandal (5)" and some facial details. Note that Song's 's method tends to distort the foreground that makes the images on MNIST more clean but some images are not realistic while AT-GAN tends to distort the background. As for perturbation based attacks, their adversarial examples are not clear enough, especially on MNIST and Fashion-Nee MNIST. due to the adversarial perturbations. There are also some unnatural samples generated by AT-GAN due to the limitation of GAN and we hope some better generative models can solve such issue. For target attack, please more examples crafted by AT-GAN in Appendix [
In general, AT-GAN can generate realistic and diverse adversarial examples as equation I forces the generated non-constrained adversarial examples to be close to the benign examples generated by the original generator.
4.3 VISUALIZATIION ON ADVERSARIAL DISTRIBUTION
As discussed in Section 3.3, we provide a brief analysis that AT-GAN can learn a distribution of adversarial examples close to the distribution of real image data. To identify it empirically, we randomly choose 5, 000 benign images and 5, 000 adversarial examples generated by different attack methods, and merge these images according to their real label for MNIST and Fashion-MNIST Then we use t-SNE (Maaten & Hinton, 2008) on these images to illustrate the distributions in two dimensions. t-SNE models each high-dimensional object in such way that similar objects
8

---

Under review as a conference paper at ICLR 2021
-1
Testset
(b) FGSM
(c) PGD
R+FGSM
(c) Song
(f) AT-GAN
Figure 4: T-SNE visualization for the combination of testset and adversarial examples generated by various attacks on MNIST (top) and Beshion-MNIST (bottom). For (a), we use total testset with 10,000 images. For (b) to (f), we use 5,000 sampled images in testset and 5,000 adversarial examples generated by various attacks. The position of each class is random due to the property of t-SNE.
modeled by nearby points and dissimilar objects are modeled by distant points with high probability, indicates that, if the adversarial examples have different distribution the benign data, t-SNE could not deal with them well and the points with different categories will overlap with each other after the dimension reduction, i.e. the results will be in chaos.
The results are illustrated in Figure 4 For AT-GAN, different categories are separated as that of the test set while those of other methods are mixed with each other, especially on MNIST (top). indicates the distribution AT-GAN learned is indeed very close to the distribution of real data.
To further validate that AT-GAN learns different distribution from the original GAN rather than just adding some constant universal perturbation vector. In Appendix F we illustrate some instances generated by the original generator and AT-GAN for the same input. We find that for different inputs, the original generator outputs different images and the difference between the instances generated by the original generator and AT-GAN is also different, indicating that AT-GAN indeed learns a different distribution from the original GAN.
5 CONCLUSION
In this work, we propose generation based adversarial attack method, called AT-GAN (Adversarial Transfer Generative Adversarial Net), that aims to lear the distribution of adversarial examples for the target classifier. The generated adversaries are "non-constrained" as we do no search at all in the neighborhood of the input, and trained AT-GAN output adversarial examples directly for any input noise drawn from arbitrary distribution (c.g. Gaussian distribution). Extensive experiments and visualizations show that AT-GAN achieves highest attack success rates against adversarially trained models and can generate diverse and realistic adversarial examples efficiently.
Our work also suggests that adversarial training, popular defense method based on perturbation- based adversarial examples, could not guarantee robustness against non-constrained adversarial examples. possible reason is that AT-GAN learns a more complete version of the adversarial example distribution, which is much more diverse than that of the perturbation-
method.
Note that any conditional GANs that craft realistic examples could be used for the implementation of AT-GAN. In this work, we provide two implementations on four datasets. In future work we plan to try advanced GANs for generating high resolution images. Our method also suggests new way of adversarial attack by designing adversarial generative model directly. There are several other interesting questions related to our work that can be explored in future work. For instance, what is the distribution of adversarial examples really like? Is it continuous smooth manifold? How close could we learn such distribution through GAN? We hope our work could inspire more researches in this direction.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein GAN. arXiv Preprint
arXiv: 17X10:7701.0777.. 2017.
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated Gradients Give a False Sense of Security: Cireumventing Defenses to Adversarial Examples. International Conference on Machine Learning, 80:274-283, 2018.
Tao Bai, Jun Zhao, Jinlin Zhu, Shoudong Han, Jiefeng Chen, and Bo Li. AI-GAN: Attack-inspired generation of adversarial examples, arXiv preprint arXiv:2002.00 2002::20002022966,, 2020.
Shumeet Baluja and Ian Fischer. Adversarial transformation networks: Learning to generate adver-
sarial examples. arXiv preprint arXiv: 170:: 093377 2017.
Arjun Bhagoji, Warren He, Bo Li, and Dawn Song. Exploring the Space of Black-bor box Attacks on Deep Neural Networks. arXiv Preprint arXiv:1712.09491, 2017.
Jacob Buckman, Aurko Roy, Colin Raffel, and Ian Goodfellow. Thermometer Encoding: One Hot Way To Resist Adversarial Examples. International Conference on Learning Reprexentations, 2018.
Yulong Cao, Chaowei Xiao, Dawei Yang, Jing Fang, Ruigang Yang, Mingyan Liu, and Bo Li. Adversarial Objects Against LiDAR- -ased Autonomous Driving Systems. arXiv Preprint 190.v009 2019.
Nicholas Carlini and David Wagner. Towards Evaluating the Robustness of Neural Networks. IEEE Symposium on Security and Privacy, pp. 39-57, 2017.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets. Neural Information Processing Systems, pp. 2172-2180, 2016.
Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. Robust Physical-World Attacks on Deep Learning Models. IEEE Conference on Computer Vision and Pattern Recognition, 2018.
Ian Goodfellow, Jean Pouget-Abadics Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Neural Information Process- ing Systems, pp.
2014.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and Harnessing Adversarial Examples. International Conference on Learning Representations, 2015.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Improved
Training of Wasserstein GANs. Neural Information Processing Systems, pp. 5767-5777, 2017.
Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens Maaten. Countering Adversarial Images using Input Transformations. International Conference on Learning Representations, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Warren He, Bo Li, and Dawn Song. Decision Boundary Analysis of Adversarial Examples. Interna- tional Conference on Learning Representations, 2018.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. Advances in Neural Information Processing Systems, 33, 2020a.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 8110-8119, 2020b.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. S. Kullback and R. A. Leibler. On information and sufficiency. Ann. Math. Statist., 22(1):79-86. 1951.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial Machine Learning at Scale. International Conference on Learning Representations, 2017.
10

---

Under review as a conference paper at ICLR 2021
Yann LeCun, Bernhard Boser, John Denker, Donnie Henderson, Richard Howard, Wayne Hubbard, and Lawrence Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):541-551, 1989.
Yandong Li, Lijun Li, Liqiang Wang, Tong Zhang, and Boqing Gong. Nattack: Learning the distributions of adversarial examples for an improved black-box attack on deep neural networks. International Conference Machine Learning (ICML), 2019.
Fangzhou Liao, Ming Liang, Yinpeng Dong. Tianyu Pang, Xiaolin Hu, and Jun Zhu. Defense against Adversarial Attacks Using High-Level Representation Guided Denoiser. IEEE Conference on Computer Vision and Pattern Recognition, pp. 1778-1787, 2018.
Aishan Liu, Xianglong Liu, Jiaxin Fan, Yuqing Ma, Anlan Zhang, Huiyuan Xie, and Dacheng
Tao. Perceptual-sense sensitive GAN for generating adversarial patches. In The Thirty-Third AAAI Conference on Artificial Intelligence, pp. 1028-1035, 2019.
Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into Transferable Adversarial Examples and Black-box Attacks. International Conference on Learning Representations, 2017.
Ziwei Liu, Ping Luo, Xiaogang Wang. and Xiaoou Tang. Deep Learning Face Attributes in the Wild. International Conference on Computer Vision, pp. 3730-373350 13300-378,22 2015.
Laurens Maaten and Geoffrey Hinton. Visualizing Data using t-SNE. Journal of Machine Learning
Research, 9:2579-2605, 2008.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards Deep Learning Models Resistant to Adversarial Attacks. International Conference on Learning Reprexentations, 2018.
Jan Metzen, Hendrik, Tim Genewein, Volker Fischer, and Bastian Bischoff. On Detecting Adversarial
Perturbations. International Conference on Learning Representations, 2017.
Gooosvii-eezfooll Alhussein Fawzi, and Pascal Frossard. Deepfool: a simple and accurate method to fool deep neural networks. In IEEE conference computer vision and pattern recognition, pp. 2574-2582, 2016.
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional Image Synthesis With Auxiliary Classifier GANs. International Conference on Learning Representations, 2017.
Poursaeed Omid, Katsman Isay, Gao Bicheng, and Serge Belongie. Generative adversarial perturba- tions. In IEEE Conference on Computer Vision and Pattern Recognition, 2018.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised Representation Learning with
Deep Convolutional Generative Adversarial Networks. International Conference on Learning Representations, 2016.
Konda Reddy Mopuri, Utkarsh Ojha, Utsav Garg, and R Venkatesh Babu. NAG: Network for adversary generation. In JEEE Conference on Computer Vision and Pattern Recognition, pp.
742-751, 2018.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer- Assisted Intervention, 2015.
Pouya Samangouei, Maya Kabkab, and Rama Chellappa. Defense- GAN: Protecting Classifiers
Against Adversarial Attacks Using Generative Models. International Conference on Learning Representations, 2018
Adi Shamir, Itay Safran, Eyal Ronen, and Orr Dunkelman. A Simple Explanation for the Existence of Adversarial Examples with Small Hamming Distance. arXiv Preprint arXiv: N.0ivv/8801.07770 .27774 2019.
Shiwei Shen, Guoqing Jin, Ke Gao, and Yongdong Zhang. APE-GAN: Adversarial Perturbation Elimination with GAN. IEEE International Conference on Acoustics.Speech and SP, pp. 3842- 3846, 2017.
Karen Simonyan and Andrew Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition. International Conference Learning Representations, 2015.
Chuanbiao Song, Kun He, Liwei Wang, and John Hopcroft. Improving the Generalization of Adver- sarial Training with Domain Adaptation. International Conference on Learning Representations, 2019.
11

---

Under review as a conference paper at ICLR 2021
Yang Song, Rui Shu, Nate Kushman, and Stefano Ermon. Constructing unrestricted adversarial examples with generative models. Neural Information Processing Systems, pp. 0322-8333, 2018.
Christian Szegedy, Wojciech Zaremba, llya Sutskever, Joan Bruna, Dumitru Erhan, lan Goodfellow, and Rob Fergus. Intriguing properties neural networks. International Conference on Learning Representations, 2014.
Florian Tramèr. Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick Mc- Daniel. Ensemble Adversarial Training: Attacks and Defenses. International Conference on Learning Representations, 201 8.
Chaowei Xiao, Bo Li, Jun-Yan Zhu, Warren He, Mingyan Liu, and Dawn Song. Generating Adversarial Examples with Adversarial Networks. International Joint Conferences on Artificial Intelligence, pp. 3905-3911 2018.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: a Novel Image Dataset for Bench- marking Machine Learning Algorithms. arXiv preprint 170.ve1708..7774,, 2017.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. British Machine Vision Confer-
ence, 2016.
12

---

Under review as a conference paper at ICLR 2021
APPENDIX
In the appendix, we provide additional related work on gradient-based adversarial attack methods, adversarial training methods and typical generative adversarial nets. Then we describe how obtain the original generator and provide theoretical analysis, as well as experimental details and additional results. In the end, we visualize the examples generated by original GAN and AT-GAN.
A ADDITIONAL RELATED WORK
A.1 GRADIENT-BASED ATTACKS
Numerous adversarial attacks have been proposed in recent years (Carlini & Wagner, 2017 Liu et al., 2017 Bhagoji et al. 2017 Li et al. 2019). In this part, we will introduce three typical adversarial attack methods. Here the components of all adversarial examples are clipped in [0. 1].
Fast Gradient Sign Method (FGSM). FGSM (Goodfellow et al., 2015) adds perturbation in the gradient direction of the training loss J on the input to generate adversarial examples.
Tade = x + e sign( a. x, Strae)),
where Virue is the true label of a sample x, is the model parameter and specifies the &x distortion between and Zadv-
Projected Gradient Descent (PGD). PGD adversary (Madry et al. 2018) is a multi-step variant of FGSM, which applies FGSM for iterations with budget a.
Lader Clipp(addo tasign(V..l (6, Zadvi Vtrue)), Tadvi e, Fodv; + c)
Tades = I, Tade = Fadee
Here clip(a', ...)) forces its input x' to reside in the range of (p.g].
Rand FGSM (R+FGSM). R+FGSM (Trmer et al., 2018) first applies small random perturbation on the benign image with a parameter a (a < <), then it uses FGSM to generate an adversarial example based the perturbed image.
Fadv =x's (c- a.) sign(V... x', ytrue)) where x' x sign(N((..I)
A.2 ADVERSARIAL TRAINING
There are many defense strategies, such as detecting adversarial perturbations (Metzen et al. 201 7) obfuscating gradients (Buckman al. 2018 Guo et al.) (2018) and eliminating perturbations (Shen
2017 Liao et al., 2018), among which adversarial training the most effective method (Athalye et al.
al. 2018) We list several adversarial training methods as follows.
Adversarial training. Goodfellow et al. (015)) first introduce the method of adversarial training, where the standard loss function f for neural network modified
J(0, x, ytrue) y 0, x, Vorue) + ( Zade Mtrue).
Here Verue is the true label of sample and is the model's parameter. The modified objective is to make the neural network more robust by penalizing to count for adversarial samples. During the training, the adversarial samples calculated with respect to the current status of the network. Taking FGSM for example, the loss function could be written as:
J(A, v, Vtrue) =a:,((0,2) Virue) (1 (0,, + csign(V J(9, Vtrue)), ytrue).
Ensemble adversarial training. Tramer et al. 2018 propose ensemble adversarial training method, which DNN is trained with adversarial examples transferred from a number fixed pre-trained models.
Iterative adversarial training, Madry et al. (0018 propose to train a DNN with adversarial examples generated by iterative methods such PGD.
13

---

Under review as a conference paper at ICLR 2021
A.3 GENERATIVE ADVERSARIAL NET
Generative Adversarial Net (GAN) (Goodfellow et al. (204)) consists of two neural networks, G and D, trained in opposition each other. The generator G is optimized estimate the data distribution and the discriminator D aims to distinguish fake samples from G and real samples from the training data. The objective of D and G can formalized as a min- max value function V)G
min max (GG, D) = Ex-p, [log Dxr) + Ezop, [log(1 D(G(=))))
Deep Convolutional Generative Adversarial Net (DCGAN) (Radford et al. 2016 is the convolutional version of GAN, which implements GAN with convolutional networks and stabilizes the training process. Auxiliary Classifier GAN (AC-GAN) (Odena et al. 2017) is another variant that extends GAN with some conditions by an extra classifier C. The objective function of AC-GAN can be formalized follows:
min max min V,G,D,C -Eawp [log (xx) Egw,, [log(1 D(G((z,))))) [
D
C
Eayup [log(1 (1- C1,, ys))] + Ezup, [log(1 - C()((((G))))))))))))
To make GAN more trainable in practice, Arjovsky al. (217)) propose Wasserstein GAN (WGAN)
that uses Wassertein distance so that the loss function has more desirable properties. Gulrajani et al. (2017) introduce WGAN with gradient penalty (WGAN_GH GP) that outperforms WGAN in practice. Its objective function formulated as:
min max (DD Eg-p_ [D(x)] - Ex-p, (D(G()) - AE I(II V 1)²}, G D
where Ps is uniformly sampled along straight lines between pairs of points sampled from the data distribution Ps and the generator distribution Pg-
B TRAINING THE ORIGINAL GENERATOR
Figure (a) illustrates the overall architecture of AC-WGAN GP that used the normal GAN. AC- GAN GP is the combination AC-GAN (Odena et al., 2017) and WGAN GP (Gurrajani al. 2017), composed by three neural networks: generator G, discriminator D and classifier J. The generator takes a random noise and a source label ys as the inputs and generates an image G(2, ys). It aims generate an image G(z, ys) that is indistinguishable discriminator D and makes the classifier ff to output label ya. The loss function of G can be formulated as:
LG = Ez-ps(>) (f (G(z, ye)), ya)l - Ez-px(=) (=) [D(G(z, ya))].
Here (a, b) the entropy between a and b. The discriminator D takes the training data x or the generated data G(2, (s) the input and tries distinguish them. The loss function of D with gradient penalty for samples & ~ ps can be formulated as:
LD Ee
(e
+ AE D())= -
The classifier takes the training data a or the generated data G(z, ys) as the input and predicts the corresponding label. The loss function is:
L -Berpear (x) (z) (z)))) Vtrue)] + (H v.)).
Different from AC-WGAN GP, styleGAN2-ade Karras al. 2020a) trains styleGAN2 (Karras et al. 2020b) with adaptive discriminator augmentation. We obtain the network and weights from Karras et (002aa).
C THEORETICAL ANALYSIS OF AT-GAN
In this section, we provide proofs for theorems in Section 3.3.
Theorem 1. Suppose maxz,y y La e, have K (T(pallgg) 0 when 0.
14

---

Under review as a conference paper at ICLR 2021
Proof. We first consider that for distribution p(z) in space X, construct another distribution ((x) by selecting points Pe(x) in the e-meighborhood of p(z)) for any z E X. Obviously, when p. (x) is close enough to p(z). ((2)) has almost the same distribution as p(x). Formally, we have the following lemma.
Lemma 1. Given two distributions P and Q with probability density function p(x) and ((x:) in space X, if there exists a constant that satisfies [|((z) p(z) < for any x E A, we could get LL(PIIQ) - when -
Proof. For two distributions P and Q with probability density function p(r) and g(z), we could get (()) p(x)) r(x) where ||r(x))] <.
KL(PIIQ) / p(x) log p(x) dr ((x))
= / p(xx) log p( / p(x) log g(x)de
| r r(x)) log p(x))d ((a(a)-
- r(x)) log q(x)dr
/ ((z))ogp(x)dr / ((2))log ((r)( / ( ((z))ogp(z)dd + r((z) log ((x)dr
(o))log g(2) dr KL(QIIP) p(x)
$ 5 clog(1 p(z)
Jdr
Obviously, when - O, we could get f log((1 + i May
)dr dr + which means DL(PII
Now, we get back to Theorem I For two distributions Pa and Py, maxy, L2 < indicates Vz
Pz, (l, :) P2(2, -) According Lemma 1 have ('L(palPp)) - when - 0. This concludes the proof.
Theorem 2. The global minimum of the virtual training of AC-WGAN GP achieved and only
if Pg Pdata-
Proof. To simplify the analysis, we choose category y of AC-WGAN GP and denote Py and Pdota (Fly) the distribution that the generator learns and the distribution of real data respectively. Then for each category, the loss function is equivalent to WGAN G_. We refers to (Samangouei et al.
(2018) to prove this property. The WGAN_ min-max loss is given by:
min max VDD G D
Exrpaara (x) [xx)] (t) (D({()))) - ABiupa (IV - 1)
Pdaaa (x) / /
(+) - 1)
/ (x (x)) pa pa( 1)
(5)
For fixed G, the optimal discriminator D that maximizes V(D, G) should be:
1
if Páato (x) (2(2)
(z(x)
(6)
0
otherwise
According to equation 5 and equation 6 we could get:
/ Pa (+)((())))) 1)2)di
V(D,G) (x) Pg(x))
(1)ppaau(44)22000)) (pdasa (x)) - Po (x)))d A pa())dd
(7)
=/ (pitasa (2) Py (2)))))
(xlppan (1)
15

---

Under review as a conference paper at ICLR 2021
Let X' (xlpdata(x)) (z(x)), in order to minimize equation 7 set Pdota( Py(x)) for any
I E X. Then, since both Py and Pdata integrate to 1, we could get:
J. Po Po(x))de = f Pataa(x)ddz
However, this contradicts equation where Pdata (x) < py(x) for x E X unless y((²s) 0 where is the Lebesgue measure.
Therefore, for each category we have Py (a(l)) Pdata (zly), which means Py Pdota (x) for AC-WGAN GP.
D ADDITIONAL DETAILS ON EXPERIMENTS
In this section, we provide more details on experimental setup, report results on transferability. do ablation study eyper-parameters. investigate the generating capacity by human evaluation, and show details for another implementation of AT-GAN on CIFAR-10 dataset. In the end, we illustrate some non-constrained adversarial examples generated by AT-GAN MNIST, Fashion-MNIST and CelebA for the target attack.
D.1 MORE EXPERIMENTAL SETUP
We first provide more details the experimental setup, including the model architectures and attack hyper-paramedn
Model Architectures for AT-GAN. We first describe the neural network architectures used for AT-GAN experiments. The abbreviations for components in the network are described in Ta- ble 4 The architecture of AC-WGAN GP for MNIST and Fashion-MNIST is shown in Table 5 where the generator and discriminator are the same as in Chen et al. (2016), while the architecture of AC_ WGAN GP for CelebA the same as in Gulrajani et (2017) and the architecture of styleGAN2-ada for CIFAR-10 is the same as in Karras et al. (2020a)
Hlyper-paramctery for Attacks. The hyper-parameters used in experiments for each attack method are described in Table 6 for MNIST, Fashion-MNIST and CelebA datasets. For CIFAR-10 dataset, we set 0.03 for FGSM, 0.03, 0.0075 and epochs= 20 for PGD, a 3. B 2 and
epochs= 1, 000 for AT-GAN,
Table 4: Abbreviations for network architectures.
Abbreviation
Description
Conv(m, x k)
convolutional layer with m filters and filter size k A transposed convolutional layer with m filters and filter size k
DeConv(m, x xx) k)
Dropout(a)
A dropout layer with probability a A fully connected layer with m outputs The sigmoid activation function
FC(m)
Sigmoid Relu LeakyRelu(a)
The Rectified Linear Unit activation function
The Leaky version of a Rectified Linear Unit with parameter The maxpooling with filter size lt and stride
Maxpool(k.s)
D.2 TRANSFERABILITY OF AT-GAN
Another important issue for adversarial examples is the transferability across different models. To demonstrate the transferability of non-constrained adversarial examples, we use adversarial examples generated by attacking Model A (MNIST and Fashion-MNISTT and CNN (CelebA), to evaluate the attack success rates on Model C (MNIST and Mushion-MNISS and VGG16 (CelebA). As shown in Table 7 non-constrained adversarial examples generated by AT-GAN exhibit moderate transferability.
16

---

Under review as a conference paper at ICLR 2021
Table Architecture of WGAN with auxiliary classifier for MNIST and FaShion-MNIST.
Generator
Discriminator
Classifier
FC(1024) Relu
Conv(64, 4 x 4) Leaky
Conv(32, 3 3) Relu
FC(7 7 x 128) + Relu
Conv(128,
LeakyRRlu(002))
pooling(2, 2) Conv(64, 3 x 3) + Relu
D6C, 4 x 4) + Sigmoid
FC(1024) + LeakyRell(0.2))
DeConv(1, x 4) Sigmoid FC(1024)
FC(1) + Sigmoid
pooling(2, 2)
Dropout(0.4)
FC(10) Softmax
Table Hyper-parameters of different attack methods on MNIST, Aeshion-MNIST and CelebA.
Datasets
Attack
MNIST
Fushion-MNIST
CelebA
Norm
FGSM
=.1
0.015
PGD
= 0.3, 0.075, epochs 20
0.1, 0.01, epochs 20
0.015, 0.005, epochs 20
foo
R+FGMM
== 0.3, = 0.15
0.2, 0.1
0.015, 0.003
foo
Song's
A1 100, by epochs
A 100, Ag epochs
100, 2g 100, epochs 200
AT-GAN
a 0= 1, epochs 100
2, 1, 1,epnchs = 100
3, 8 2, epochs 200
N/A
Table Transferability of non-constrained adversarial examples and other search-based adversarial examples on three datasets. For MNIST and Fashion-MNIST, we attack Model C with adversarial examples generated on Model For CelebA dataset, we attack VGG16 using adversarial examples generated CNN. Numbers represent the attack success rate (%).
MNIST
Fashion-MNII -NISS
CelebA
Nor.
Adv.
Ens.
Iter.
Adv.
Nor.
Adv.
Ens.
Iter. Adv.
Nor.
Adv.
Ens.
Iter. Adv.
FGSM
46.7
4.2
1.7
4.6
68.9
23.1
20.8
14.8
15.6
4.3
3.3
4.1
PGD
97.5
6.5
4.1
4.1
84.7
27.6
14.6
18.3
4.3
3.1
4.1
R+FGSM
82.3
6.7
4.8
4.1
21.2
32.1
17.5
26.3
11.0
3.3
3.8
Song's
23.8
20.8
20.6
20.1
39.2
34.0
31.5
30.3
9.6
31.8
21.5
38.8
AT-GAN
65.3
24.6
27.9
17.2
58.0
22.7
32.0
15.2
63.7
15.4
16.5
17.6
D.3 ABLATION STUDY
In this subsection, we investigate the impact of using different in the loss function. As could be constrained by both to and foo norm, we test various bounds, using Model on MNIST dataset, for p in to and fo.. respectively.
We first fix ||p|| 0.5 and try various values for ||p||o. i.e. 0, 100, 200, 300, 400 (the maximum possible value is 784 for 28*28 input). The attack success rates are in Table We can observe that different values of ||p||o only have a little impact on the attack success rates, and the performances are very close for ||e||o = 0. 100, 200. Figure 5 further illustrates some generated adversarial examples, among which we can see that there exist some slight differences on the examples. When || 0, AT-GAN tends to change the foreground (body) of the digits. When increase the value of ||l (100 and 200), AT-GAN is more likely to add tiny noise to the background and the crafted examples realistic humans (for instance, smoother (300 or 400), AT-GAN tends to add more noise and the quality of the generated examples decays. To have a good tradeoff on attack performance and generation quality, we set ||p||o 200.
digit 4). But if we continue increase ||lo
Table Attack success rate (ASR, %) AT-GAN with various values for ||p||u using Model A on MNIST dataset.
lello 0
100
200
300
400
ASR
98.9
98.8
98.7
96.7
95.8
We then fix l|pllo- = 200 and test different values for || Le. 0, 0.1, 0.2, 0.3, 0.4, 0.5 (the maximum possible value 1). The attack success rates in Table
We observe that different values of
17

---

5 5 Under review as a conference paper at ICLR 2021
[+
/
2
3
7
9
9
2-
3
2
7
5
5
6 5
6
b
9
!
2-
2
5
7
5
5
5
6
6
8
9
/
2-
7
9
5
5
bk
2
3
5 7666666777
Figure 5: The adversarial examples generated by AT AT-GAN for various values Ip
ale have very little impact on the attack performance. Figure 6 further illustrates some generated adversarial examples, among which we can that little bit more noises are added for bigger Il but the differences are very tiny when || = 0.2 to 0.5, So we simply set ll = 0.5 in experiments, but other values of |o (0.2, 0.3, 0.4) also work.
Table Attack success rate (ASR, %) of AT-GAN with various values for ||p|| using Model A on MNIST dataset.
0.1
0.2
0.3
0.4
0.5
ASR
98.9
99.2
98.9
98.9
98.9
98.7
2. 2.
3 3 2
2
4
4
5
5
5
6
7
7
8
3
8
4
4
4
9
Id-
2 2. 2
4
4
5
8
5
5
8
3
4
4
9
9
Id--99
2.
3
4
4
4
5
5
5
5
7
7
8
7
9
9
9
Idw-02
2.
3 2
4
4
4
5
5
9
9
4
9
9
2
4 4
9
9
9
Im--95
2 2
3
6
7
7
Figure 6: The adversarial examples generated by AT-GAN for various values of |
D.4 HUMAN EVALUATION
To investigate the generating capacity of AT-GAN, we use the same input, and randomly pick 100 images for each category of MNIST generated by AT-GAN and the original generator, respectively. We then conduct human evaluation to determine whether each example is realistic. The evaluation results are Table I0 We see that adversarial examples in some categories (c.g. 2, 4) are harder to be semantically meaningful than other categories (e.g. 0, 1). On average, however, the generating capability is close to that of the original generator.
Table 10: The evaluation results on the percentage realistic images by human evaluation.
Category
0
1
2
3
5
6
7
8
Average
Original
100.0
100.0
93.0
94.0
98.0
96.0
99.0
100.0
98.0
100.0
97.8
AT-GAN
100.0
100.0
85.0
91.0
80.0
90.0
97.0
98.0
92.0
100.0
93.3
D.5 AT-GAN ON CIFAR-10 10 DATASET
To further demonstrate the flexibilityy of AT-GAN, we implement AT-GAN on CIFAR-10 dataset using StyleGAN2-ada (Karras et al. (020a), recently proposed conditional GAN. The target classifier wide ResNet w32-10 (Zagoruyko & Komodakis 2016) by normal training (Nor.) and Iterative adversarial training (Iter.). The attack success rates are in Table IT On normally trained models, PGD achieves the attack success rate of 100% while AT-GAN achieves the attack success rate of 93.5%. However, the adversarially trained model exhibits little robustness against AT-GAN and AT-GAN achieves attack success rate of 73.0%. In Figure] we illustrate some generated adversarial
examples on CIFAR- 10 dataset.
Table 11: Attack success rate (%) of adversarial examples generated by FGSM, PGD and AT-GAN
against wide ResNet w32-10 by normal training (Nor.) and iterative adversarial training (Iter.).
Model
FGSM
PGD
AT-GAN
Nor. 92.3
100.0
93.5
Iter. 49.2
54.6
73.0
18

---

Under review as a conference paper at ICLR 2021
D.6 AT-GAN ON TARGET ATTACK
Here we show some non-constrained adversarial examples generated by AT-GAN for the target attack. The results are illustrated in Figure 8 for MNIST and Fashion-MNIST, and Figure 9 for CelebA Instead of adding perturbations the original images, AT-GAN transfers the generative model (GAN) so that the generated adversarial instances are not in the same shape of the initial examples (in diagonal) generated by the original generator. Note that for CelebA, the target adversarial attack is equivalent to the untarget adversarial attack as is a binary classification task.
Figure 7: The adversarial examples generated by AT-GAN on CIFAR- 10 dataset,
Target Classification
Target Classficatio
C
O
Tabkitt
Touwerr
Pullowr
Deess
Cost
Sandal
Shirt
Bia
Anndl boot
2
1
f
i
i
T-shirt
2
2
2
Q
Trouser
Source Clecsiicaioo
9
0
3
3
3
4
3
6
3
3
3
Cleesicatioo
4
7
4
6
4
4
Dress
Cost
5
5
S
5
Sandal
4
6
6
6
6
Shirt
7
7
2
7
7
7
7
7
7
Sneaker
8
8
8
6
8
0
Bag
4
4
4
4
5
9
Ankle boot
(b) Fashion-MNIST
(a) MNIST
Figure 8: Adversarial examples generated by AT-GAN to various targets with the same random noise input for each row. The images the diagonal are generated by G Goriginat which are not adversarial examples and treated as the initial instances AT-GAN.
Figure 9: Adversarial examples generated by AT-GAN on CelebA dataset for the target attack.
19

---

Under review as a conference paper at ICLR 2021
E VISUALIZATIONS FOR THE ORIGINAL GAN AND AT-GAN
Here we provide some instances generated by the original GAN and AT-GAN with the same input noise and their difference on MNIST and Fashion-MNIST. The results are depicted in Figure I0 and For different input noise, both the original GAN and AT-GAN output different instances. For each category with the same input noise, the difference between original GAN and AT-GAN is mainly related to the main content of image. For two different input noises, the differences between the original GAN and AT-GAN are not the same with each other, indicating that AT-GAN learns distribution of adversarial examples different from the original GAN rather than just adds some universal perturbation vectors on original GAN.
/ 2
2
3
3
3
4
4
6
5
6
6 6
8 8
8
/
2 7-
2
3
3
3
5
8
8
9
9
9
Figure 10: The instances generated by the original GAN and AT-GAN with the input MNIST. First row: the output of original GAN. Second row: the output of AT-GAN. Third row: The difference between the above rows.
Figure 11: The instances generated by the original GAN and AT-GAN with the same input Fashion-MNIST. First row: the output of original GAN. Second row: the output of AT-GAN. Third row: The difference between the above two rows.
20


Reviews:
[{"review_id": "mfwKnrx953P", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed the adversarial transfer on generative adversarial net (AT-GAN) to train an adversarial generative model that can directly produce adversarial examples. In the other way, AT-GAN could generate the adversarial examples directly for any input noise. Such a generative model was able to draw non-constrained adversarial examples.\n\nPros: \nThis paper is clearly written with reasonable paper organization covering background, model design, mathematical formula and experiments. The goal of this work is obvious with experimental justification. Mathematical description and experimental illustration are desirable to show the merit of this method.\n\nCons: \nThe reasons of using AC-GAN and WGAN-GP as the pre-train stage are missing.", "extra_scores": {}, "extra_text": {}}, {"review_id": "d24gIqvaa8x", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes AT-GAN (Adversarial Transfer on Generative Adversarial Net) to train an adversarial generative model that can directly produce adversarial examples. Different from previous works, the study aims to learn the distribution of adversarial examples so as to generate semantically meaningful adversaries. AT-GAN achieves this goal by ﬁrst learning a generative model for real data, followed by transfer learning to obtain the desired generative model. Once trained and transferred, AT-GAN could generate adversarial examples directly for any input noise, denoted as non-constrained adversarial examples. Some experiments and visualizations show that AT-GAN can generate some diverse adversarial examples that are realistic to human perception, and yields higher attack success rates against adversarially trained models. \n\nOverall, the idea seems straightforward. Benefiting from the GAN, the proposed model could learn the distribution of adversarial examples to attach the target models. The paper is clearly written and some experiments are conducted. However, I have some concerns as below:\n\n1. In the loss function, $\\rho$ controls the difference between the outputs of the original and attach GANs, it is expected to see the performance and generated examples with different $\\rho$. \n\n2. The idea seems incremental. The main contribution is to transfer a pre-trained GAN to attach GAN to fool the classifiers. The novelty could be further summarized by highlighting the difference with most related works including but not limited to the aforementioned ones. The current manuscript makes the work seem like a straightforward combination of many existing approaches. \n\n3. Some experiment settings are not clear. A brief introduction to Model A to B should be given in the main paper, though the details is provided in Appendix.\n\nAs most concerns of mine are addressed by the rebuttal and I would like to rise my score. \n   \n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "9qJB45VB5iR", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is to train a generative neural networks that can output adversarial examples. The main idea is to first train a normal GAN and then use the idea of transfer learning based on adversarial examples. The aim sounds good but the authors fail to clearly distinguish the idea with the exiting related methods theoretically or numerically. The idea of transferring is good (although not new), but after checking the implementation details, I have to say in the current version, the fact of transferring is quite limited.  \n\nDetails:\n+ the idea of generating adversarial examples by a trained GAN is interesting.\n+ the writing is quite clear.\n-  lack of comparison with existing related methods. \n   Consider the core formulation, namely (2), which well describes the idea of this authors. But it is necessary to consider the following ideas: \n\n   1). generating adversarial permutation (AdvGAN, AI-GAN):  min_G \\| G(z,y) \\|_p, s.t., f(z+G((z,y)) = y_t \\neq y_s.\n   It is to train the difference of G_original and G_attack and I think in the training aspects, this is almost equal to the proposed idea. The authors try to argue that the proposed model does not require an input. But in my opinion, no input is a disadvantage:  if only adversarial examples are needed, AdvGAN etc. can feed an random input to original GAN and then add perturbations; but if one wants to attack a specific image, the proposed method will fail.  \n\n   2). attack a GAN to generate adversarial examples (Song's): min_z' \\|z - x\\|, s.t., f(G(z,y)) \\neq f(G(z'),y). \n   The author may argue the Song's attack procedure takes longer time. However, the there is no training time additionally needed . Moreover, I guess the generating capability of Song's idea, which relies on the GAN and there are many well-designed ones, is better than the proposed one.  I would like to see the generating performance of the proposed method on more complicated datasets, e.g., on CIFAR or other HIGH-RESOLUTION images. Another good point of Song's idea is that almost all the attacks on images could be parallelly used. I do not know whether its ASR could be easily improved. \n\n- The idea of transferring the original GAN to the attacking one is interesting. However, except of using the original GAN as the starting point, I cannot find other facts of \"transferring\". I would like to know if transferring learning technique could be used to reduce the number of required adversarial examples. \n\n- The attack transferbility has not been tested. Since there is adversarial samples involved, the obtained GAN is expected to be related to the victim model. \n\nAdditional questions, mainly for the experiments' result \n1. It is good that attack performance on adversarial trained NN is included. But where the adversarial examples come from? Are the examples are generated by AT-GAN?\n\n2. How many examples and time are needed to train the AT-GAN?\n\n3. Since the GAN has been changed, how about the generating capability, i.e., generating failure ratio of the AT-GAN should be reported. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: I thank the authors and reviewers for their discussions about this paper. The proposed AT-GAN is a GAN-based method to generate adversarial examples. Similar methods (e.g. Song et al) have been proposed to use GANs to generate adv. examples more efficiently. Authors show their method has some numerical benefits. However, more experiments are needed to further justify it. Also, creating "unrestrictive" adv. examples can cause a risk of generating samples where the true label is flipped. Authors need to clarify it. Given all, I think the paper needs a bit of more work to be accepted. I recommend authors to address the aforementioned concerns in the updated draft.   

-AC
Decision: Reject