AT-GAN: An Adversarial Generative Model for Non-constrained Adversarial Examples

1 INTRODUCTION . In recent years , Deep Neural Networks ( DNNs ) have been found vulnerable to adversarial examples ( Szegedy et al. , 2014 ) , which are well-crafted samples with tiny perturbations imperceptible to humans but can fool the learning models . Despite the great success of the deep learning empowered applications , many of them are safety-critical , for example under the scenario of self-driving cars ( Eykholt et al. , 2018 ; Cao et al. , 2019 ) , raising serious concerns in academy and industry . Numerous works of adversarial examples have been developed on adversarial attacks ( Goodfellow et al. , 2015 ; Carlini & Wagner , 2017 ; Madry et al. , 2018 ) , adversarial defenses ( Goodfellow et al. , 2015 ; Kurakin et al. , 2017 ; Song et al. , 2019 ) and exploring the property of adversarial examples ( He et al. , 2018 ; Shamir et al. , 2019 ) . For adversarial attacks , most studies focus on the perturbation-based adversarial examples constrained by input images , which is also the generally accepted conception of adversarial examples . Generative models are also adopted recently to generate adversarial perturbations from an input noise ( Reddy Mopuri et al. , 2018 ; Omid et al. , 2018 ) or from a given image ( Xiao et al. , 2018 ; Bai et al. , 2020 ) , and such perturbations are added to the original image to craft adversarial examples . Song et al . ( 2018 ) propose to search a neighborhood noise around the input noise of a Generative Adversarial Net ( GAN ) ( Goodfellow et al. , 2014 ) such that the output is an adversarial example , which they denoted as unrestricted adversarial example as there is no original image in their method . However , their output is still constrained by the input noise , and the search is time-consuming . In this work , we propose an adversarial generative model called AT-GAN ( Adversarial Transfer on Generative Adversarial Net ) , which aims to learn the distribution of adversarial examples . Unlike previous works that constrain the adversaries in the neighborhood of input image or input noise , including the prominent work of Song et al . ( 2018 ) that searches over the neighborhood of the input noise of a pre-trained GAN in order to find a noise whose output image is misclassified by the target classifier , AT-GAN is an adversarial generative model that could produce semantically meaningful adversarial examples directly from any input noise , and we call such examples the non-constrained adversarial examples . Specifically , we first develop a normal GAN to learn the distribution of benign data so that it can produce plausible images that the classifier and a human oracle will classify in the same way . Then we transfer the pre-trained GAN into an adversarial GAN called AT-GAN that can fool the target classifier while being still well recognized by the human oracle . AT-GAN is a conditional GAN that has learned to estimate the distribution of adversarial examples for the target classifier , so AT-GAN can directly generate adversarial examples from any random noise , leading to high diversity and efficiency . We implement AT-GAN by adopting AC-GAN ( Odena et al. , 2017 ) and WGAN-GP ( Gulrajani et al. , 2017 ) in the pre-training stage , then do transfer learning for the adversary generation . Here we develop AT-GAN on three benchmark datasets , namely MNIST , Fashion-MNIST and CelebA , and apply typical defense methods to compare AT-GAN with existing search-based attacks . Empirical results show that the non-constrained adversarial examples generated by AT-GAN yield higher attack success rates , and state-of-the-art adversarially trained models exhibit little robustness against ATGAN , indicating the high diversity of our adversaries . In addition , AT-GAN , as a generation-based adversarial attack , is more efficient than the search-based adversarial attacks . Note that all conditional GANs that can craft realistic examples could be used for the implementation of AT-GAN . For another demonstration , we adopt StyleGAN2-ada ( Karras et al. , 2020a ) and develop AT-GAN on CIFAR-10 benchmark dataset using wide ResNet w32-10 ( Zagoruyko & Komodakis , 2016 ) as the target classifier . Empirical results show that AT-GAN can produce plausible adversarial images , and yield higher attack success rates on the adversarially trained models . 2 PRELIMINARIES . In this section , we provide definitions on several types of adversarial examples and adversarial attacks , and give a brief overview of adversarial attacks using GAN . Other related works on typical adversarial attacks and defenses ( Goodfellow et al. , 2015 ; Madry et al. , 2018 ; Tramèr et al. , 2018 ) , as well as some typical GANs ( Goodfellow et al. , 2014 ; Radford et al. , 2016 ; Odena et al. , 2017 ; Arjovsky et al. , 2017 ; Gulrajani et al. , 2017 ) are introduced in Appendix A . 2.1 DEFINITIONS ON ADVERSARIES . Let X be the set of all digital images under consideration for a learning task , Y ∈ R be the output label space and pz ∈ Rm be an arbitrary probability distribution ( e.g . Gaussian distribution ) where m is the dimension of pz . A deep learning classifier f : X → Y takes an image x ∈ X and predicts its label f ( x ) . Suppose px and padv are the distributions of benign images and adversarial examples , respectively . Assume we have an oracle classifier o : X → Y , which could always predict the correct label for any image x ∈ X , we define several types of adversarial examples as follows . For perturbation-based adversarial examples ( Szegedy et al. , 2014 ; Goodfellow et al. , 2015 ; MoosaviDezfooli et al. , 2016 ) , tiny perturbations are added to the input images , which are imperceptible to humans but can cause the target classifier to make wrong predictions . Definition 1 . Perturbation-based Adversarial Examples . Given a subset ( trainset or testset ) images T ⊂ X and a small constant > 0 , the perturbation-based adversarial examples can be defined as : Ap = { xadv ∈ X |∃x ∈ T , ‖x− xadv‖p < ∧ f ( xadv ) 6= o ( xadv ) = f ( x ) = o ( x ) } . Song et al . ( 2018 ) define a new type of adversarial examples called unrestricted adversarial examples , which is not related to the subset ( trainset or testset ) images , by adding adversarial perturbation to the input noise of a mapping , such as GAN , so that the output of the perturbed noise is an adversary to the target classifier . Definition 2 . Unrestricted Adversarial Examples . Given a mappingG from z ∼ pz toG ( z , y ) ∼ pθ , where pθ is an approximated distribution of px , and a small constant > 0 , the unrestricted adversarial examples can be defined as : Au = { G ( z∗ , ys ) ∈ X |∃z ∼ pz , z∗ ∼ pz , ‖z − z∗‖p < ∧ f ( G ( z∗ , ys ) ) 6= o ( G ( z∗ , ys ) ) = f ( G ( z , ys ) ) = o ( G ( z , ys ) ) = ys } where ys is the source label . In this work , we train a conditional GAN to learn the distribution of adversarial examples and output the corresponding adversary directly from any input noise . To clarify the difference with Song et al . ( 2018 ) , we call our generated adversaries the non-constrained adversarial examples . Definition 3 . Non-constrained Adversarial Examples . If there is a mapping G∗ from z ∼ pz to G∗ ( z , y ) ∼ qθ , where qθ is an approximated distribution of padv , the non-constrained adversarial examples can be defined as An = { G∗ ( z , ys ) ∈ X |f ( G∗ ( z , ys ) ) 6= o ( G∗ ( z , ys ) ) = ys } where ys is the source label . Here we need to find a mapping G∗ , e.g . a generative model , such that for z ∼ pz , G∗ ( z , y ) is an image in X and the output distribution is an approximated distribution of padv , for example using the Kullback-Leibler divergence ( Kullback & Leibler , 1951 ) , KL ( qθ||padv ) < for a small constant . In summary , perturbation-based adversarial examples are based on perturbing an image x ∈ X , and unrestricted adversarial examples ( Song et al. , 2018 ) perturbs an input noise z ∼ pz for an existing mapping G. Most perturbation-based adversarial attacks and Song et al . ( 2018 ) fall into the search-based adversarial attack . Definition 4 . Search-based Adversarial Attack . Given an input vector v ∈ V ( either benign image x or random vector z ) , the search-based adversarial attack searches a vector v′ : ‖v− v′‖p < where v′ leads to an adversarial example for the target classifier . In contrast , non-constrained adversarial examples are more generalized so that we need to learn a mapping G∗ such that for any input noise sampled from distribution pz , the output is an adversarial image . Such a mapping to be learned is called an adversarial generative model , and our method falls into the generation-based adversarial attack . Definition 5 . Generation-based Adversarial Attack . Given an input vector v ∈ V ( either benign image x or random vector z ) , the generation-based adversarial attack generates adversarial perturbation or adversarial example directly from v , usually adopting generative models . 2.2 GENERATIVE MODELS FOR ADVERSARIAL ATTACK . Generative models have been adopted for adversarial attack in recent works ( Baluja & Fischer , 2017 ) . Reddy Mopuri et al . ( 2018 ) propose a Network for Adversary Generation ( NAG ) that models the distribution of adversarial perturbations for a target classifier so that their NAG can craft adversarial perturbations from any given random noise , which will be added to the natural image to fool the target classifier . Omid et al . ( 2018 ) propose to generate universal or image-dependent adversarial perturbations using U-Net ( Ronneberger et al. , 2015 ) or ResNet Generator ( He et al. , 2016 ) from any given random noise . Xiao et al . ( 2018 ) propose to train AdvGAN that takes an original image as the input and generate adversarial perturbation for the input to craft an adversarial example . Bai et al . ( 2020 ) further propose AI-GAN that adopts projected gradient descent ( PGD ) ( Madry et al. , 2018 ) in the training stage to train a GAN to generate target adversarial perturbation for the input image and target class . The above attack methods all fall into the generation-based adversarial attack , and their crafted examples fall into the perturbation-based adversarial examples . Another recent work called PS-GAN ( Liu et al. , 2019 ) pre-processes an input seed patch ( a small image ) to adversarial patch that will be added to a natural image to craft an adversarial example , and an attention model is used to locate the attack area on the natural image . Different from the above methods that generate adversarial perturbations or patches , Song et al . ( 2018 ) propose to search a random noise z∗ around the input noise z of AC-GAN ( Odena et al. , 2017 ) such that the corresponding output of AC-GAN is an adversarial example for the target classifier . Their method falls into the search-based adversarial attack , and their crafted examples fall into the unrestricted adversarial examples as there is no original image in their method . AT-GAN falls into the generation-based adversarial attack , and the crafted examples fall into the non-constrained adversarial examples . To clearly distinguish our work , we highlight the differences with most related works as follows : NAG , AdvGAN and AI-GAN vs. AT-GAN . NAG ( Reddy Mopuri et al. , 2018 ) , AdvGAN ( Xiao et al. , 2018 ) and AI-GAN ( Bai et al. , 2020 ) focus on crafting adversarial perturbations by GANs . NAG takes random noise as input and crafts image-agnostic adversarial perturbation . AdvGAN and AI-GAN both use natural images as inputs , and generate the corresponding adversarial perturbations for the input image . AI-GAN uses adversarial examples generated by PGD for the training . In contrast , AT-GAN does not use any natural image as the input , and generates adversarial examples directly from any random noise . Further , compared with AI-GAN , we do not use any adversarial examples for the training . Song ’ s vs. AT-GAN . Song ’ s method ( Song et al. , 2018 ) searches over the neighborhood of the input noise for the pre-trained AC-GAN in order to find a noise whose output image is misclassified by the target classifier . They define such adversaries as the unrestricted adversarial examples , however , their adversaries are still constrained by the original input noise . Their method is essentially based on search , while AT-GAN is trained as an adversarial generative model , and our output is not constrained by any neighborhood .

Summary: The paper proposes AT-GAN (Adversarial Transfer on Generative Adversarial Net) to train an adversarial generative model that can directly produce adversarial examples. Different from previous works, the study aims to learn the distribution of adversarial examples so as to generate semantically meaningful adversaries. AT-GAN achieves this goal by ﬁrst learning a generative model for real data, followed by transfer learning to obtain the desired generative model. Once trained and transferred, AT-GAN could generate adversarial examples directly for any input noise, denoted as non-constrained adversarial examples. Some experiments and visualizations show that AT-GAN can generate some diverse adversarial examples that are realistic to human perception, and yields higher attack success rates against adversarially trained models.