Featurized Bidirectional GAN: Adversarial Defense via Adversarially Learned Semantic Inference

1 INTRODUCTION . The existence of adversarial examples causes serious security concern about reliability of deep neural networks ( DNN ) . DNN may mislabel the perturbed images with high confidence even though the perturbation is too small to be recognized by human . Moreover , adversarial examples will often fool several models simultaneously , even if these models have different architectures ( Szegedy et al. , 2014 ) . One possible explanation is that when recognizing images , human usually catch highlevel and semantic features , such as the shape of the digits in MNIST dataset , which are robust under small perturbation ; DNN may easily catch low-level and weak features , such as the gray-scale values of certain area in the images , which are non-robust when the pixel-wise perturbation accumulates ( Tsipras et al. , 2018 ) . Most previous adversarial defense methods fall into two classes : adversarial training and gradient masking . Adversarial training methods ( Szegedy et al. , 2014 ; Tramèr et al. , 2017 ; Madry et al. , 2017 ; Sinha et al. , 2017 ) apply adversarial perturbations on training data online , and feed both the clean data and the adversarial data to train the classifier , i.e. , solve a minimax game iteratively . However , it is flawed by the high computational cost to generate adversarial examples , especially for more complex dataset and harder attacks . Gradient masking methods modify the architecture of the classifier such that the attacker can not get useful gradient information of the inputs . One example is the thermometer encoding ( Buckman et al. , 2018 ) which preprocesses the input in a one hot vector , and such discretization prevent the attacker from backpropagating through the input to calculate the adversarial purtabation . However , Athalye et al . ( 2018 ) shows that gradient masking methods can be circumvented and lead to a false sense of security in defenses against adversarial attacks . Both of adversarial training and gradient masking methods defend adversarial attacks by improving the classifier . We take another approach by denoising the adversarial examples without changing the classifier ( Meng & Chen , 2017 ; Ilyas et al. , 2017 ; Liao et al. , 2018 ) . Our defense is motivated by human cognition process . The fact that adversarial examples can not fool human suggests that human do classification based on some semantic features that are unchanged after the perturbation . Hence , it is natural to extract those semantic features and doing the inference solely based on semantic information . One closely related work is Defense-GAN ( Samangouei et al. , 2018 ) , which trains a GAN ( Goodfellow et al. , 2014a ) to generate the manifold of unperturbed images , then finds the nearest point on the manifold to the adversarial example as the denoising result . While it is a novel way to leverage generative model to filter the adversarial perturbation , it takes iterations to search the nearest point on the manifold , which is time consuming . In this paper , we propose Featurized Bidirectional GAN ( FBGAN ) , an encoding and generative model that extracts the semantic features of the input images ( either original or perturbed ) , and reconstructs the unperturbed images from these features . We take advantage of the generative capability of Bidirectional GAN ( Donahue et al. , 2016 ; Dumoulin et al. , 2016 ) , where an encoder is learned to map the input to its latent codes directly , instead of doing the manifold search iterations . Inspired by InfoGAN ( Chen et al. , 2016 ) , we maximize the mutual information ( MI ) between all the latent codes and the generated images . The MI regularization can significantly reduce the dimension of latent space , as well as disentangle the semantic features of inputs in different components of the latent codes , e.g. , the tilt angle and stroke thickness of digits in MNIST . We call the MIenhanced latent codes as semantic codes ( Figure 1 ) . FBGAN is pre-trained on the clean dataset in an unsupervised manner . With the feature-extraction and reconstruction procedure , we can denoise the adversarial examples and fed them into any pre-trained classifier , which shows effective defense against both white-box and gray-box attacks ( see section 4 for details ) . Our contribution 1 . FBGAN depicts a bidirectional mapping between a high-dimensional data space and a low-dimensional semantic latent space . We can extract the semantic features of the images , which is unchanged after the adversarial perturbation ; we can also generate new images with indicated semantic features , such as the category and tilt angle of the digits . 2 . We denoise the adversarial example by extracting semantic features and reconstructing via FBGAN . This defense method is shown to be effective for any given pre-trained classifier under both white-box and gray-box attacks . 2 PRELIMINARIES . 2.1 GENERATIVE ADVERSARIAL NETWORKS AND ITS DERIVATIVES . Generative Adversarial Networks GAN ( Goodfellow et al. , 2014a ) is a generative model to learn high-dimensional data distribution via an adversarial process . Instead of modeling the probability density function , GAN learns a generator G which is a mapping from low-dimensional latent space Ωz to high-dimensional data space Ωx . Then a standard distribution ( usually Gaussian ) z ∼ Pz in the latent space can be transferred into the distributionG ( z ) ∼ PG in the data space . PG is supposed to approximate the objective data distribution Px , thus a discriminator D is proposed to distinguish between samples from Px and PG . The generator G and discriminator D are represented by DNN and updated in the following minimax game : min G max D VGAN ( D , G ) : = Ex∼Px [ logD ( x ) ] + Ez∼Pz [ log ( 1−D ( G ( z ) ) ) ] . ( 1 ) It can be shown that the theoretical optimal discriminator D ? satisfies : D ? ( x ) = Px ( x ) Px ( x ) + PG ( x ) , VGAN ( D ? , G ) = 2DJS ( Px‖PG ) − 2 log 2 , ( 2 ) where P ( · ) denotes the probability density of distribution P , and DJS is the Jensen-Shannnon divergence between two distribution . Thus the theoretical optimal generator G ? will recover the data distribution , i.e . PG ? = Px . Bidirectional GAN BiGAN ( Donahue et al. , 2016 ; Dumoulin et al. , 2016 ) considers the inverse mapping of the generator to learn the latent codes z as feature representation given data x . The encoder E is introduced as a mapping from data space Ωx to latent space Ωz , and the discriminator takes a tuple of data point and latent codes ( x , z ) as inputs , distinguishing between the joint distribution of ( x , E ( x ) ) and ( G ( z ) , z ) . The minimax objective becomes min G , E max D VBiGAN ( D , G , E ) : = Ex∼Px [ logD ( x , E ( x ) ) ] + Ez∼Pz [ log ( 1−D ( G ( z ) , z ) ) ] . ( 3 ) The optimal condition for D ? is replacing Px and PG by Px , E ( x ) and PG ( z ) , z in ( 2 ) . The optimal encoder and generator can guarantee G ? ( E ? ( x ) ) = x for x ∈ Ωx and E ? ( G ? ( z ) ) = z for z ∈ Ωz . InfoGAN InfoGAN ( Chen et al. , 2016 ) is an extension of GAN that is able to learn disentangled semantic representation . For example , one discrete latent code may represent the class of the image while another continuous code may control tilt angles . InfoGAN decomposes the latent codes into two parts z = ( c , z′ ) where the semantic codes c target the meaningful features , and noise codes z′ which stand for incompressible noise . Then an information-theoretic regularization is introduced to maximize MI between semantic codes c and generated G ( c , z′ ) : min G max D VInfoGAN ( D , G ) : = Ex∼Px [ logD ( x ) ] +Ez∼Pz [ log ( 1−D ( G ( z ) ) ) ] −λI ( c ; G ( c , z′ ) ) , ( 4 ) where the mutual information I ( c ; x ) = H ( c ) −H ( c|x ) and H is the entropy . 2.2 ADVERSARIAL ATTACKS . In the image classification task , given a vectorized clean image x ∈ [ 0 , 1 ] d , a classifier C will output a label y = C ( x ) . All adversarial attacks aim to find a small perturbation ρ to fool the classifier such that C ( x+ ρ ) 6= y ( Szegedy et al. , 2014 ) . It can be formulated as min ρ ‖ρ‖ , s.t . x+ ρ ∈ [ 0 , 1 ] d , C ( x+ ρ ) 6= y . Various attacking algorithms have been proposed to fool DNN ( Akhtar & Mian , 2018 ; Papernot et al. , 2016 ) , and here are two most famous attacks . Fast Gradient Sign Method FGSM ( Goodfellow et al. , 2014b ) is a single-step attack . LetL ( x , y ) be the loss function of the classifier C given input x and label y. FGSM defines the perturbation ρ as ρ = ε · sign ( ∇xL ( x , y ) ) , where ε is a small scalar . FGSM simply chooses the sign of change at each pixel to increase the loss L ( x , y ) and fool the classifier . Projected Gradient Descent PGD ( Madry et al. , 2017 ) is a more powerful multi-step attack with projected gradient descent : xPGD0 = x , x PGD t+1 = ΠS [ xPGDt + α · sign ( ∇xL ( xPGDt , y ) ) ] where ΠS is the projection onto S = { x′ : ‖x′ − x‖∞ ≤ ε } . 3 FEATURIZED BIDIRECTIONAL GAN . 3.1 ROUTE MAP . We use BiGAN framework to adversarially learn the bidirectional feature mapping , and MI regularization to reduce the dimension of semantic codes and disentangle the semantic features . In adversarial defense task , first we train FBGAN on clean dataset , which is an unsupervised learning for semantic encoder E and image generator G. Second , given a pre-trained classifier C and adversarial data x , we reconstruct x as x̃ = G ( E ( x ) ) to filter the non-semantic noise , then feed x̃ to the classifier and use C ( x̃ ) as the prediction . 3.2 FORMULATION . BiGAN provides a good approach to map high-dimensional image data x to low-dimensional latent codes z = E ( x ) , yet it has no restriction on the semantic meaning of the latent codes z . To eliminate the non-semantic noise in adversarial examples , we maximize mutual information between latent codes z and generated G ( z ) . Unlike InfoGAN where the latent codes is decomposed into semantic codes and incompressible noise z = ( c , z′ ) and only I ( c ; G ( c , z′ ) ) is maximized , here we regard all latent codes as semantic and maximize I ( z , G ( z ) ) directly . Although the former method may improve the diversity of the generation , our method focuses on the main semantic features which is more robust under adversarial attack . To maximize the mutual information I ( z ; G ( z ) ) , we use Variational Information Maximization technique . Suppose the underlying joint distribution is ( x , z ) ∼ P , then I ( z ; x ) = H ( z ) −H ( z|x ) = H ( z ) + EP [ logP ( z|x ) ] = H ( z ) + max Q EP [ logQ ( z|x ) ] , where Q is taken over all possible joint distributions of ( x , z ) . Assume that each semantic codes z contain one categorical code zc and n continuous codes z1 , . . . , zn . Assume that Q ( ·|x ) is a factored distribution Q ( z|x ) = Qc ( zc|x ) ∏n i=1Qi ( zi|x ) . For the categorical code , rewrite the discrete probability Qc ( ·|x ) as a vector ϕc ( x ) , i.e . ϕc ( x ) k = Qc ( zc = k|x ) , then logQc ( zc|x ) = −H ( zc , ϕc ( x ) ) where H is the cross entropy of two vectors regarding zc as a one-hot vector . For the continuous codes , assumeQi ( ·|x ) is a GaussianN ( ϕi ( x ) , σ2 ) for fixed variance σ . Now , define MI gap as the following distance dist ( z , ϕ ( x ) ) : = − logQ ( z|x ) = H ( zc , ϕc ( x ) ) + C n∑ i=1 ‖zi − ϕi ( x ) ‖2 ( 5 ) where ϕ is the concatenation of ( ϕc , ϕ1 , . . . , ϕn ) and C is a constant . Note that the MI gap is a useful approach to maximize MI between two variables . In the defense task , we want to pay more attention to the encoding E ( x ) and reconstruction G ( E ( x ) ) on given data x , and take the pair ( G ( E ( x ) ) , E ( x ) ) into consideration . Therefore , FBGAN has the following objective function ( as illustrated in Figure 1 ) min G , E , ϕ max D VFBGAN ( D , G , E ) : = Ex [ logD ( x , E ( x ) ) ] + 1 2 [ Ez [ log ( 1−D ( G ( z ) , z ) ) ] + Ex [ log ( 1−D ( G ( E ( x ) ) , E ( x ) ) ) ] ] + λEz dist ( z , ϕ ( G ( z ) ) ) . ( 6 ) 3.3 IMPLEMENTATION φ ( x ) Conv 5 layers FC 2 layers Concat FC 3 layers D ( x , z ) x z FC 2 layers Deconv 6 layers Conv 6 layers x E ( z ) G ( x ) ( x ) φc ( x ) φi zc z1 : n DiscriminatorAuxiliary function GeneratorEncoder Figure 2 : Implementation The encoder E ( x ) is a convolutional network and the generatorG ( z ) is a deconvolutional network . The discriminator D ( x , z ) shares parameters with the auxiliary function ϕ ( x ) . z = ( zc , z1 : n ) stands for the categorical and continuous codes . Figure 2 shows the implementation of FBGAN . E , G and D take the standard BiGAN architectures ( Dumoulin et al. , 2016 ) . We replace all ReLU activation with ELU in E and G for smoothness , and use weight normalization instead of batch normalization in order to ensure E ( x ) and G ( z ) depend only on x and z instead of the whole minibatch ( Kumar et al. , 2017 ) . E are trained by feature matching methods , while G and D are trained by the original GAN loss objectives ( Salimans et al. , 2016 ) . The hyperparameter λ = 1 . In relatively complicated dataset such as SVHN , we add an auto-encoder term Ex∼Px‖G ( E ( x ) ) − x‖2 in the objective function for only the last 1 % training steps to further improve the reconstruction quality .

Summary: This paper presents a new adversarial defense based on "cleaning" images using a round trip through a bidirectional gan.  Specifically, an image is cleaned by mapping it to latent space and back to image space using a bidirectional gan.  To encourage the bidirectional gan to focus on the semantic properties, and ignore the noise, the gan is trained to maximize the mutual information between z and x, similar to the info gan.