Disentangling Adversarial Robustness in Directions of the Data Manifold

1 INTRODUCTION . In recent years , deep neural networks ( DNNs ) ( Krizhevsky et al . ( 2012 ) ; Hochreiter and Schmidhuber ( 1997 ) ) have become popular and successful in many machine learning tasks . They have been used in different problems with great success . But DNNs are shown to be vulnerable to adversarial examples ( Szegedy et al . ( 2013 ) ; Goodfellow et al . ( 2014a ) ) . A well-trained model can be easily attacked by adding a small perturbation to the image . An effective way to solve this issue is to train the robust model using training data augmented with adversarial examples , i.e . adversarial training . With the growing success of generative models , researchers have tried to use generative adversarial networks ( GAN ) ( Goodfellow et al . ( 2014b ) ) and variational autoencoder ( VAE ) ( Kingma and Welling ( 2013 ) ) to generate adversarial examples ( Xiao et al . ( 2018 ) ; Zhao et al . ( 2017 ) ; Song et al . ( 2018a ) ; Kos et al . ( 2018 ) ; Song et al . ( 2018b ) ) to fool the classification model with great success . They found that standard adversarial training can not defend these new attacks . Unlike the regular adversarial examples , these new adversarial examples are perceptible by humans but they preserve the semantic information of the original data . A good DNN should be robust to such semantic attacks . Since the GAN and VAE are approximations of the true data distribution , these adversarial examples will stay in the data manifold . Hence they are called on-manifold adversarial examples by ( Stutz et al . ( 2019 ) ) . On the other hand , experimental evidences support that regular adversarial examples leave the data manifold ( Song et al . ( 2017 ) ) . We call the regular adversarial examples as off-manifold adversarial examples . The concepts of on-manifold and off-manifold adversarial examples are important . Because they can help us to understand the issue of conflict between adversarial robustness and generalization ( Stutz et al . ( 2019 ) ; Raghunathan et al . ( 2019 ) ) , which is still an open problem . In this paper , we study the attacking mechanisms of these two types of examples , as well as the corresponding adversarial training methods . This study , as far as we know , has not been done before . Specifically , we consider a generative attack method that adds a small perturbation in the latent space of the generative models . Since standard adversarial training can not defend this attack , we consider the training methods that use training data augmented with these on-manifold adversarial examples , which we call latent space adversarial training . Then we compare it to standard adversarial training ( training with off-manifold adversarial examples ) . Contributions : We study the theoretical properties of latent space adversarial training and standard adversarial training in the Gaussian mixture model with a linear generator . We give the excess risk analysis and saddle point analysis in this model . Based on this case study , we claim that : • Regular adversarial examples attack in directions of small variance of the data manifold and leave the data manifold . • Standard adversarial training increases the model robustness by amplifying the small variance . Hence , it extends the boundary of the data manifold in directions of small variance . • Generative adversarial examples attack in directions of large variance of the data manifold and stay in the data manifold . • Latent space adversarial training increases the model robustness by amplifying the large variance . Hence , it extends the boundary of the data manifold in directions of large variance . We provide experiments on MNIST and CIFAR-10 and show that the above phenomena also exist in real datasets . It gives us a new perspective to understand the behavior of on-manifold and off-manifold adversarial examples . Finally , we study the robustness trade-off between generative and regular adversarial examples . On MNIST , robustness trade-off is unavoidable , but the conflict between generative adversarial examples and regular adversarial examples are much smaller than the conflict between regular adversarial examples of different norms . On CIFAR-10 , there is nearly no robustness trade-off between generative and regular adversarial examples . 2 RELATED WORK . Our work is related to attack and defense methods . Specifically , we care about attacks and defenses with generative models . Attack Adversarial examples for deep neural networks were first intruduced in ( Szegedy et al . ( 2013 ) ) . However , adversarial machine learning or robust machine learning has been studied for a long time ( Biggio and Roli ( 2018 ) ) . In the setting of white box attack ( Kurakin et al . ( 2016 ) ; Papernot et al . ( 2016 ) ; Moosavi-Dezfooli et al . ( 2016 ) ; Carlini and Wagner ( 2017 ) ) , the attackers have fully access to the model ( weights , gradients , etc. ) . In black box attack ( Chen et al . ( 2017 ) ; Su et al . ( 2019 ) ; Ilyas et al . ( 2018 ) ) , the attackers have limited access to the model . First order optimization methods , which use the gradient information to craft adversarial examples , such as PGD ( Madry et al . ( 2017 ) ) , are widely used for white box attack . Zeroth-order optimization methods ( Chen et al . ( 2017 ) ) are used in black box setting . Li et al . ( 2019 ) improved the query efficiency in black-box attack . HopSkipJumpAttack ( Chen et al . ( 2020 ) ) is another query-efficient attack method . Generative adversarial examples Recently , generative models have been used to craft adversarial examples ( Xiao et al . ( 2018 ) ; Song et al . ( 2018b ) ; Kos et al . ( 2018 ) ; Schott et al . ( 2018 ) ) . The adversarial examples are more natural ( Zhao et al . ( 2017 ) ) . These adversarial examples lie in the data manifold , and they are called on-manifold adversarial examples . Defense Training algorithms against adversarial attacks can be subdivided into the following categories . Adversarial training : The training data is augmented with adversarial examples to make the models more robust ( Madry et al . ( 2017 ) ; Szegedy et al . ( 2013 ) ; Tramèr et al . ( 2017 ) ) . Preprocessing : Inputs or hidden layers are quantized , projected onto different sets or other preprocessing methods ( Buckman et al . ( 2018 ) ; ( Guo et al . ( 2017 ) ; Kabilan et al . ( 2018 ) ) . Stochasticity : Inputs or hidden activations are randomized ( Prakash et al . ( 2018 ) ; Dhillon et al . ( 2018 ) ; Xie et al . ( 2017 ) ) . However , some of them are shown to be useless defenses given by obfuscated gradients ( Athalye et al . ( 2018 ) ) . Adaptive attack ( Tramer et al . ( 2020 ) ) is used for evaluating defenses to adversarial examples . Defense with generative model Using generative models to design defense algorithms have been studied extensively . Using GAN , we can project the adversarial examples back to the data manifold ( Jalal et al . ( 2017 ) ; Samangouei et al . ( 2018 ) ) . VAE is also used to train robust model ( Schott et al . ( 2018 ) ) . 3 PROBLEM DESCRIPTION . Original space adversarial training : Consider the classification problem of training a classifer fθ to map the data points x ∈ X ⊂ Rd to the labels y ∈ Y , where X and Y are the input data space and the label space . The classifier fθ is parameterized by θ . We assume that the data pairs ( x , y ) are sampled from the distribution P ( X , Y ) over X × Y . Standard training is to find the solution of minθ E ( x , y ) ∼P ` ( fθ ( x ) , y ) , where ` ( · , · ) is the loss function . The goal of dversarial training is to solve the minimax problem min θ E ( x , y ) ∼P max ‖x−x′‖≤ε ` ( fθ ( x ′ ) , y ) , ( 1 ) where ε is the threshold of perturbation . Here we can use ` 1 , ` 2 or ` ∞-norm ( Madry et al . ( 2017 ) ) . The inner maximization problem is to find the adversarial examples x′ to attack the given classifier fθ . The outer minimization problem is to train the classifier to defend the given adversarial examples x′ . we refer to these attacks as regular attacks . We refer to these minimax problems as standard adversarial training or original space adversarial training . Latent space adversarial training : We assume that the data lie in a low dimensional manifold of Rd . Furthermore , we assume the true distribution D is a pushforward from a prior Guassian distribution z ∼ N ( 0 , I ) using G ( z ) , where G : Z → X is a mapping from the latent space Z to the original space X . This is a basic assumption of GAN or VAE . Let I : X → Z be the inverse mapping of G ( z ) . The goal of latent space adversarial training is to solve the following minimax problem min θ E ( x , y ) ∼P max ‖z′−I ( x ) ‖≤ε ` ( fθ ( G ( z ′ ) ) , y ) . ( 2 ) Unlike the regular attacks , the distance between the original examples and adversarial examples can be large . To preserve the label of the data , we use the conditional generative models ( e.g . C-GAN ( Mirza and Osindero ( 2014 ) ) and C-VAE ( Sohn et al . ( 2015 ) ) ) , i.e . the generator Gy ( z ) and inverse mapping Iy ( x ) are conditioned on the label y , for adversarial training . We refer to these attacks as generative attack , and these adversarial training as latent space adversarial training . Regular attack algorithms Two widely used gradient-based attack algorithms for the inner maximization problem in equation ( 1 ) are fast gradient sign method ( FGSM ) ( Goodfellow et al . ( 2014a ) ) and projected gradient descend ( PGD ) ( Madry et al . ( 2017 ) ) . Using FGSM , the adversarial examples are calculated by x′ = x+ εsgn ( ∇x ` ( fθ ( x ) , y ) ) , where ∇x denotes the gradient with respect to x. PGD attempts to find a near optimal adversarial example for the inner maximization problem ( 1 ) in multiple steps . In the tth step , xt+1 = Πx+S [ x t + α∇x ` ( fθ ( xt ) , y ) /‖∇x ` ( fθ ( xt ) , y ) ‖ ] , where α is the step size , Πx+S [ · ] is the projection operator to project the given vector to the constraint x+ S = { x′|‖x− x′‖ ≤ ε } . In the whole paper , we refer to these as FGSM-attack and PGD-attack , and the corresponding original space adversarial training as FGSM-adv and PGD-adv . FGSM-attack is a weak attack and PGD-attack is a stronger attack . In section 5 , we use them to show that a strong original space adversarial training , PGD-adv , does not work well against a weak attack , FGSM-attack in the latent space . Conversely , latent space adversarial training can not defend a simple FGSM-attack in the original space . Generative attack algorithm In our experiments , we use FGSM in the latent space for the inner maximization problem in equation ( 2 ) z′ = I ( x ) + εsgn ( ∇z ` ( fθ ( G ( z ) ) , y ) ) . Because of the mode collapse issue of GAN ( Salimans et al . ( 2016 ) ; Gulrajani et al . ( 2017 ) ) , adding a small perturbation in the latent space of GAN may output the same images . Thus we use VAE in our experiments . we refer to this generative attack and latent space adversarial training as VAE-attack and VAE-adv .

Summary: This paper analytically considers two flavours of adversarial training in a Gaussian mixture model.  The first uses regular adversarial examples, and the second uses examples drawn from a generative model.  The authors show that the adversarial perturbations generated in the two cases differ in a cleanly-characterisable  way: in the first case the perturbations differ from real data in a direction aligned with the smallest eigenvalues  of the data covariance. In the latter case the perturbations are in a direction aligned with the largest eigenvalues.  Experimental results on MNIST and CIFAR are presented to illustrate how the analysis transfers to real datasets.