Year: 2020

Abstract:
Using generative models (GAN or VAE) to craft adversarial examples, i.e. generative adversarial examples, has received increasing attention in recent years. Previous studies showed that the generative adversarial examples work differently compared to that of the regular adversarial examples in many aspects, such as attack rates, perceptibility, and generalization. But the reasons causing the differences between regular and generative adversarial examples are unclear. In this work, we study the theoretical properties of the attacking mechanisms of the two kinds of adversarial examples in the Gaussian mixture model. We prove that adversarial robustness can be disentangled in directions of the data manifold. Specifically, we find that: 1. Regular adversarial examples attack in directions of small variance of the data manifold, while generative adversarial examples attack in directions of large variance. 2. Standard adversarial training increases model robustness by extending the data manifold boundary in directions of small variance, while on the contrary, adversarial training with generative adversarial examples increases model robustness by extending the data manifold boundary directions of large variance. In experiments, we demonstrate that these phenomena also exist on real datasets. Finally, we study the robustness trade-off between generative and regular adversarial examples. We show that the conflict between regular and generative adversarial examples is much smaller than the conflict between regular adversarial examples of different norms.

heading: 1 INTRODUCTION
text: In recent years, deep neural networks (DNNs) (Krizhevsky et al. (2012); Hochreiter and Schmidhuber (1997)) have become popular and successful in many machine learning tasks. They have been used in different problems with great success. But DNNs are shown to be vulnerable to adversarial examples (Szegedy et al. (2013); Goodfellow et al. (2014a)). A well-trained model can be easily attacked by adding a small perturbation to the image. An effective way to solve this issue is to train the robust model using training data augmented with adversarial examples, i.e. adversarial training.
With the growing success of generative models, researchers have tried to use generative adversarial networks (GAN) (Goodfellow et al. (2014b)) and variational autoencoder (VAE) (Kingma and Welling (2013)) to generate adversarial examples (Xiao et al. (2018); Zhao et al. (2017); Song et al. (2018a); Kos et al. (2018); Song et al. (2018b)) to fool the classification model with great success. They found that standard adversarial training cannot defend these new attacks. Unlike the regular adversarial examples, these new adversarial examples are perceptible by humans but they preserve the semantic information of the original data. A good DNN should be robust to such semantic attacks. Since the GAN and VAE are approximations of the true data distribution, these adversarial examples will stay in the data manifold. Hence they are called on-manifold adversarial examples by (Stutz et al. (2019)). On the other hand, experimental evidences support that regular adversarial examples leave the data manifold (Song et al. (2017)). We call the regular adversarial examples as off-manifold adversarial examples.
The concepts of on-manifold and off-manifold adversarial examples are important. Because they can help us to understand the issue of conflict between adversarial robustness and generalization (Stutz et al. (2019); Raghunathan et al. (2019)), which is still an open problem. In this paper, we study the attacking mechanisms of these two types of examples, as well as the corresponding adversarial
training methods. This study, as far as we know, has not been done before. Specifically, we consider a generative attack method that adds a small perturbation in the latent space of the generative models. Since standard adversarial training cannot defend this attack, we consider the training methods that use training data augmented with these on-manifold adversarial examples, which we call latent space adversarial training. Then we compare it to standard adversarial training (training with off-manifold adversarial examples).
Contributions: We study the theoretical properties of latent space adversarial training and standard adversarial training in the Gaussian mixture model with a linear generator. We give the excess risk analysis and saddle point analysis in this model. Based on this case study, we claim that:
• Regular adversarial examples attack in directions of small variance of the data manifold and leave the data manifold.
• Standard adversarial training increases the model robustness by amplifying the small variance. Hence, it extends the boundary of the data manifold in directions of small variance.
• Generative adversarial examples attack in directions of large variance of the data manifold and stay in the data manifold.
• Latent space adversarial training increases the model robustness by amplifying the large variance. Hence, it extends the boundary of the data manifold in directions of large variance.
We provide experiments on MNIST and CIFAR-10 and show that the above phenomena also exist in real datasets. It gives us a new perspective to understand the behavior of on-manifold and off-manifold adversarial examples. Finally, we study the robustness trade-off between generative and regular adversarial examples. On MNIST, robustness trade-off is unavoidable, but the conflict between generative adversarial examples and regular adversarial examples are much smaller than the conflict between regular adversarial examples of different norms. On CIFAR-10, there is nearly no robustness trade-off between generative and regular adversarial examples.

heading: 2 RELATED WORK
text: Our work is related to attack and defense methods. Specifically, we care about attacks and defenses with generative models.
Attack Adversarial examples for deep neural networks were first intruduced in (Szegedy et al. (2013)). However, adversarial machine learning or robust machine learning has been studied for a long time (Biggio and Roli (2018)). In the setting of white box attack (Kurakin et al. (2016); Papernot et al. (2016); Moosavi-Dezfooli et al. (2016); Carlini and Wagner (2017)), the attackers have fully access to the model (weights, gradients, etc.). In black box attack (Chen et al. (2017); Su et al. (2019); Ilyas et al. (2018)), the attackers have limited access to the model. First order optimization methods, which use the gradient information to craft adversarial examples, such as PGD (Madry et al. (2017)), are widely used for white box attack. Zeroth-order optimization methods (Chen et al. (2017)) are used in black box setting. Li et al. (2019) improved the query efficiency in black-box attack. HopSkipJumpAttack (Chen et al. (2020)) is another query-efficient attack method.
Generative adversarial examples Recently, generative models have been used to craft adversarial examples (Xiao et al. (2018); Song et al. (2018b); Kos et al. (2018); Schott et al. (2018)). The adversarial examples are more natural (Zhao et al. (2017)). These adversarial examples lie in the data manifold, and they are called on-manifold adversarial examples.
Defense Training algorithms against adversarial attacks can be subdivided into the following categories. Adversarial training: The training data is augmented with adversarial examples to make the models more robust (Madry et al. (2017); Szegedy et al. (2013); Tramèr et al. (2017)). Preprocessing: Inputs or hidden layers are quantized, projected onto different sets or other preprocessing methods (Buckman et al. (2018); (Guo et al. (2017); Kabilan et al. (2018)). Stochasticity: Inputs or hidden activations are randomized (Prakash et al. (2018); Dhillon et al. (2018); Xie et al. (2017)). However, some of them are shown to be useless defenses given by obfuscated gradients (Athalye et al. (2018)). Adaptive attack (Tramer et al. (2020)) is used for evaluating defenses to adversarial examples.
Defense with generative model Using generative models to design defense algorithms have been studied extensively. Using GAN, we can project the adversarial examples back to the data manifold
(Jalal et al. (2017); Samangouei et al. (2018)). VAE is also used to train robust model (Schott et al. (2018)).

heading: 3 PROBLEM DESCRIPTION
text: Original space adversarial training: Consider the classification problem of training a classifer fθ to map the data points x ∈ X ⊂ Rd to the labels y ∈ Y , where X and Y are the input data space and the label space. The classifier fθ is parameterized by θ. We assume that the data pairs (x, y) are sampled from the distribution P (X,Y ) over X × Y . Standard training is to find the solution of minθ E(x,y)∼P `(fθ(x), y), where `(·, ·) is the loss function. The goal of dversarial training is to solve the minimax problem
min θ E(x,y)∼P max ‖x−x′‖≤ε
`(fθ(x ′), y), (1)
where ε is the threshold of perturbation. Here we can use `1, `2 or `∞-norm (Madry et al. (2017)). The inner maximization problem is to find the adversarial examples x′ to attack the given classifier fθ. The outer minimization problem is to train the classifier to defend the given adversarial examples x′. we refer to these attacks as regular attacks. We refer to these minimax problems as standard adversarial training or original space adversarial training.
Latent space adversarial training: We assume that the data lie in a low dimensional manifold of Rd. Furthermore, we assume the true distribution D is a pushforward from a prior Guassian distribution z ∼ N (0, I) using G(z), where G : Z → X is a mapping from the latent space Z to the original space X . This is a basic assumption of GAN or VAE. Let I : X → Z be the inverse mapping of G(z). The goal of latent space adversarial training is to solve the following minimax problem
min θ E(x,y)∼P max ‖z′−I(x)‖≤ε
`(fθ(G(z ′)), y). (2)
Unlike the regular attacks, the distance between the original examples and adversarial examples can be large. To preserve the label of the data, we use the conditional generative models (e.g. C-GAN (Mirza and Osindero (2014)) and C-VAE (Sohn et al. (2015))), i.e. the generator Gy(z) and inverse mapping Iy(x) are conditioned on the label y, for adversarial training. We refer to these attacks as generative attack, and these adversarial training as latent space adversarial training.
Regular attack algorithms Two widely used gradient-based attack algorithms for the inner maximization problem in equation (1) are fast gradient sign method (FGSM) (Goodfellow et al. (2014a)) and projected gradient descend (PGD) (Madry et al. (2017)). Using FGSM, the adversarial examples are calculated by x′ = x+ εsgn(∇x`(fθ(x), y)), where ∇x denotes the gradient with respect to x. PGD attempts to find a near optimal adversarial example for the inner maximization problem (1) in multiple steps. In the tth step,
xt+1 = Πx+S [x t + α∇x`(fθ(xt), y)/‖∇x`(fθ(xt), y)‖],
where α is the step size, Πx+S [·] is the projection operator to project the given vector to the constraint x+ S = {x′|‖x− x′‖ ≤ ε}. In the whole paper, we refer to these as FGSM-attack and PGD-attack, and the corresponding original space adversarial training as FGSM-adv and PGD-adv. FGSM-attack is a weak attack and PGD-attack is a stronger attack. In section 5, we use them to show that a strong original space adversarial training, PGD-adv, does not work well against a weak attack, FGSM-attack in the latent space. Conversely, latent space adversarial training cannot defend a simple FGSM-attack in the original space.
Generative attack algorithm In our experiments, we use FGSM in the latent space for the inner maximization problem in equation (2)
z′ = I(x) + εsgn(∇z`(fθ(G(z)), y)). Because of the mode collapse issue of GAN (Salimans et al. (2016); Gulrajani et al. (2017)), adding a small perturbation in the latent space of GAN may output the same images. Thus we use VAE in our experiments. we refer to this generative attack and latent space adversarial training as VAE-attack and VAE-adv.

heading: 4 THEORETICAL ANALYSIS
text: In this section, we study the difference between adversarial training in the latent space and in the original space. We study the simple binary classification setting proposed by Ilyas et al. (2019). The main reason for using this model is that we can find the optimal closed-form solution, which gives us insights to understand adversarial training. For a more complex model, we can only solve it numerically. We leave all the proofs of the lemmas and theorems in appendix A.

heading: 4.1 THEORETICAL MODEL SETUP
text: Gaussian mixture (GM) model Assume that data points (x, y) are sampled according to y ∼ {−1, 1} unifomly and x ∼ N (yµ∗,Σ∗), where µ∗ and Σ∗ denote the true mean and covariance matrix of the data distribution. For the data in the class y = −1, we replace x by −x, then we can view the whole dataset as sampled from D = N (µ∗,Σ∗).
Classifier The goal of standard training is to learn the parameters Θ = (µ,Σ) such that
Θ = arg min µ,Σ L(µ,Σ) = arg min µ,Σ Ex∼D[`(x;µ,Σ)], (3)
where `(·) represents the negative log-likelihood function. The goal of adversarial training is to find
Θr = arg min µ,Σ Lr(µ,Σ) = arg min µ,Σ Ex∼D[ max ‖x−x′‖≤ε `(x′;µ,Σ)]. (4)
We use L and Lr to denote the standard loss and adversarial loss. After training, we classify a new data point x to the class sgn(µTΣ−1x).
Generative model In our theoretical study, we use a linear generative model, that is, probabilistic principle components analysis (P-PCA) (Tipping and Bishop (1999)). P-PCA can be viewed as linear GAN (Feizi et al. (2017); Feizi et al. (2020)) and linear VAE (Dai et al. (2017)).
Given dataset {xi}ni=1 ⊂ Rd, let µ and S be the sample mean and sample covariance matrix. The eigenvalue decomposition of S is S = UΛUT , then using the first q eigenvactors, we can project the data to a low dimensional space. P-PCA is to assume that the data are generated by
x = W z + µ+ where z ∼ N (0, I) , ∼ N (0, σ2I),
z ∈ Rq and W ∈ Rd×q. Then we have x ∼ N (µ,WW T + σ2I), x|z ∼ N (W z + µ, σ2I) and z|x ∼ N (P−1W T (x−µ), σ2P−1) whereP = W TW +σ2I . The maximum likelihood estimator ofW and σ2 are
WML = Uq(Λq − σ2MLI)1/2 and σ2ML = 1
d− q d∑ i=q+1 λi,
where Uq is the matrix of the first q columns of U , Λq is the matrix of the first q eigenvalues of Λ. In the following study, we assume that n is large enough such that we can learn the true µ∗ and Σ∗. Thus we have S = Σ∗, Uq = Uq∗, Λq = Λq∗ for the generative model.

heading: 4.2 MINIMAX PROBLEM OF LATENT SPACE ADVERSARIAL TRAINING
text: To perturb the data in the latent space, Data will go through the encode-decode process x→ z → ∆z + z → x′. Based on the probabilistic model, we may choose z with the highest probability or just sample it from the distribution we learned. Hence, we could have different strategies. Below we list 3 strategies. Strategy 1 is used in practice and the other two are alternative choices. In lemma 1 we show that these strategies are equivalent under the low dimensional assumption. Hence, we do not need to worry about the effect of sampling strategies.
Strategy 1: Sample x ∼ D, encode z = arg max q(z|x) = P−1W T (x − µ∗), add a perturbation ∆z, and finally, decode xadv = arg max p(x|z + ∆z) = W (z + ∆z) + µ∗. Strategy 2: Sample x ∼ D, then sample z ∼ q(z|x), add a perturbation ∆z, and finally, sample xadv ∼ p(x|z + ∆z).
Strategy 3: Sample z ∼ N (0, I), add a perturbation ∆z, and then sample xadv ∼ p(x|z + ∆z). In this strategy, xadv can be viewed as the adversarial example of x = arg maxx q(z|x). The following lemma shows that the adversarial examples can be unified in one formula. Hence sampling strategies will not affect our analysis. Lemma 1 (Adversarial examples perturbed in the latent space). Using these 3 strategies, the adversarial examples can be unified as
xadv = x ′ +W∆z and x′ ∼ D′j = N (µ∗,U∗Λ(j)UT∗ ), j = 1, 2, 3, (5)
where
Λ(1) =
[ (Λq − σ2I)2Λ−1q 0
0 0
] , Λ(3) = [ Λq 0 0 σ2I ] ,
Λ(2) =
[ (Λq − σ2I)2Λ−1q + (Λq − σ2I)Λ−1q σ2 + σ2I 0
0 σ2I
] .
If the data lie in a q dimensional subspace, i.e. the covariance matrix Σ∗ is rank q, we have Λ(1) = Λ(2) = Λ(3) = Λ∗. Then D′ = D.
In general, the adversarial example can be decomposed into 2 parts, the change of distribution x′ ∼ D′ and the small perturbationW∆z. Therefore the adversarial expected risk can be written as the following minimax problem
min µ,Σ Lls(µ,Σ;D′j) = min µ,Σ Ex′∼D′j max‖∆z‖≤ε `(x′ +W∆z,µ,Σ), j = 1, 2, 3. (6)
We aim to analyze the different properties between the minimax problems in equations (4) and (6). We give the excess risk and optimal saddle point analysis in the following subsections.

heading: 4.3 EXCESS RISK ANALYSIS
text: We consider the difference between Lls and L given the true Θ∗, i.e. Lls(Θ∗;D′j)−L(Θ∗;D). It characterizes the excess risk incured by the optimal perturbation. To derive the expression of excess risk, we decompose it into two parts
Lls(Θ∗;D′j)− L(Θ∗;D) = Lls(Θ∗;D′j)− L(Θ∗;D′j)︸ ︷︷ ︸ perturbation +L(Θ∗;D′j)− L(Θ∗;D)︸ ︷︷ ︸ change of distribution . (7)
To simplify the notation, we consider the Lagrange penalty form of the inner maximization problem in equation (6), i.e. max `(x′+W∆z,µ,Σ)−L‖∆z‖2/2, where L is the Lagrange multiplier. The following theorem gives the solution in the general case. Theorem 2 (Excess risk). Let Lls and L be the loss with and without perturbation in latent space (equations (6) and (3) respectively), given the non-robustly learned Θ∗ = (µ∗,Σ∗), thus the excess risk caused by perturbation is
Lls(Θ∗,D′j)− L(Θ∗,D′j) = 1
2 q∑ i=1 [ (1 +
λi − σ2
(L− 1)λi + σ2 )2 − 1 ]λ(j)i λi , j = 1, 2, 3.
and the excess risk caused by the changed of distribution is
L(Θ∗,D′j)− L(Θ∗,D) = 1 2 log [∏d i=1 λ (j) i∏d
i=1 λi
] + 1
2 ( d∑ i=1 λ (j) i λi − d ) .
It is hard to see which part dominates the excess risk. If we further assume that the data lie in a q dimensional manifold, the excess risk caused by the change of distribution becomes 0 by Lemma 1. We have the following corollary. Corollary 3 (Excess risk). Let Lls and L be the loss with or without perturbation in latent space (equation (6) and (3) respectively), given the non-robustly learned Θ∗ = (µ∗,Σ∗), and rank(Σ∗) = q. The excess risk
Lls(Θ∗,D′j)− L(Θ∗,D) = O(qL−2).
The optimal perturbation in the latent space will incur an excess risk in O(qL−2). The adversarial vulnerability depends on the dimension q and the Lagrange multiplier L. It does not depend on the shape of the data manifold. This is because the perturbation constraint (the black block) aligns with the shape of the data manifold (the ellipse) as we demonstrate in Figure 1 (c). Thus generative attacks will focus on the directions of the largest q variance.
Then, we analyze the excess risk of original space adversarial training. Since the perturbation thresholds, ε, are on different scales in the original space attack and latent space attack, the corresponding Lagrange multipliers L are different. We use L′ for original space adversarial training in the following Theorem.
Theorem 4 (Excess risk of original space adversarial training). Let Lr and L be the loss with or without perturbation in original space (equations (4) and (3) respectively), given the non-robustly learned Θ∗ = (µ∗,Σ∗). Denote λmin be the smallest eigenvalue of Σ∗. The excess risk is
Ω((λminL ′)−2) ≤ Lr(Θ∗,D)− L(Θ∗,D) ≤ O(d(λminL′)−2).

heading: If the data lie in a low dimensional manifold, i.e. λmin = 0, the excess risk equals to +∞.
text: Optimal perturbation in the original space will incur an excess risk in O(d(λminL′)−2). The adversarial vulnerability depends on the smallest eigenvalues λmin, the dimension d, and the Lagrange multiplier L′. The λmin comes from the misalignment between the perturbation constraint (the black block) and the shape of the data manifold (the ellipse) as we demonstrate in Figure 1 (a). Notice that λmin also appears in the lower bound. Hence, the excess risk equals to +∞ when λmin = 0. Thus regular attacks focus on the directions of small variance. Specifically, when λmin = 0, regular adversarial examples leave the data manifold.

heading: 4.4 SADDLE POINT ANALYSIS
text: In this subsection we study the optimal solution of optimization problem (6). Since it is not a standard minimax problem, we consider a modified problem:
min µ,Σ max Ex′‖∆z‖=ε
Ex′∼D′j `(x ′ +W∆z,µ,Σ), j = 1, 2, 3. (8)
We will explain more about the connection between optimization problems in equation (6) and (8). See appendix A. The following theorem is our main result. It gives the optimal solution of latent space adversarial training.
Theorem 5 (Main result: Optimal Saddle point). The optimal solution of the modified problem in equation (8) is
µls = µ∗ and Σls = U∗ΛlsUT∗ ,
where
λlsi = 1
4
[ 2λ
(j) i + 4(λi − σ2) L + 2λ (j) i
√ 1 +
4(λi − σ2) λ
(j) i L
] for i = 1 ≤ q, λlsi = λ (j) i for i > q,
and j = 1, 2, 3 corresponding to strategies 1,2 and 3.
We assume that the data lie in a q-dimensional manifold again. Then we have λlsi /λi = 1/2 + 1/L+√ 1/4 + 1/L ≥ 1 for i ≤ q and λlsi /λi = 0 for i > q. Latent space adversarial training increases the model robustness by amplifying large eigenvalues of the data manifold. The illustration of the two dimensional case is in Figure 1, (c) and (d).
In the same setting, the optimal solution of standard adversarial training (Problem (4)) on the original space is in Theorem 6 (which is Theorem 2 in Ilyas et al. (2019)).
Theorem 6 (Optimal saddle point Ilyas et al. (2019)). The optimal solution of the problem in equation (4) is
µr = µ∗ and Σr = 1
2 Σ∗ +
1
L′ I +
√ 1
L′ Σ∗ +
1 4 Σ2∗.
Theorem 6 is for the problem that the covariance matrix is restricted to be diagonal. Consider the ratio
λ (r) i
λi =
1 2 + 1 L′λi +
√ 1
L′λi +
1 4 .
For small true eigenvalue λi, the ratio is large. Standard adversarial training increases the robustness by amplifying the small eigenvalues of the data manifold. The illustration of the two dimensional case in Figure 1, (a) and (b).

heading: 5 EXPERIMENTS
text: In this section we report our experimental results on training LeNet on MNIST (LeCun et al. (1998)) and ResNet (He et al. (2016)) on CIFAR-10 (Krizhevsky et al. (2009)) to confirm our theoretical findings. Both of the two datasets contain 50000 training samples and 10000 test samples. Details of the hyperparameters setting are in appendix B.

heading: 5.1 EIGENVALUES OF MNIST WITH AND WITHOUT ADVERSARIAL TRAINING
text: In this subsection, we show that the properties of the eigenvalues are also valid on the real dataset. We use MNIST as an example. Firstly, for each image in the test set Dtest of MNIST, we vectorize it into a 784 dimension vector. Then we calculate the EVD (Eigenvalue decomposition) of the covariance matrix of each class. We plot the 784 eigenvalues of class 0 and 1 in Figure 2 in blue line as examples (the other 8 classes are shown in Appendix B.3). Secondly, we adversarially train a classifier fr(x) in original space. For each point x in Dtest, we construct xr by PGD-attack with ∇xfr(x). Then we obtain a robust test set Drtest. We plot the intra class eigenvalues of the covariance matrix in Figure 2 in orange line. Lastly, we adversarially train a classifier fls(x) in latent space. Then we construct latent space robust dataset Dlstest by generative attack against fls(x) on Dtest. The eigenvalues of the covariance matrix are shown in Figure 2 in green line. We show all the eigenvalues in the first column in Figure 2. After adding a small perturbation in the original space or in the latent space, the distribution of Drtest and Dlstest are close to Dtest. We show the details of the figure of large eigenvalues (the first 30 eigenvalues) in the second column and small eigenvalues (last 754 eigenvalues) in the last column.
Original space adversarial training focuses on the small variance direction We adversarially train a robust classifier fr(x) in the original space. The robust test set Drtest against fr(x) amplified the small eigenvalues a lot. In the third column, the orange line is significantly larger that the other 2 lines. While in the second column, the orange line is below the other two lines. The experiments give us an understanding of how the adversarial examples leave the data manifolds. The original dataset Dtest lies in a low dimension affine plane in R784. After adversarial training, the data move towards the small variance directions. Finally Drtest stay in a full dimension subspace.
Latent space adversarial training focuses on the large variance direction We adversarially train a robust classifier fls(x) in the latent space. The latent space robust test set Dlstest against fls(x) amplified the large eigenvalues. In the second column, we can see that the green line is above the other two lines. And in the last column, the green line shows that adversarial training in the latent space will not affect the small eigenvalues. The adversarial examples in Dlstest move along the large variance direction. Therefore, Dlstest stay in the low dimension affine space.

heading: 5.2 ROBUST TEST ACCURACIES
text: In this subsection, we compare the test accuracy of different attacks (FGSM, PGD, VAE) versus different defense (adversarial training with FGSM, PGD, and VAE) to help explain our theoretical results. We explain the results of MNIST in Table 1 as an example.
On-manifold and off-manifold adversarial examples The test accuracies of the standard training model on PGD and VAE-attack data are 3.9% and 42.4% respectively. Firstly, They show that on-manifold adversarial examples exist. Then, we can see that on-manifold adversarial examples are harder to find.
Attack versus defense Our theory tells us that original or latent space adversarial training increase the model robustness by amplifying the small or large eigenvalues, which are the variance of the distribution, in the data manifold. In experiments, the test accuracy of VAE-adv vs PGD-attack is 1.23%, which shows that extending the manifold boundary in directions of large variance gives no contribution to defending attacks in directions of small variance. Similarly, the test accuracy of PGD-adv vs VAE-attack is 52.18%, which shows that amplifying the boundary in the directions of small variance does not work well in defending attacks in directions of high variance. Further more, we can see that latent space adversarial training does not work well against a simple FGSM attack, and vice versa.
We see that PGD-adv can increase the test accuracies on VAE-attack from 42% to 52% on MNIST and from 19.40% to 26.31% on CIFAR-10. A possible reason is that original space adversarial training can amplify the small eigenvalues of the first q dimension but fails to amplify the larger ones. As it is indicated in Figure 2, both original space and latent space adversarial training will increase the eigenvalues which are small but not equal to zero (around the 100th eigenvalue) of the covariance matrix.

heading: 5.3 ROBUSTNESS TRADE-OFF
text: Robustness trade-off are common in practice (Su et al. (2018); Tramèr and Boneh (2019)). For example, `1 and `∞ adversarial examples cannot be defended simultaneously. The experiments on robustness trade-off are shown in Table 2. On MNIST, the jointly-trained model using both `1 and `∞ adversarial examples will decrease the robust test accuracy by 20% comparing to the single trained model. On CIFAR-10, the jointly-trained model will decrease the test accuracies by 6% on `1-attack and by 10% on `∞-attack. A possible reason is that they are all original space attacks and focus on
the same directions of small variance. Hence they conflict with each other. We study the robustness trade-off between regular and generative adversarial examples in this subsection.
Our theory suggests that adversarial robustness can be disentangled in different directions. Hence adversarial robustness against attacks on directions of low variance and large variance can be guaranteed simultaneously. On MNIST, the jointly-trained model decreases the test accuracies by 6% comparing to the single trained model. The conflict between on-manifold and off-manifold adversarial examples is much small than the conflict between off-manifold adversarial examples of different norms. On CIFAR-10, the jointly-trained model gets test accuracies 74% on PGD-attack and 42% on VAE-attack, which exhibit nearly no robustness trade-off. Therefore, if our goal is to defend mixture of regular and generative attacks, The jointly-train model will perform well.
Discussion Under the q-dimensional manifold assumption, there is an overlap between the directions of the q largest variance and the directions of small variance. This is supported by Theorem 5, Theorem 6, and the first experiment. A possible reason for the robustness trade-off between regular and generative attacks is that they conflict with each other in the overlap directions. The conflict is not serious because they mainly focus on different directions.

heading: 6 CONCLUSION
text: In this paper, we show that adversarial robustness can be disentangled in directions of small variance and large variance of the data manifold. Theoretically, we study the excess risk and optimal saddle point of the minimax problem of latent space adversarial training. Experimentally, we show that these phenomena also exist in real datasets.
Future works One may design defense algorithms based on this property. We can generate adversarial examples based on the directions of the dataset without access to the model architecture. In white-box settings, we can use them for data augmentation to accelerate adversarial training or even increase the model robustness. However, we need to carefully compare the computational cost between calculating the EVD of the datasets and that of standard adversarial training. In black-box attacks, we can use them to query the target model. But the data manifolds should be transferred from some known datasets (since there is no information available for the training dataset of the target model). Theoretically, it is unclear whether we can find the closed-form solution under nonlinear models. How to analyze the nonlinear model is an open problem. Using linear models, we can further analyze the conflict between robustness and generalization.

heading: OVERVIEW
text: In appendix A, we provide the proof of the Theorems. In appendix B, we show the settings about the experiments. Appendix C is a further discussion of data augmentation using generative models. It is not closely related to our main paper.

heading: A PROOF OF THE THEOREMS

heading: A.1 PROBLEM DESCRIPTION
text: Lemma 1 (Adversarial examples perturbed in the latent space). Using these 3 strategies, the adversarial examples can be unified as
xadv = x ′ +W∆z and x′ ∼ D′j = N (µ∗,U∗Λ(j)UT∗ ), j = 1, 2, 3,
where
Λ(1) =
[ (Λq − σ2I)2Λ−1q 0
0 0
] Λ(3) = [ Λq 0 0 σ2I ]
Λ(2) =
[ (Λq − σ2I)2Λ−1q + (Λq − σ2I)Λ−1q σ2 + σ2I 0
0 σ2I
] .
If the data lie in a q dimensional subspace, i.e. the covariance matrix Σ∗ is rank q, we have Λ(1) = Λ(2) = Λ(3) = Λ∗. Then D′ = D.
Proof:
Recall that x ∼ N (µ,WW T + σ2I), x|z ∼ N (W z + µ, σ2I) and z|x ∼ N (P−1W T (x − µ), σ2P−1) where P = W TW + σ2I . The maximum likelihood estimator ofW and σ2 are
WML = Uq(Λq − σ2I)1/2 and σ2ML = 1
d− q d∑ i=q+1 λi.
Strategy 1 Sample x ∼ D, encode z = arg max q(z|x) = P−1W T (x− µ∗), add a perturbation ∆z, and finally, decode xadv = arg max p(x|z + ∆z) = W (z + ∆z) + µ∗. Then
xadv =W (P −1W T (x− µ∗) + ∆z) + µ∗
=WP−1W T (x− µ∗) + µ∗ +W∆z =x′ +W∆z.
Since x ∼ (µ∗,Σ∗), we have x− µ∗ ∼ (0,Σ∗), Then
x′ ∼ N (µ∗,WP−1W TΣ∗(WP−1W T )T ),
With WP−1W TΣ∗(WP −1W T )T
=U∗
[ Λq − σ2 0
0 0 ]1/2 [ Λq 0 0 σ2I ]−1 [ Λq − σ2 0 0 0 ]1/2 Λ∗[
Λq − σ2 0 0 0 ]1/2 [ Λq 0 0 σ2I ]−1 [ Λq − σ2 0 0 0 ]1/2 UT∗
=U∗
[ (Λq − σ2I)2Λ−1q 0
0 0
] UT∗
=U∗Λ (j)UT∗ , j = 1.
Strategy 2 Sample x ∼ D, then sample z ∼ q(z|x), add a perturbation ∆z, and finally, sample xadv ∼ p(x|z + ∆z). Then
z ∼ N (0,P−1W TΣ∗(P−1W T )T + σ2P−1)
and
xadv ∼ N (µ∗ +W∆z,WP−1W TΣ∗(P−1W T )TW T +Wσ2P−1W T + σ2I),
xadv = x ′ +W∆z,
With WP−1W TΣ∗(P −1W T )TW T +Wσ2P−1W T + σ2I
=U∗
[ (Λq − σ2I)2Λ−1q 0
0 0
] UT∗ +U∗ [ (Λq − σ2I)Λ−1q σ2 0
0 0
] UT∗ + σ 2I
=U∗
[ (Λq − σ2I)2Λ−1q + (Λq − σ2I)Λ−1q σ2 + σ2I 0
0 σ2I
] UT∗
=U∗Λ (j)UT∗ , j = 2.
Strategy 3 Sample z ∼ N (0, I), add a perturbation ∆z, and then sample xadv ∼ p(x|z + ∆z). In this strategy, xadv can be viewed as the adversarial example of x = arg maxx q(z|x).
xadv ∼ N (µ∗ +W∆z,WW T + σ2I),
With WW T + σ2I
=U∗ [ Λq 0 0 σ2I ] UT∗
=U∗Λ (j)UT∗ , j = 3.
In these 3 strageties, the adversarial examples can be summerized as
xadv = x ′ +W∆z and x′ ∼ D′j , j = 1, 2, 3,
where j = 1, 2, 3 corresponding to strategy 1,2 and 3.
If the data lie in a low dimensional space, i.e. the covariance matrix Σ∗ is rank q. Then the maximum likelihood of σ2ML = ∑d i=q+1 λi/(d− q) = 0. Then
Λ(1) = Λ(2) = Λ(3) = [ Λq 0 0 0 ] = Λ∗.
There is no difference among these 3 strategies and there is no change of distribution, i.e. D′ = D.

heading: A.1.1 EXCESS RISK ANALYSIS
text: Before we prove Theorem 2. We need to prove the following lemma first. Lemma 2 (optimal perturbation). Given Θ = (µ,Σ) the optimal solution of the inner max problem in equation 6 is ∆z∗ = W T (LΣ−WW T )−1(x′ − µ), where L is the lagrange multiplier satisfying ‖∆z∗‖ = ε.
Proof: Consider problem max ‖∆z‖≤ε `(x′ +W∆z,µ,Σ).
The Lagrangian function is
`(x′ +W∆z,µ,Σ)− L 2 (‖∆z‖2 − ε2)
= d
2 log(2π) +
1 2 log |Σ|+ 1 2 (x′ − µ+W∆z)TΣ−1(x′ − µ+W∆z)− L 2 (‖∆z‖2 − ε2).
Notice that this quadratic objective function is concave when L is larger than the largest eigenvalue ofW TΣ−1W . Calculate the partial derivative with respect to ∆z and set it to be zero, we have
W TΣ−1(x′ − µ+W∆z∗)− L∆z∗ = 0 ⇔(L−W TΣ−1W )∆z∗ = W TΣ−1(x′ − µ) ⇔∆z∗ = (L−W TΣ−1W )−1W TΣ−1(x′ − µ) ⇔∆z∗ = W T (LΣ−WW T )−1(x′ − µ).
The last equation comes from the Woodbury matrix inversion Lemma. We can obtain L by solving the equation ‖∆z∗‖ = ε. We don’t have a closed form solution of L but we can solve it numerically. L→∞ as ε→ 0. We only need to know L is a constant in our whole theory. Theorem 2 (Excess risk). Let Lls and L be the loss with or without perturbation in latent space (equation 6 and 3 respectively), given the non-robustly learned Θ∗ = (µ∗,Σ∗), The excess risk caused by perturbation is
Lls(Θ∗,D′j)− L(Θ∗,D′j) = 1
2 q∑ i=1 [ (1 +
λi − σ2
(L− 1)λi + σ2 )2 − 1 ]λ(j)i λi , j = 1, 2, 3
and the excess risk caused by changed of distribution is
L(Θ∗,D′j)− L(Θ∗,D) = 1 2 log [∏d i=1 λ (j) i∏d
i=1 λi
] + 1
2 ( d∑ i=1 λ (j) i λi − d ) .
Proof: Since x′ ∼ Dj = N (µ∗,Σj) = N (µ∗,U∗Λ(j)UT∗ ).
Denote v = x′ − µ∗ ∼ N (0,U∗Λ(j)UT∗ ).
And we have
WW T = Uq(Λq − σ2I)UTq = U∗ [ Λq − σ2I 0
0 0
] UT∗ .
The excess risk caused by perturbation is
2(Lls(Θ∗,D′j)− L(Θ∗,D′j)) =E(v +WW T (LΣ∗ −WW T )−1v)TΣ−1∗ (v +WW T (LΣ∗ −WW T )−1v)− EvTΣ−1∗ v =Tr [ (I +WW T (LΣ∗ −WW T )−1)TΣ−1∗ (I +WW T (LΣ∗ −WW T )−1)EvvT
] −Tr [ Σ−1∗ EvvT
] =Tr [ U∗ [ [I + (Λq − σ2I)((L− 1)Λq + σ2I)−1]2 0
0 I
] Λ−1∗ Λ (j)UT∗ ] − Tr [ Λ−1∗ Λ (j) ]
=Tr [ [[I + (Λq − σ2I)((L− 1)Λq + σ2I)−1]2 0
0 I
] Λ−1∗ Λ (j) ] − Tr [ Λ−1∗ Λ (j) ]
= q∑ i=1 [ (1 +
λi − σ2
(L− 1)λi + σ2 )2 − 1 ]λ(j)i λi , j = 1, 2, 3.
and the excess risk caused by changed of distribution is
2(L(Θ∗,D′j)− L(Θ∗,D)) = log |Σj | − log |Σ∗|+ Ex′(x′ − µ∗)TΣ−1∗ (x′ − µ∗)− Ex(x− µ∗)TΣ−1∗ (x− µ∗) = log |Σj | − log |Σ∗|+ Tr(Σ−1∗ Ex′(x′ − µ∗)(x′ − µ∗)T )− Tr(Σ−1∗ Ex(x− µ∗)(x− µ∗)T )
= log [∏d i=1 λ (j) i∏d
i=1 λi
] + Tr(Λ−1∗ Λ (j))− Tr(Λ−1∗ Λ∗)
= log [∏d i=1 λ (j) i∏d
i=1 λi ] + ( d∑ i=1 λ (j) i λi − d ) .
It is hard to see which part dominates the excess risk. If we further assume that the data lie in a q dimension manifold. The excess risk caused by the change of distribution becomes 0. We have the following corollary.
Corollary 3 (Excess risk). Let Lls and L be the loss with or without perturbation in latent space (equation (6) and (3) respectively), given the non-robustly learned Θ∗ = (µ∗,Σ∗), and rank(Σ∗) = q. The excess risk
Lls(Θ∗,D′j)− L(Θ∗,D) = O(qL−2).
Proof: By Lemma 1, we have σ2 = 0. λ(j)i = λi and D′j = D. Hence the excess risk caused by changed of distribution L(Θ∗,D′j)− L(Θ∗,D) = 0. The excess risk caused by perturbation is
2(Lls(Θ∗,D′j)− L(Θ∗,D′j))
= q∑ i=1 [ (1 +
λi − σ2
(L− 1)λi + σ2 )2 − 1 ]λ(j)i λi
= q∑ i=1 [ (1 +
1
(L− 1) )2 − 1 ] =O(qL−2).
Theorem 4 (Excess risk of orginal space adversarial training). Let Lr and L be the loss with or without perturbation in original space (equations (4) and (3) respectively), given the non-robustly learned Θ∗ = (µ∗,Σ∗). Denote λmin be the smallest eigenvalue of Σ∗. The excess risk
Ω((λminL) −2) ≤ Lr(Θ∗,D)− L(Θ∗,D) ≤ O(d(λminL)−2).
Theorem 4 can be viewed as a corollary of Theorem 1 in Ilyas et al. (2019). We give the prove here.
Proof: Consider the Lagrange multiplier form of the inner maximization problem in equation 4.
max ‖∆x‖≤ε `(x+ ∆x,µ,Σ).
The Lagrangian function is
`(x+ ∆x,µ,Σ)− L 2 (‖∆x‖2 − ε2)
= d
2 log(2π) +
1 2 log |Σ|+ 1 2 (x− µ+ ∆x)TΣ−1(x− µ+ ∆x)− L 2 (‖∆x‖2 − ε2).
Notice that this quadratic objective function is concave when L is larger than the largest eigenvalue of Σ−1. Calculate the partial derivative with respect to ∆x and set it to be zero, we have
Σ−1(x− µ+ ∆x∗)− L∆x∗ = 0 ⇔∆x∗ = (LΣ− I)−1(x− µ).
The excess risk is
2(Lr(Θ∗,D)− L(Θ∗,D)) =E(v + (LΣ∗ − I)−1v)TΣ−1∗ (v + (LΣ∗ − I)−1v)− EvTΣ−1∗ v =Tr [ (I + (LΣ∗ − I)−1)TΣ−1∗ (I + (LΣ∗ − I)−1)EvvT ] − Tr [ Σ−1∗ EvvT
] =
d∑ i=1 [(1 + 1 Lλi − 1 )2 − 1].
On the one hand, d∑ i=1 [(1 + 1 Lλi − 1 )2 − 1]
≥[(1 + 1 Lλmin − 1 )2 − 1] ≥Ω((Lλmin)−2).
On the other hand, d∑ i=1 [(1 + 1 Lλi − 1 )2 − 1]
≤d[(1 + 1 Lλmin − 1 )2 − 1] ≤O(d(Lλmin)−2).

heading: A.1.2 SADDLE POINT ANALYSIS
text: Theorem 5 (Main result: Optimal Saddle point). The optimal solution of the modified problem in equation (8) is
µls = µ∗ and Σls = U∗ΛlsUT∗ ,
where
λlsi = 1
4
[ 2λ
(j) i + 4(λi − σ2) L + 2λ (j) i
√ 1 +
4(λi − σ2) λ
(j) i L
] for i = 1 ≤ q and λlsi = λ (j) i for i > q.
j = 1, 2, 3 corresponding to strategies 1,2 and 3.
Problem 6 is not a standard minimax problem, consider the modified problem
min µ,Σ max Ex′‖∆z‖=ε
Ex′∼D′j `(x ′ +W∆z,µ,Σ), j = 1, 2, 3. (9)
By lemma 3, the optimal perturbation ∆z∗ is a matrix M times x− µ. Consider the problem
min µ,Σ max Ex′‖M(x′−µ)‖=ε
Ex′∼D′j `(x ′ +WM(x′ − µ),µ,Σ), j = 1, 2, 3. (10)
Lemma 6 (optimal perturbation). Given Θ = (µ,Σ) the optimal solution of the inner max problem of 10 is
M∗ = W T (LΣ−WW T )−1.
Proof: Consider the problem
max E‖M(x′−µ)‖=ε
E`(x′ +WM(x′ − µ),µ,Σ)).
The lagrangian function is
E [ `(x′ +WM(x′ − µ),µ,Σ)− L
2 (‖M(x′ − µ)‖2 − ε2)
] .
Let x′ − µ = v, Take the gradient with respect to M and set it to be zero, we have
∂ ∂M E [ `(x′ +WM(x′ − µ),µ,Σ)− L 2 (‖M(x′ − µ)‖2 − ε2) ] =∇ME [ vTMW TΣ−1v + 1
2 vTMW TΣ−1WMv − LvTMMv/2 ] = [ W TΣ−1 +W TΣ−1WM − LM ] E[vvT ]
=0.
Then we have M∗ =(L−W TΣ−1W )−1W TΣ−1
=W T (LΣ−WW T )−1.
The last equality is the Woodbury matrix inversion Lemma. Notice that lemma 3 and Lemma 6 have the same form of solution. This is why we can use Problem 10 to approximate Problem 6. To solve the problem 10, we need to introduce Danskin’s Theorem.
Theorem 7 (Danskin’s Theorem). Suppose φ(x, z) : X × Z → R is a continuous function of two arguments, where Z ⊂ Rm is compact. Define f(x) = maxz∈Z φ(x, z). Then, if for every z ∈ Z , φ(x, z) is convex and differentiable in x, and ∂φ/∂x is continuous: The subdifferential of f(x) is given by
∂f(x) = conv {∂φ(x, z)
∂x , z ∈ Z0(x)
} ,
where conv(·) is the convex hull, and Z0(x) is
Z0(x) = {z̄ : φ(x, z̄) = maxφ(x, z)}.
If the outer minimization problem is convex and differentiable, we can use any maximizer for the inner maximization problem to find the saddle point. But the outer problem of problem 4 is not convex, we need to modify the problem again. Assume that we have already obtained the eigenvector U∗ from the ML estimator. The optimization variables of the outer minimization problem are µ and Λ. Then we have Σ = U∗ΛUT∗ in problem 10. Another reason to make this assumption is that we only need to consider the eigenvalue problem to compare with standard adversarial training (Theorem 2 of Andrew Ilyas et al., 2019).
Proof of Theorem 5:
By Lemma 6, we have
M∗ =W T (LΣ−WW T )−1
=
[ Λq − σ2 0
0 0
]1/2 ( LΛ− [ Λq − σ2 0
0 0
])−1 UT∗ .
Which is a diagonal matrix ΛM times UT∗ . Let T = Λ −1, m = Λ−1UT∗ µ and x ′′ = UT∗ x ′. The optimization problem becomes
min m,T max ΛM
Ex′∼D′j `(x ′ +WΛMU T ∗ (x ′ − µ),m, T )
s.t. Ex′‖ΛMU∗(x′ − µ)‖2 = ε2. (11)
Obviously, the inner constraint is compact (by Heine-Borel theorem), we only need to prove the convexity of the outer problem to use Danskin’s Theorem. For any x′ and ΛM ,
`(x′ +WΛTMU∗(x ′ − µ),m, T )
= d
2 log(2π) +
1 2 log |Σ|+ 1 2 (x′ − µ+WM(x′ − µ)TΣ−1(x′ − µ+WM(x′ − µ).

heading: Let u = UT∗ (x
text: ′ − µ), and A = (I +
[ Λq − σ2 0
0 0
] ΛM ) 2, consider the third term, we have
1 2 log |Σ|+ 1 2 (x′ − µ+WM(x′ − µ)TΣ−1(x′ − µ+WM(x′ − µ)
= 1
2 uTA2Tu.
By Daskalakis et al., 2018Daskalakis et al. (2018), The hessian matrix is H = Covz∼N (T−1m,(AT )−1) [(vec(− 12AzzT )
z
) , ( vec(− 12Azz
T ) z
)] 0.
Therefore, this is a convex problem. By the same calculation in Lemma 6, a maximizer of the inner problem is
Λ∗M =
[ Λq − σ2 0
0 0
]1/2 ( LΛ− [ Λq − σ2 0
0 0
])−1 .
Then
A = [ I + [ Λq − σ2 0
0 0
]( LΛ− [ Λq − σ2 0
0 0
])−1]2 .
Then the first order derivative (by Daskalakis et al. (2018)) is ∇[T,m]T ` = [ 1 2AΛ (j) − 12T −1
AT−1m−AUT∗ µ∗
] = 0.
From the second equation, we directly have µls = µ∗. From the first equation, for i > q, we have
(1 + 0)2λ (j) i = λ ls i .
For i ≤ q, we have (1 + (λi − σ2)/(Lλlsi − λi + σ2))2λ (j) i = λ ls i .
It equivalents to a second order equation of √ λlsi√
λlsi
2 − √ λ
(j) i √ λlsi − λi − σ2
L = 0.
Solving this equation, we obtained
λlsi = 1
4
[ 2λ
(j) i + 4(λi − σ2) L + 2λ (j) i
√ 1 +
4(λi − σ2) λ
(j) i L
] for i = 1 ≤ q and λlsi = λ (j) i , for i > q.

heading: B EXPERIMENTS SETTINGS

heading: B.1 MNIST
text: For Mnist, we use LeNet5 for the classifier and 2 layers MLP (with hidden size 256 and 784) for the encoder and decoder of conditional VAE. For standard training of the classifier, we use 30 epochs, batch size 128, learning rate 10−3, and weight decay 5× 10−4. For the CVAE, we use 20 epochs, learning rate 10−3, batch size 64, and latent size 10.
For standard adversarial training, we use ε = 0.25 for FGSM and PGD. in PGD, we use 40 steps for the inner part. Adversarial training start after 10 epochs standard training.
For generative adversarial training, we use ε = 1 in the latent space with FGSM. Adversarial training start after 10 epoches standard training.
In the attack part, we use ε = 0.2 for norm-based attack and ε = 1 for generative attack on the test set.

heading: B.2 CIFAR10
text: For CIFAR10, we use ResNet32 for the classifier and 4 layers CNN for the encoder and decoder of conditional VAE. For standard training of the classifier, we use 200 epochs, batch size 128, learning rate 10−3, and weight decay 5× 10−4. For the CVAE, we use 100 epochs, learning rate 10−3, batch size 64, and latent size 128.
For standard adversarial training, we use ε = 4/255 for FGSM and PGD. in PGD, we use 10 steps for the inner part. Adversarial training start after 100 epochs standard training.
For generative adversarial training, we use ε = 0.1 in the latent space with FGSM. Adversarial training start after 100 epoches standard training. Since we see that the modeling power of VAE in CIFAR10 is not good enough. For each of the image, the encode variance is very small. When we
add a small perturbation to the encode mean value, the output image are blured. Hence we only use a small ε = 0.1.
In the attack part, we use ε = 4/255 for norm-based attacks and ε = 0.1 for generative attack on the test set. The test accuracy of VAE-adv against VAE attack is 40.18% in our experiments. It is not good enough because of the modeling power of VAE. But the results on standard adversarial training versus VAE-attacks are worse, and vice versa. The experiments support our findings.

heading: B.3 EIGENVALUES OF COVARIANCE MATRIX OF MNIST
text: We plot the eigenvalues of all the classes in this section, see Figure 3.

heading: C FINITE SAMPLES CASE: DATA AUGMENTATION OR ADVERSARIAL TRAINING
text: In this section we discuss the question that can we use the generative model to generate more examples for training in our theoretical framework. Because it is not closely related to our main results, we only discuss it in appendix. Let us focus on the case that the number of samples are not enough.
Generative model cannot help data augmentation Given dataset {xi}ni=1 ⊂ Rd, let µ̂ and Ŝ = ÛΛ̂Û be the sample mean and sample covariance matrix. The generative model learned by this dataset is x = Ûq(Λ̂q − σ2I)1/2z + µ̂+ . The distribution of the data sample from the model is
xgen ∼ N (µ̂,WW T + σ2I) = N (µ̂, ÛΛ̂genÛ) , N (µ̂, Ŝ′). (12) If we use n samples from the original dataset and m samples from the generative model, the loss function is
L(µ,Σ) = min µ,Σ
1
n+m n∑ i=1 `(xi;µ,Σ) + m n+m Ex∼N (µ̂,Ŝ′)[`(x;µ,Σ)]. (13)
Theorem 8 (Data augmentation by generative model). Given MLE µ̂ and Ŝ, The optimal solution training with n true samples and m generated samples is µda = µ̂ and Σda = ÛΛdaÛT , where
λdai = λ̂i for i ≤ q and λdai = n
n+m λ̂i +
m
(n+m)(d− q) d∑ k=q+1 λ̂k for i > q.
Proof: Consider the derivative of Problem 13.
∇µL = 1
n+m n∑ i=1 Σ−1(xi − µ) + m n+m ExΣ−1(x− µ) = 0.
Then we can obtain µda = µ̂.
∇Σ−1L = −Σ + 1
n+m n∑ i=1 (xi − µ)(xi − µ)T + m n+m Ex(x− µ)(x− µ)T = 0.
Then we have Σls = n
n+m Ŝ +
m
n+m Ŝ′.
For i ≤ q, λlsi = n
n+m λ̂i +
m
n+m λ̂i = λ̂i.
For i > q,
λlsi = n
n+m λ̂i +
m
n+m σ̂2 =
n
n+m λ̂i +
m
(n+m)(d− q) d∑ k=q+1 λ̂k.
The optimal solution is a little bit destoryed. In this perspective, generative models give no help to data augumentation.
Generative model can help robust training Theorem 9 (Adversarial learned parameters). The Adversarial learned features of problem 10 under finite samples is µls = µ̂ and Σls = ÛΛlsÛT where Λls is of the same form in Theorem 5 except replacing λi by λ̂i under strategy 3.
Proof: Considering strategy 3. In this case
W = Û(Λ̂q − σ2)1/2.
and x′ ∼ (µ̂,WW T + σ2I).
Since
WW T + σ2I = Û [ Λ̂q 0 0 σ2I ] ÛT .
In words, the eigenvectors of the dsitribution Û is the same as the one in perturbationW∆z. The optimization problem is the same as the one we use in the proof of Theorem 5 if we replace λi by λ̂i.
If the data lie in a q dimensional subspace, we do not neet to make the assumption on strategy 3.

Review summaries:
This paper analytically considers two flavours of adversarial training in a Gaussian mixture model.  The first uses regular adversarial examples, and the second uses examples drawn from a generative model.  The authors show that the adversarial perturbations generated in the two cases differ in a cleanly-characterisable  way: in the first case the perturbations differ from real data in a direction aligned with the smallest eigenvalues  of the data covariance. In the latter case the perturbations are in a direction aligned with the largest eigenvalues.  Experimental results on MNIST and CIFAR are presented to illustrate how the analysis transfers to real datasets.