Disentangling Adversarial Robustness in Directions of the Data Manifold
Authors: Jiancong Xiao
Liusha Yang
Zhi-Quan Luo
Venue:  (2021)
Keywords: Adversarial Robustness
Adversarial Training
Generative Models

Abstract: Using generative models (GAN or VAE) to craft adversarial examples, i.e. generative adversarial examples, has received increasing attention in recent years. Previous studies showed that the generative adversarial examples work differently compared to that of the regular adversarial examples in many aspects, such as attack rates, perceptibility, and generalization. But the reasons causing the differences between regular and generative adversarial examples are unclear. In this work, we study the theoretical properties of the attacking mechanisms of the two kinds of adversarial examples in the Gaussian mixture data model case. We prove that adversarial robustness can be disentangled in directions of the data manifold. Specifically, we find that: 1. Regular adversarial examples attack in directions of small variance of the data manifold, while generative adversarial examples attack in directions of large variance. 2. Standard adversarial training increases model robustness by extending the data manifold boundary in directions of small variance, while on the contrary, adversarial training with generative adversarial examples increases model robustness by extending the data manifold boundary directions of large variance. In experiments, we demonstrate that these phenomena also exist on real datasets. Finally, we study the robustness trade-off between generative and regular adversarial examples. We show that the conflict between regular and generative adversarial examples is much smaller than the conflict between regular adversarial examples of different norms.

Under review as a conference paper at ICLR 2021
DISENTANGLING ADVERSARIAL ROBUSTNESS IN DI- RECTIONS OF THE DATA MANIFOLD
Anonymous authors Paper under double-bling review
ABSTRACT
Using generative models (GAN or VAE) to craft adversarial examples, i.e. gen- erative adversarial examples, has received increasing attention in recent years. Previous studies showed that the generative adversarial examples work differently compared to that of the regular adversarial examples in many aspects, such as attack rates, perceptibility, and generalization. But the reasons causing the differences between regular and generative adversarial examples are unclear. In this work, we study the theoretical properties of the attacking mechanisms of the two kinds of adversarial examples in the Gaussian mixture model. We prove that adversarial robustness be disentangled in directions of the data manifold. Specifically, we find that: 1. Regular adversarial examples attack in directions of small variance of the data manifold, while generative adversarial examples attack in directions of large variance. 2. Standard adversarial training increases model robustness by extending the data manifold boundary in directions of small variance, while on the contrary, adversarial training with generative adversarial examples increases model robustness by extending the data manifold boundary directions of large variance. In experiments, we demonstrate that these phenomena also exist real datasets. Finally, we study the robustness trade-off between generative and regular adversarial examples. We show that the conflict between regular and generative adversarial examples much smaller than the conflict between regular adversarial examples of different norms.
INTRODUCTION
In recent years, deep neural networks (DNNs) (Krizhevsky et al. (2012): Hochreiter and Schmidhuber (1997) have become popular and successful in many machine learning tasks. They have been used in different problems with great success. But DNNs are shown to be vulnerable to adversarial examples (Szegedy et al. (2013): Goodfellow et al. (2014a)). A well-trained model can be easily attacked by adding small perturbation the image. An effective way solve this issue is train the robust model using training data augmented with adversarial examples, i.e. adversarial training.
With the growing success of generative models, researchers have tried to generative adversarial networks (GAN) (Goodfellow et al. (2014b)) and variational autoencoder (VAE) (Kingma and Welling (2013)) to generate adversarial examples (Xiao al. (2018); Zhao al. (2017): Song et al. (2018a); Kos et al. 2018); Song et al. 2018b)) to fool the classification model with great success. They found that standard adversarial training cannot defend these new attacks. Unlike the regular adversarial examples, these new adversarial examples are perceptible by humans but they preserve the semantic information of the original data. A good DNN should be robust such semantic attacks. Since the GAN and VAE are approximations of the true data distribution, these adversarial examples will stay in the data manifold. Hence they are called on-manifold adversarial examples by (Stutz et (2019)). On the other hand, experimental evidences support that regular ad adversarial examples leave the data manifold (Song et al. (201)). We call the regular adversarial examples as off-manifold adversarial examples.
The concepts of on manifold and off- off-manifold adversarial examples are important. Because they can help to understand the issue of conflict between adversarial robustness and generalization (Stutz) et al. (2019): Raghunathan et al. (2019), which is still an open problem. In this paper, we study the attacking mechanisms of these two types of examples, as well as the corresponding adversarial

---

Under review as a conference paper at ICLR 2021
training methods. This study, as far we know, has not been done before. Specifically, we consider a generative attack method that adds a small perturbation in the latent space of the generative models. Since standard adversarial training cannot defend this attack, we consider the training methods that use training data augmented with these on-manifold adversarial examples, which we call latent space adversarial training. Then we compare it standard adversarial training (training with off-manifold adversarial examples).
We study the thcoretical properties of latent space adversarial training and standard adversarial training in the Gaussian mixture model with linear generator. We give the excess risk analysis and saddle point analysis in this model. Based on this case study, we claim that:
Contributions:
Regular adversarial examples attack in directions of small variance of the data manifold and leave the data manifold.
Standard adversarial training increases the model robustness by amplifying the small vari- ance. Hence, extends the boundary of the data manifold directions of small variance.
Generative adversarial examples attack in directions of large variance of the data manifold and stay in the data manifold.
Latent space adversarial training increases the model robustness by amplifying the large variance. Hence, it extends the boundary of the data manifold in directions large variance.
We provide experiments on MNIST and CIFAR-IO and show that the above phenomena also exist in real datasets. It gives new perspective to understand the behavior of on-manifold and off- off-manifold adversarial examples. Finally, we study the robustness trade-off between generative and regular adversarial examples. On MNIST, robustness trade-off is unavoidable, but the conflict between generative adversarial examples and regular adversarial examples are much smaller than the conflict between regular adversarial examples of different norms. On CIFAR-10. there is nearly no robustness trade-off between generative and regular adversarial examples.
2
RELATED WORK
Our work is related attack and defense methods. Specifically, we care about attacks and defenses with generative models.
Attack Adversarial examples for deep neural networks were first intruduced in (Szggdyy et al. (2013)) However, adversarial machine learning robust machine learning has been studied for a long time (Biggio and Roli (2018).. In the setting of white box attack (Kurakin et al. (2016); Papernot al. (2016); MMosaoi--Dzzooree Dezfooli et al. (2016); Carlini and Wagner (2017)). the attackers have fully access to the model (weights, gradients, etc.). In black box attack (Chen et al. 2017) Suu et al. (2019) Iyyas et al. ) the attackers have limited access to the model. First order optimization methods, which use the gradient information to craft adversarial examples, such as PGD (Madry
al. (2017)) are widely used for white box attack. Zeroth-order optimization methods (Chen et al.
(2017) are used in black box setting. Li et al. 2019) improved the query efficiency in black-box attack. HopSkiplumpAAtael (Chen et al. 2020 is another query-efficient attack method.
Generative adversarial examples Recently, generative models have been used to craft adversarial examples (Xiao et al. (2018); [Song et al. (20188 8b); Kos et al. (2018); Schott et al. (2018)). The adversarial examples are more natural (Zhao et al. (2017)) These adversarial examples lie the data manifold, and they are called on- manifold adversarial examples.
Training algorithms against adversarial attacks can be subdivided into the following cate- gories. Adversarial training: The training data is augmented with adversarial examples to make the models more robust (Madry et al. (2017): (Szegedy et al. (2013); Tramèr et al. 2017)). Preprocessing: Inputs or hidden layers are quantized, projected onto different sets or other preprocessing methods (Buckman et al. 2018); (Guo et al. (2017 Kabilan et al. (2018)). Stochasticity: Inputs hidden activations are randomized (Prakash et al. (2018): Dhillon et al. (2018); Xie et al. (2017). However, some of them shown be useless defenses given by obfuscated gradients (Athalye et al, (2018). Adaptive attack [Tamer et al. (2020)) is used for evaluating defenses to adversarial examples.
Defense
Defense with generative model Using generative models to design defense algorithms have been studied extensively. Using GAN, project the adversarial examples back to the data manifold
2

---

Under review as a conference paper at ICLR 2021
Jalal al. (2017); Samangouei et al. 2018)). VAE is also used to train robust model (Schott et al. 2018
PROBLEM DESCRIPTION
Original space adversarial training: Consider the classification problem of training a classifer So to map the data points æ E X C R2 to the labels E y where X' and y are the input data space and the label space. The classifier fe parameterized by 0. We assume that the data pairs (z,, y) are sampled from the distribution P(X, Y) over X' x y. Standard training is to find the solution of ming ((..,))) (x)), y). where {(:,)) is the loss function. The goal of dversarial training is to solve the minimax problem
min E (aNU) max {(fo(z').)) |x-x''Se
0
where e the threshold perturbation. Here we can use L, la or loo -norm (Madry et al. (2017)) The inner maximization problem is to find the adversarial examples x' to attack the given classifier fg. The outer minimization problem is to train the classifier to defend the given adversarial examples x². refer to these attacks as regular attacks. We refer to these minimax problems as standard adversarial training or original space adversarial training.
Latent space adversarial training: We assume that the data lie in low dimensional manifold of Rd. Furthermore, we assume the true distribution D is a pushforward from a prior Guassian distribution ~N00, I) using G(z), where G Z X is mapping from the latent space Z to the original space X'. This is a basic assumption of GAN or VAE. Let / X + Z be the inverse mapping of G(z). The goal of latent space adversarial training is to solve the following minimax problem
min minnE(ey)>P ((.))~>
n18X (=)
((Jo(G(z))),))
0
||
Unlike the regular attacks, the distance between the original examples and adversarial examples can be large. To preserve the label of the data, we use the conditional generative models (e.g. C-GAN (Mirza and Osindero (2014)) and C-VAE (Sohn et al. (2015)), i.e. the generator (z) and inverse mapping (x) are conditioned on the label y, for adversarial training. We refer to these attacks as generative attack, and these adversarial training as latent space adversarial training.
Two widely used gradient- bbssed attack algorithms for the inner maxi- are fast gradient sign method (FGSM) (Goodfellow et al. (2014)) and projected gradient descend (PGD) (Madry et al. (2017)). Using FGSM, the adversarial exam- ples calculated by
Regular attack algorithms mization problem in equation ①
= x + esgn( V Va (f(()))))))
where V denotes the gradient with respect to x. PGD attempts to find a near optimal adversarial example for the inner maximization problem ( multiple steps. In the t step,
++1 II+s[ø + aV
where is the step size, II s[ is the projection operator to project the given vector to the constraint E).
{a'||=
In the whole paper. we refer to these as FGSM-attack and PGD-attack, and the corresponding original space adversarial training as FGSM-adv and PGD-ady. FGSM-attack is weak attack and PGD--ttack is a stronger attack. In section 5 we use them to show that strong original space adversarial training, PGD-adv, does not work well against a weak attack, FGSM-attack in the latent space. Conversely, latent space adversarial training cannot defend simple FGSM-attack in the original space.
Generative attack algorithm In our experiments, we use FGSM the latent space for the inner maximization problem in equation
L' = [(x) esgn V ( v)).
Because of the mode collapse issue of GAN Salimans et al. (016)); Guirajani et al. (011 adding small perturbation in the latent space of GAN may output the same images. Thus we VAE in our experiments. we refer to this generative attack and latent space adversarial training as VAE-attack and VAF-adv.
3

---

Under review as a conference paper at ICLR 2021
4 THEORETICAL ANALYSIS
In this section, we study the difference between adversarial training in the latent space and in the original space. We study the simple binary classification setting proposed by [[yas et al. (2019). The main reason for using this model is that we can find the optimal closed-form solution, which gives us insights understand adversarial training. For more complex model, we can only solve numerically. We leave all the proofs of the lemmas and theorems in appendix A.
4.1 THEORETICAL MODEL SETUP
Gaussian mixture (GM) model Assume that data points (ar, y) are sampled according to y ~ {-1,1} unifomly and (ypp, E,), where μa, and Σ, denote the true mean and covariance matrix of the data distribution. For the data in the class y = -1, we replace x by -r, then we can view the whole dataset as sampled from D (N,, E.).
Classifier
The goal of standard training is to learn the parameters O = (n, Σ) such that arg min L(u, Σ) arg min Ee= nt Σ)}, 2.
2.E
where ((-) represents the negative log- lggelikl function. The goal of adversarial training is find
arg min min min Ezwpl max f(z': 1 E)] θ, = arg min C. Σ)
(4)
E.E
E.E
-0-v'l|se
We use and C, to denote the standard loss and adversarial loss. After training, we classify a new data point x to the class sgn( / Σ E-)r).
Generative model In our theoretical study, we use a linear generative model, that is, probabilistic principle components analysis (P-PCA) (Tipping and Bishop (999) ). P-PCA can be viewed as linear GAN Feizi et al. (2017): Feizi et al. (2000) and linear VAE (Dai et al. (2017)).
Given dataset {=i Rd, let μ and S be the sample mean and sample covariance matrix. The eigenvalue decomposition of S is S UAU", then using the first q eigenvactors, can project the data to low dimensional space. P-PCA is to assume that the data are generated by
W2 + where ~ ~~((0, C o(0,0²1),
E R2 and W E Rdxq Then we have x N(u, WWT + o²T), s/= N(W μt, o²1) and 2| ~ N(P WT (z - 4)) 02 ppp-)) where = W W + The maximum likelihood estimator
of W and x² are
WML = U, - ML. 1) and
ML
A
i=9+1
where U, is the matrix of the first & columns of U, Ag is the matrix of the first eigenvalues of A. In the following study, assume that large enough such that we can learn the μa. and Σ.. Thus we have S E,, Ug = Ugn, Ag = Age for the generative model.
4.2 MINIMAX PROBLEM OF LATENT SPACE ADVERSARIAL TRAINING
To perturb the data in the latent space, Data will go through the encode-decode process -> -> A= + x'. Based the probabilistic model, we may choose with the highest probability or just sample from the distribution we learned. Hence, we could have different strategies. Below we list 3 strategies. Strategy 1 is used in practice and the other two are alternative choices. In lemma I we show that these strategies are equivalent under the low dimensional assumption. Hence, we do not need worry about the effect of sampling strategies.
Sample x ~ D, encode & = arg max (<([r)) = P- P- (a p.),, add a perturbation Az, and finally, decode Fadv arg max p(x 2 Az) = W + Az) pb.
Strategy 1:
Strategy 2: Sample x ~ D, then sample z ~ q(2(z), add perturbation Az, and finally, sample Tado p + Az).
4

---

Under review as a conference paper at ICLR 2021
Strategy 3: Sample ~ (0 I), add perturbation Az, and then sample Tado p(xl2 + Az). In this strategy, Fado can be viewed as the adversarial example of x = arg max, ((1/z))
The following lemma shows that the adversarial examples can be unified in one formula. Hence sampling strategies will not affect our analysis.
Lemma (Adversarial examples perturbed in the latent space), Using these strategies, the adver- sarial examples can be unified as
Eado = + W Az
and a' ~ D'- N(K,,U,N
J = 1,,,3,
(5)
where
(1) (Ag 021)2A
07
A
[ 0
0
0
0-1
( (Aq - o2 021)2 - (Ag-o21)A 021
0
0
0²1
If the data lie in a dimensional subspace, iee. the covariance matrix E. is rank q. we have A A A (3) A,. Then D' DD
In general, the adversarial example can be decomposed into parts, the change of distribution x' 'D' and the small perturbation WA a.. Therefore the adversarial expected risk can written as the following minimax problem
min (H, Σ; DJ) = min E 3.E
D'NDY max P.E
Elz' W.,, 1,2,333.
(6)
We aim to analyze the different properties between the minimax problems in equations and (6). give the excess risk and optimal saddle point analysis in the following subsections.
4.3 EXCESS RISK ANALYSIS
We consider the difference between Lts and L given the true e., i.e. Li, DJ) L(O.: It characterizes the excess risk incured by the optimal perturbation. To derive the expression of excess risk, we decompose into parts
-((,)))))(((0))))
-
perturbation
change of distrsbution
To simplify the notation, we consider the Lagrange penalty form of the inner maximization problem equation (6). i.e max E(a' + W A2, Σ) L.|| |=|l2/2, where the Lagrange multiplier. The following theorem gives the solution in the general case.
Theorem (Excess risk). Let Lis and L be the loss with and without perturbation in latent space (equations o and (3) respectively), given the non-robustly learned O_ (Jo:: E.), thus the excess risk caused by perturbation is
X; o2
-1100
(1)
Lis (o. C(). D().:
(1
0² -
1,2,3.
(L 1)A
X
i=1
and the excess risk caused by the changed of distribution
T (() I A;
x A
1
((O..
L(O., -
log
+
d).
2
is hard to see which part dominates the excess risk. If we further assume that the data lie in a dimensional manifold, the excess risk caused by the change of distribution becomes by Lemma We have the following corollary.
Corollary (Excess risk). Let Lis and L be the loss with or without perturbation latent space (equation (6) and 3 respectively), given the non-rohine con-robustly learned O2 = (Mxx E.), and rank Σ.) The excess risk
Lis(O.. D') L(O.. Og...-
5

---

Under review as a conference paper at ICLR 2021
The optimal perturbation in the latent space will incur excess risk in O(gL The adversarial vulnerability depends on the dimension 9 and the Lagrange multiplier L. It does not depend on the shape of the data manifold. This is because the perturbation constraint (the black block) aligns with the shape of the data manifold (the ellipse) as we demonstrate in Figure (c). Thus generative attacks will focus the directions of the largest variance.
Then, we analyze the excess risk of original space adversarial training. Since the perturbation thresh- olds, e, are on different scales in the original space attack and latent space attack, the corresponding Lagrange multipliers L are different. We use L' for original space adversarial training in the following Theorem.
Theorem (Excess risk of original space adversarial training). Let C, and L be the loss with or without perturbation in original space (equations 4 and (3) respectively), given the non-robustly learned O (Hr, E.). Denote mmin be the smallest eigenvalue of Σx. The excess risk is
D(((AminL')
S -()
L(G..))
((d((min
If the data lie in low dimensional manifold, i.e. Amin = 0, the excess risk equals to +
Optimal perturbation in the original space will incur an excess risk in (d(2min L')-²). The adversarial vulnerability depends on the smallest eigenvalues Amins the dimension d. and the Lagrange multiplier L'. The Amin comes from the misalignment between the perturbation constraint (the black block) and the shape of the data manifold (the ellipse) as we demonstrate Figure (a). Notice that Amin also appears in the lower bound. Hence, the excess risk equals to +00 when Amin 0. Thus regular attacks focus on the directions of small variance. Specifically, when Amin = 0, regular adversarial examples leave the data manifold.
4.4 SADDLE POINT ANALYSIS
In this subsection we study the optimal solution of optimization problem (6). Since is not standard minimax problem, we consider a modified problem:
min E E =e
max
=c E BE Ela' + WA:, E), = 1,2,3.
We will explain more about the connection between optimization problems in equation ( and (8) See appendix A. The following theorem our main result. gives the optimal solution of latent space adversarial training.
Theorem 5 (Main result: Optimal Saddle point). The optimal solution of the modified problem in
equation 8 is
= u. and Eis = U.A.UT
where
X
1 22,00
4(h;- -4--0))
02) + 21
1 1+ 1(h: 02) 0221
for =1 1 X = x²) for q.
L
X
and j = 1,2,3 3 corresponding to strategies 1,2 and 3.
We assume that the data lie in a q-dimensional manifold again. Then we have 1/2+ 1/4 + 1/L > 1 for i S q and / = 0 for i >> Latent space adversarial training increases the model robustness by amplifying large eigenvalues of the data manifold. The illustration of the two dimensional case is in Figure (c) and (d).
In the same setting, the optimal solution of standard adversarial training (Problem on the original
space is in Theorem 6 (which is Theorem 2 in Ilyas et al. (2019)).
Theorem 6 (Optimal saddle point llyas et al. 2019)). The optimal solution the problem in equation is
1 1
1 I +
1 E. + E2
My = ph and Σ,
E.
+ T/
Σ.
2

---

Under review as a conference paper at ICLR 2021
Theorem i for the problem that the covariance matrix is restricted to be diagonal. Consider the ratio
-
()) A; - L'X
L'X
For small true eigenvalue A, the ratio is large. Standard adversarial training increases the robustness by amplifying the small eigenvalues of the data manifold. The illustration of the two dimensional case in Figure D (a) and (b).
-4
4
Figure 1:
Demonstration of theoretical analysis: (a) Regular attacks directions: (b) Optimal saddle point of original space adversarial training; (c) Generative attacks directions; (d) Optimal saddle point of latent space adversarial training.
EXPERIMENTS
In this section we report our experimental results on training LeNet on MNIST (LeCun al. (1998) and ResNet (He et al. (2016)) on CIFAR-10 (Krizhevsky et al. (2009) to confirm our theoretical findings. Both of the two datasets contain 50000 training samples and 10000 test samples. Details of the hyperparameters setting are in appendix B.
5.1 EIGENVALUES OF MNIST WITH AND WITHOUT ADVERSARIAL TRAINING
In this subsection, we show that the properties of the eigenvalues are also valid on the real dataset. We use MNIST as an example. Firstly, for each image in the test set Drest of MNIST, we vectorize it into a 784 dimension vector. Then we calculate the EVD (Eigenvalue decomposition) of the covariance matrix of each class. We plot the 784 eigenvalues of class and I in Figure iin blue line as examples (the other 8 classes are shown in Appendix B.3) Secondly, we adversarially train classifier fz( in original space. For each point x: in Dtest, we construct 2r- by PGD-attack with v=f+(z)). Then we obtain a robust test set Di We plot the intra class eigenvalues of the covariance matrix in Figure 2 in orange line. Lastly, we adversarially train a classifier fis(x) latent space. Then we construct latent space robust dataset Dis by generative attack against fis (z) Diest. The eigenvalues of the covariance matrix are shown in Figure i] green line.
We show all the eigenvalues in the first column in Figure 2 After adding small perturbation in the original space or in the latent space, the distribution of Diest and Dia are close to Drest- We show the details of the figure of large eigenvalues (the first 30 eigenvalues) the second column and small eigenvalues (last 754 eigenvalues) in the last column.
Original space adversarial training focuses on the small variance direction We adversarially train robust classifier f-(22) in the original space. The robust test set D against fr(z) amplified the small eigenvalues a lot. In the third column, the orange line is significantly larger that the other lines. While in the second column, the orange line is below the other us an understanding of how the adversarial examples leave the data manifolds. The original dataset Drest lies in low dimension affine plane in IR 784 After adversarial training, the data move towards the small variance directions. Finally Diess stay in a full dimension subspace.
lines. The experiments give
Latent space adversarial training focuses on the large variance direction We adversarially train robust classifier fi)( in the latent space. The latent space robust test set D/s against fia (x) amplified the large eigenvalues. In the second column, we can see that the green line is above the other two lines. And in the last column, the green line shows that adversarial training in the latent space will not affect the small eigenvalues. The adversarial examples in D/s move along the large variance direction. Therefore, Dis Pi stay in the low dimension affine space.
7

---

Under review as a conference paper at ICLR 2021
0
y
30
clean data VNE ottack
100
clean data WA attack
dlwn data
WE attack
31
AUl eigenvalues
class
The first 30 elgenvalues
The last 754 eigenvalues
clean data
deaa data
dican data
PCDD wttack
PCGD -attack
#EiD-attack
100
200
80
220
30
26
80
120 The last 754 cigenvalues
X6
05
120
109
All zigenvalles in class y
The first eigenvalues
Figure 2: Eigenvalues of MNIST: The first and the second row of classes 0 and respectively, The first column is about all the 784 eigenvalues of the dataset. The second column plots the large cigenvalues and the last column plots the small eigenvalues.
5.2 ROBUST TEST ACCURACIES
In this subsection, we compare the test accuracy of different attacks (FGSM, PGD, VAE) versus different defense (adversarial training with FGSM, PGD, and VAE) to help explain our theoretical results. We explain the results of MNIST in Table as an example.
On-manifold and off-manifold adversarial examples The test accuracies of the standard training model on PGD and VAE-attack data are 3.9% and 42.4% respectively. Firstly, They show that on-manifold adversarial examples exist. Then, we can see that on-manifold adversarial examples harder to find.
Attack versus defense Our theory tells us that original or latent space adversarial training increase the model robustness by amplifying the small or large eigenvalues, which are the variance of the distribution, in the data manifold. In experiments, the test accuracy of VAE-ady vs PGD-attack is 1.23%, which shows that extending the manifold boundary in directions of large variance gives contribution to defending attacks in directions of small variance. Similarly, the test accuracy of PGD-ady vs VAE-attack is 52.18 which shows that amplifying the boundary in the directions of small variance does not work well in defending attacks in directions of high variance. Further more, we can see that latent space adversarial training does not work well against simple FGSM attack. and vice versa.
We see that PGD-adv can increase the test accuracies on VAE-attack from 42% 52% on MNIST and from 19.40% to 26.31% % on 10. A possible reason is that original space adversarial training can amplify the small eigenvalues of the first q dimension but fails to amplify the larger ones. As it is indicated in Figure 2 both original space and latent space adversarial training will increase the eigenvalues which are small but not equal to zero (around the 100th eigenvalue) of the covariance matrix.
5.3 ROBUSTNESS ORADE-OFF
Robustness trade-off are common in practice (Su et al. 20018] Tramer and Bonch (011 ). For example, & and foo adversarial examples cannot be defended simultancously. The experiments on robustness trade-off are shown in Table On MNIST, the lointly-trainee model using both & and for adversarial examples will decrease the robust test accuracy by 20% comparing to the single trained model. On CIFAR-10, 100, the jointly-traines model will decrease the test accuracies by 6% on é attack and by 10% attakk, A possible reason is that they are all original space attacks and focus
8

---

Under review as a conference paper at ICLR 2021
MNIST
Std training
FGSM-Adv
PGD-Adv
VAE-Adv
clean data
98.82% %
98. 79%
98.73%
98.28%
FGSM-Attack
47.43%
91.08%
96.50%
27.94%
PGD-Attack
3.90%
15.46%
95.51%
1.23%
VAE-Attack
42.40%
46.94%
52.18%
96.66
CIFAR-I0
Std training
FGSM-Adv
PGD-Adv
VAE-Adv
clean data
92.14%
87.03%
86.25%
85.59%
FGSM-Attack
38.39%
76.92%
77.34%
34.50%
PGD-Attack
7.40%
75.14%
75.78%
10.53%
VAE-Attack
19.40%
31.80%
26.31%
40.18%
Table 1: Test accuracies of different attacks and defense algorithms MNIST and CIFAR-10.
the same directions of small variance. Hence they conflict with each other. We study the robustness trade-off between regular and generative adversarial examples in this subsection.
MNIST
41AAv
Advv
joint
PGID-Adv
VAE-Adv
joint
attttcck
78.50%
12.13%
54.24%
PGD-attack
95.51 %
1.23%
89.50%
Ex -attack
0.5%
95.51%
78.81%
VAB-attack
52.18%
96.66%
90.28%
CIFAR-10 10
41-Avv
Lxo Adv
joint 60.23%
PGD-Adv
VAE-Adv
joint
aatttcck
66.22%
16.41%
PGD-attack
75.78%
10.53%
74.22%
8o0 -attack
52.8%
75.78%
65.1 1%
VAE-attack
26.31%
40.18%
42.47
Table 2: Comparison of the robustness trade-off' between & and Loo attack and the trade-off between original space and latent space attack on MNIST and CIFAR-1O.
Our theory suggests that adversarial robustness can be disentangled in different directions. Hence adversarial robustness against attacks directions of low variance and large variance be guaranteed simultaneously. On MNIST, the jointly-trained model decreases the test accuracies by 6% comparing to the single trained model. The conflict between an-manifold and off-manifold adversarial examples is much small than the conflict between off-manifold adversarial examples of different norms. On CIFAR-10, the jointly-trained model gets accuracies 74% PGD-attack and 42% on VAE-attack, which exhibit nearly no robustness trade-off. Therefore, if our goal is to defend mixture of regular and generative attacks, The jointly-train model will perform well.
Discussion Under the g-dimensional manifold assumption, there is an overlap between the direc- tions of the largest variance and the directions of small variance. This supported by Theorem Theorem 6, and the first experiment. A possible reason for the robustness trade-off between regular and generative attacks is that they conflict with each other in the overlap directions. The conflict is not serious because they mainly focus different directions.
CONCLUSION
In this paper, we show that adversarial robustness can be disentangled in directions of small variance and large variance of the data manifold. Theoretically, study the excess risk and optimal saddle point of the minimax problem of latent space adversarial training. Experimentally, we show that these phenomena also exist real datasets.
Future works One may design defense algorithms based on this property. We can generate adversarial examples based on the directions of the dataset without access to the model architecture. In white-box settings, we can use them for data augmentation to accelerate adversarial training or increase the model robustness. However, between calculating the EVD of the datasets and that of standard adversarial training. In black-box attacks, we can use them to query the target model. But the data manifolds should transferred from some known datasets (since there no information available for the training dataset of the target model). Theoretically, it is unclear whether we can find the closed-form form solution under nonlinear models. How to analyze the nonlinear model is an open problem. Using linear models, we can further analyze the conflict between robustness and generalization.
need to carefully compare the computational cost

---

Under review as a conference paper at ICLR 2021
REFERENCES
Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097-1105, 2012.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1335-178 1997.
Christian Szegedy, Wojciech Zaremba, llya Sutskever, Joan Bruna, Dumitru Erhan, lan Goodfellow. and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.66 1312.6199, 2013.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014a.
lan Goodfellow, Jean Pouget-Abadic. Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672-2680, 2014b.
Diederik P Kingma and Max Welling. Auto Auto-encoding variational bayes. arXiv preprint arXiv: 1312.6111 2013.
Chaowei Xiao, Bo Li, Jun Zhu, Warren He, Mingyan Liu, and Dawn Song. Generating adversarial examples with adversarial networks. arXiv preprint arXiv:/801.00 (et1.00010000 2018.
Zhengli Zhao, Dheeru Dua, and Sameer Singh. Generating natural adversarial examples. arXiv preprint 1710v 2017.
Yang Song. Rui Shu, Nate Kushman, and Stefano Ermon. Generative adversarial examples. arXiv preprint arXiv: 1805.07894 2018a.
Jernej Kos, lan Fischer, and Dawn Song. Adversarial examples for generative models. In 2018 IEEE Security and Privacy Workshops (SPW), pages 36-42. IEEE, 2018.
Yang Song, Rui Shu, Nate Kushman, and Stefano Ermon. Constructing unrestricted adversarial examples with generative models. In Advances in Neural Information Processing Systems, pages 8312-8323, 201 8b.
David Stutz, Matthias Hein, and Bernt Schicle. Disentangling adversarial robustness and generalization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6976-6987, 2019.
Yang Song, Taesup Kim, Schastian Nowozin, Stefano Ermon, and Nate Kushman. Pixeldefend: Leveraging generative models understand and defend against adversarial examples. arXiv preprint arXiv: 10766, 2017.
Aditi Raghunathan, Sang Michael Xie, Fanny Yang, John C Duchi, and Percy Liang. Adversarial training can hurt generalization. arXiv preprint 1000002000.0033,, 2019.
Battista Biggio and Fabio Roli. Wild patterns: Ten years after the rise of adversarial machine learning. Pattern Recognition, 84:317-331 2018.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples the physical world. arXiv preprint 1.XXv:6607.0253.. 2016.
Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson. Z Berkay Celik, and Ananthram Swami. The limitations of deep learning in adversarial settings. In 2016 IEEE European symposium on security and privacy (EuroS&P), pages 372-387. IEEE, 2016.
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. Deepfool: simple and accurate method to fool deep neural networks. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 2574-2582, 2016.
Nicholas Carlini and David Wagner. Towards evaluating the robustness neural networks. 2017 ieee symposium on security and privacy (sp). pages 39-57. IEFF, 2017.
Pin-Yo Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsich. Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pages 15-26, 2017.
Jiawei Su, Danilo Vasconcellos Vargas, and Kouichi Sakurai. One pixel attack for fooling deep neural networks. IEEE Transactions Evolutionary Computation, 2385)8828-411 2019.
10

---

Under review as a conference paper at ICLR 2021
Andrew Ilyas, Logan Engstrom, and Aleksander Madry. Prior convictions: Black-box adversarial attacks with bandits and priors. arXiv preprint 1807, 2018.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. arXiv preprint arXivv/706.06608,, 2017.
Yandong Li, Lijun Li, Liqiang Wang. Tong Zhang, and Boqing Gong. Nattack: Learning the distributions of adver- sarial examples for improved black-box attack on deep neural networks. arXiv preprint 1rxiv:195..0044)
2019.
Jianbo Chen, Michacl Jordan, and Martin J Wainwright. Hopskipjumpattack: A query-efficient decision- attack. In 2020 ieee symposium security and privacy (xp), pages 1277-1294, IECE, 2020.
Lukas Schott. Jonas Rauber, Matthias Bethge, and Wieland Brendel. Towards the first adversarialyy robust neural network model mnist. arXiv preprint 1rXiv:1805.0911 2018.
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel. Ensemble adversarial training: Attacks and defenses. arXiv preprint I 2017.
Jacob Buckman, Aurko Roy, Colin Raffel, and Ian Goodfellow. Thermometer encoding: One hot way to resist adversarial examples. In International Conference on Learning Representations, 2018.
Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens Van Der Maaten. Countering adversarial images using input transformations. arXiv preprint arXiv: 1110//111100111111 17, 2017.
Vishaal Munusamy Kahilan, Brandon Morris, Hoang-Phuong Nguyen, and Anh Nguyen. Vectordefense: Vectorization as a defense to adversarial examples. In Soft Computing for Biomedical Applications and Related Topics, pages 19-35. Springer, 2018.
Aaditya Prakash, Nick Moran, Solomon Garber, Antonella DiLillo, and James Storer. Deflecting adversarial attacks with pixel deflection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8571 8580, 2018
Guneet S Dhillon, Kamyar Azizzadenesheli, Zachary C Lipton, Jeremy Bernstein, Jean Kossaifi, Aran Khanna, and Anima Anandkumar. Stochastic activation pruning for robust adversarial defense. arXiv preprint 1rXiv:1803.011 084.00440.... 2018.
Cihang Xie. Jianyu Wang, Zhishuai Zhang, Zhou Ren, and Alan Yuille. Mitigating adversarial effects through randomization. arXiv preprint arXiv: 1711.01991 2017.
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. arXiv preprint arXiv: 1002.0040, 2018.
Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses. arXiv preprint 2rXiv:2002.083477 2020.
Ajil Jalal, Andrew Ilyss, Constantinos Daskalakis, and Alexandros G Dimakis. The robust manifold defense: Adversarial training using generative models. arXiv preprint arXiv:1712.09116 2017.
Pouya Samangouei, Maya Kabkab, and Rama Chellappa. Defense-gan: Protecting classifiers against adversarial attacks using generative models. arXiv preprint arXiv:1805.0666 2018
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.178. 2014.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems. pages 3483-3491, 2015.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, pages 2234-2242 2016.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courviile. Improved training of wasserstein gans. In Advances in neural information processing systems, pages 5767-5777, 2017.
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In Advances in Nearal Information Processing Systems. pages 125- 136, 2019.
Michael F. Tipping and Christopher M Bishop. Probabilistic principal component analysis. Journal of the Royal Statistical Sociery: Series B (Statistical Methodology), 61(3):61 1-622, 1999.
11

---

Under review as a conference paper at ICLR 2021
Soheil Feizi, Farzan Farnia, Tony Ginart, and David Tse. Understanding gans: the lag setting, arXiv preprint arXiv:1710.10793, 2017.
Soheil Feizi, Farzan Farnia, Tony Ginart, and David Tse. Understanding gans in the lag setting: Formulation. generalization and stability. IEEE Journal on Selected Areas in Information Theory, 2020.
Bin Dai, Yu Wang. John Aston, Gang Hua, and David Wipf. Hidden talents of the variational autoencoder. arXiv preprint arXiv:1706. 05148, 2017.
Yann LeCun. Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document 1998.
recognition. Pruceedings of IEEE, 86(11):2278
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Dong Su, Huan Zhang. Hongge Chen, Jinfeng Yi, Pin-Yo Yu Chen, and Yupeng Gao. Is robustness the cost of accuracy?-a comprehensive study the robustness of 18 deep image classification models. In Proceedings of the European Conference on Computer Vision (ECCV), pages 631-648, 2018.
Florian Tramèr and Dan Boneh. Adversarial training and robustness for multiple perturbations. In Advances in Neural Information Processing Systems, pages 5858-5868, 2019.
Constantinos Daskalakis, Themis Gouleakis, Chistos Tzamos, and Manolis Zampetakis. Efficient statistics, in high dimensions. from truncated samples. In 2018 IEEE 59th Annual Symposium on Foundations of Computer Science (FOCS), pages 639-649. IEEE, 2018.
12

---

Under review as a conference paper at ICLR 2021
OVERVIEW
In appendix A, we provide the proof of the Theorems. In appendix B, show the settings about the experiments. Appendix C is further discussion of data augmentation using generative models. It is not closely related to our main paper.
A PROOF OF THE THEOREMS
A.1 PROBLEM DESCRIPTION
Lemma (Adversarial examples perturbed in the latent space). Using these 3 strategies, the adver- sarial examples can be unified as
Fado
x' + WA= and x' ~ D'J NH,,, UAAW
j 111,,33
where
A (1) (Ag - - 0
Ag 0
0
A
0
0.1
2 (Ag 02 (Ag o2 0² T²T
0
0
021
the data lie a q dimensional subspace, i.e. the covariance matrix E. is rank q. we have A (2) A (3) = Ax. Then D = D.
Proof:
Recall that & N(,, WWT + o²1), x/= ~ N(W + μ, 02 and z|æ N(P--W μ4), 2-11 where P = WTW + 0²I. The maximum likelihood estimator of W and o² are
WML Ug - 122/2/2 and oh
A
q 1=
Strategy Sample a ~ D, encode z = arg max ((<|0)) [ P-1 W (x pa),, add a perturbation
Az, and finally, decode Fadv arg max p(z Az) W (2 (z + Az) 1e. Then
-W(P-lW2 p.) A2) + p4+
Tads
=WP- 'T (I u + p4 + WA:
=x' WAz
Since (p., E.), we have 4.. (0. E.). Then
x' N WP w E. (WP (WP'''
With
WP
W
E,
(WP
WT
A
0
1/2
A
-1
0²
1/2
q
A.
0
0
2
0
1/2
A
-
A
2
1/2
UT
0
021 I
0
0
(A
-1
-U,
UT
0
-U.ADUT 1=1.
13

---

Under review as a conference paper at ICLR 2021
Strategy 2 Sample D, then sample q(2[2)) add perturbation Az, and finally, sample Fade ~ p(r 2 + Az). Then
2 ~~ PP -2P-1
and
Fado NN WA, WP W W
+ Wo'p-
Todv = x' WA:
With
WP- E, ( W + Wo P WT +
(Ag A - + U.
(A, - d'
0
-U.
0
U
+U
-U+ (Ag 0 + (Ag 0² I) + 0²1
0
U
o21
-UAAUUT ==
Strategy 3 Sample ~ (0, I). add a perturbation A and then sample Zade ~ p(z/2 + Az). In this strategy, Tado can be viewed as the adversarial example of = arg max, ((2/2r)
Eado ~ Nu + WA=, WW +
With
WWT 1
[ U
-U. 0
021
U. j == 3.
In these 3 strageties, the adversarial examples can be summerized as Tado z' + WA and x'~ D'j j 1,2,23
where j = 2, 3 corresponding to strategy 1,2 and 3. If the data lie in low dimensional space, i.e. the covariance matrix Σ, is rank q. Then the maximum likelihood of ou Eiggt >x/(d
4) Then
Ag 0
A (1) (2) A (3)
A.
0
There is no difference among these 3 strategies and there no change of distribution, i.e. D' = D.
A.1.1 EXCESS RISK ANALYSIS
Before we prove Theorem 2. We need to prove the following lemma first.
Lemma (optimal perturbation). Given (1,, Σ) the optimal solution of the inner max problem in equation 6 is
A WT LLE WWT)- -14.
where L is the lagrange multiplier satisfying |
Proof: Consider problem
max E(x' + WA E). |A=|IS=
The Lagrangian function is
L
llx' + WAw, p1, Σ)
2 [||= - =²)
d
log(2xT)
+ log Σ + 7 (z' - 4
WAe) (1)=))
21 - =?).
2
14

---

Under review as a conference paper at ICLR 2021
Notice that this quadratic objective function is concave when L is larger than the largest eigenvalue of W Σ- S.''WWWW -W. Calculate the partial derivative with respect to A and set it to be zero, we have
W ++ W LAx" =0
+L -' = - (z' -1)
+A= -L- ( -- 1)
AA:- -LE - -w).
The last equation comes from the Woodbury matrix inversion Lemma. We can obtain L by solving the equation ||=*1| || = E. We don't t have closed form solution of I. but we can solve it numerically. - 00 as e -> O. We only need to know I. is a constant our whole theory.
Theorem 2 (Excess risk). Let Lie and L be the loss with without perturbation in latent space (equation 6 and 3 respectively), given the non-robustly learned O, = (Ha, E.), The excess risk caused by perturbation is
A 02 02 (L- + 0²
1 A;
_ ==1
[1+
)²
j= 1,2,3
and the excess risk caused by changed of distribution is
d
[(e., = = log
A d).
A/
Proof: Since
x' ~ DD = =N(H.: Ej) - - N()
Denote
v= x' po ~ N(O,U.AWUUT)
And we have
WW = - = U.
A
Ag -
0²1
0
U..
0
0
The excess risk caused by perturbation is
(0. Dj) - L(O.. D())
=E(u+ + WWT(LE, -
EU TET
=Tr((I- WW (LE, (I
-Tr (r.'''BO "E''E
[1 (Ag - 021)((L - 1)A, + 11 0] N-A TT [T[AA
=Tr
1
0
-Tr[
[I+ (Ag - -(2)((L- -
1111-112 (1)- +
0
|-'
Tr AA
0
I
Σ (L - 1)A, + oz ) -
4 0²
1
X, 1=1,2,3.
(
and the excess risk caused by changed of distribution is
2(C(O., D') - L(O,,, ())
=log E - log Σ+1 + Eg (x' - (u)) ) TT- (I' (.)) - Ex - )₂) ". (x - -4x)
logg Σ5 - log E+1 + +TrrEE -
Ex (I' )x) -mm)²) - -rr TrEE. "Ez - (x) - )2,)
log II ( [L A;
+ Tr(A (1())) -T(AA=AA)
(11
d ()
= =log
+ Σ
- d). x
A
i=1
15

---

Under review as a conference paper at ICLR 2021
is hard to see which part dominates the excess risk. If we further assume that the data lie in a dimension manifold. The excess risk caused by the change of distribution becomes O. We have the following corollary.
Corollary (Excess risk). Let Lis and L be the loss with or without perturbation in latent space (equation )6) and 33) respectively), given the non-robustly learned θ (1hes D.), and rank (E.)
The excess risk
(i,(O.: DJ) -00. =
Proof:
By Lemma we have =²= 0. X X; and Dj DD Hence the excess risk caused by
changed of distributton
L D') L D) = 0.
The excess risk caused by perturbation is
2(C1,(O, ())) ((,,,))))
9
x; 0²
19-1] 11
11
+
1)1, + 0²
Σ
i=1 (L (L- ---1)==-- 1
(1+
1
-QqLT
Theorem (Excess risk of orginal space adversarial training). Let L. and L. be the loss with or without perturbation in original space (equations f and (3) respectively), given the non-robustly learned O. (Jr,, E.). Denote Min be the smallest eigenvalue of Ex. The excess risk
Ω2(A L) 3) < (0,, D) - L( D) < O (admmmmn))) L) ?)).
Theorem can be viewed as corollary of Theorem in Ilyas et al. 2019). We give the prove here.
Proof: Consider the Lagrange multiplier form of the inner maximization problem in equation
max E(x Az, p1 E).
|Ar|se
The Lagrangian function is
L
((x + Ax. ,e Σ) (1|rr| -
2
d
1 log Σ + (2 (x 4 Az)
1
(2 μ Ar) _ (||=] -=?).
L
12g(2 2T)
2
2
Notice that this quadratic objective function is concave when L is larger than the largest eigenvalue of E Calculate the partial derivative with respect to Ax and set it to be zero, we have
Σ- - # Ar") LAx"
+Ax" = (LE 1 u).
The excess risk is
- D))
-E(u+ (LE, 1)- Σ- (y (LE, - 1)) Ev Σーー1
=Tr[(I+ + (LE, 11 (LE, I) Eve T| Tr -e-'EEE 'Evv"
= (1 I1 +
Lb, - ----111
1
16

---

Under review as a conference paper at ICLR 2021
On the one hand,
Li 1 )2 32-11 1]
2111 + LAmin - 32
20 ((((mm)))) -
On the other hand,
)²
LA;
1=1
<d((1 + Tdmmn )2
<O(d(L)min) 3).
A.1.2 SADDLE POINT ANALYSIS
Theorem (Main result: Optimal Saddle point). The optimal solution of the modified problem in equation 8 is
His = u.. and Eis U.A.U..
where
X 2
4(A, o²)
+2 11 1
4(), 02)
for i 1 and X 11) for q-
L
j 1,2, corresponding strategies 1,2 and 3.
Problem 6 is not standard minimax problem, consider the modified problem
min
max BArDDy E + WAz, n. E), 1,,233. E. E =e
(9)
By lemma 3, the optimal perturbation A=" is a matrix M times μ.. Consider the problem
min P.
max M(x"
Ev'wDy NewwDy(e E(x' + WM p. 1,22.3
(10)
-ill=
Lemma 6 (optimal perturbation). Given O (1,, E) the optimal solution of the inner max problem
of 70 is
M WT -IEE-W''--
WWT)-
Proof: Consider the problem
El(x' + WM - 14, E)).
M| 11]l=m
The lagrangian function is
L (1IM('' - 11ll - e²]]
E B[e(x' + WM M,),.
Let x' =4=u v. Take the gradient with respect to M and set it to be zero, we have
L (1 - 11)1 {
E. E[l(x' + WM - p), Σ)
aM
1 100 MW - L Lo MMv/2]
-VME M x-
-|W t WWM LM El
=0.
17

---

Under review as a conference paper at ICLR 2021
Then we have
M -(--WW----W - -
The last equality is the Woodbury matrix inversion Lemma. Notice that lemma and Lemma 6 have the same form of solution. This is why we can use Problem T0 to approximate Problem 6 To solve the problem TO need to introduce Danskin's 's Theorem.
x 2 R is continuous function of two arguments, where Z C Th is compact. Define f(z) = max,ez o( z). Then, for every z Z,
Theorem 7 (Danskin's Theorem). Suppose o(x,
6(x,2) 2) is convex and differentiable in x, and 0ép Ör is continuous: The subdifferential of (x) is given by
00(x,2)
0f(x) = c
E 2o(x)},
dx
where conv(.) is the convex hull, and Zo (a:) is
Zo(x) {a Φ(I 2) = max (xx, 2)}.
the outer minimization problem is convex and differentiable, we can use any maximizer for the inner maximization problem find the saddle point. But the outer problem of problem 4 is convex, we need to modify the problem again. Assume that we have already obtained the eigenvector U from the ML estimator. The optimization variables of the outer minimization problem are μ and A. Then we have Σ U-AU? in problem I0] Another reason to make this assumption is that we only need consider the eigenvalue problem compare with standard adversarial training (Theorem
2 of Andrew Ilyas et al., 2019).
Proof of Theorem
By Lemma 6, we have
M T
(LE - WWT)-1
Ag 0²
0
1/2
Ag 0² 0 U".
LA
0
0 0
Which is diagonal matrix AM times UT. optimization problem becomes
Let T A-1, m A A-U μ and x" UT x'. The
min max m,T As
EEEDDY Elx' + U. (x' -p),, m,
(II)
s.t. E E= = 2
Obviously, the inner constraint is compact (by Heine- Borel theorem), we only need to prove the convexity of the outer problem to use Danskin's Theorem. For any x' and A A.
E(z' + WA Un m,
log( (2x) log ΣΣ| + (0' - - + WM((' - -1)" -) - p WM 14).
Ag o²
0
Let Uf(2'' 11), and A = (1+
A 2² consider the third term, we have
0
0
log Σ + 5(1 (x' + WM(w
- - # W 1)
2
T A2Tu.
"
By Daskalakis et al., 201 201&Daskalakis et al. (2018). The hessian matrix is (vec( - 2 Azz²) A=z²)) )) (vee( 2 Azz²)
)
H -or.-N/or'
2 0.
18

---

Under review as a conference paper at ICLR 2021
Therefore, this is convex problem. By the same calculation in Lemma 6, maximizer of the inner problem is
AM Aq
1/2
Ag
o²
0 0 )
LA
Then
Ag
o²
0
A σ²
) 2
)
A
1+
LA
Then the first order derivative (by Daskalakis et al. (2018) is
AA (i) - 177 ATT m AUT p
[mmjr==
From the second equation, directly have Mll Mpx. From the first equation, for i >> 9. we have
(1 0)+0) = A.1
For i S g. we have
021)2{(0) X/. (1 (1+ - - +
It equivalents to second order equation of v V
\ 02 =0 L
Solving this equation, we obtained
X 22 + 4(x: 0²) + 22 1
4(A: -02)
for 1 a and X12 = 11), for g.
1 T.
B EXPERIMENTS SETTINGS
B. MNIST
For Mnist, we use LeNet5 for the classifier and 2 layers MLP (with hidden size 256 and 784) for the encoder and decoder of conditional VAE, For standard training of the classifier, we use 30 epochs, batch size 128, learning rate 10-3, and weight decay 5 x 10-4 For the CVAE, we use 20 epochs, learning rate 10-3 batch size 64, and latent size 10.
For standard adversarial training, we use = 0.25 for FGSM and PGD. in PGD, we use 40 steps for the inner part. Adversarial training start after 10 epochs standard training.
For generative adversarial training, we use in the latent space with FGSM. Adversarial training start after 10 epoches standard training.
In the attack part, we use 0.2 for norm-based attack and e = 1 for generative attack on the test
B.2 CIFARIO
For CIFAR10, we use ResNet32 for the classifier and 4 layers CNN for the encoder and decoder of conditional VAE. For standard training the classifier, we use 200 epochs, batch size 128, learning rate 10 and weight decay 5 x 10 For the CVAE, use 100 epochs, learning rate 10 batch size 64, and latent size 128.
For standard adversarial training, for the inner part. Adversarial training start after 100 epochs standard training.
4/255 for FGSM and PGD. in PGD,
10 steps
For generative adversarial training, training start after 100 epoches standard training. Since we see that the modeling power of VAE in CIFARI 10 is not good enough. For each of the image, the encode variance very small. When
0.1 in the latent space with FGSM. Adversarial
19

---

Under review as a conference paper at ICLR 2021
add small perturbation to the encode mean value, the output image blured. Hence only use small E = 0.1.
In the attack part, test set. The test accuracy of VAE-adv against VAE attack is 40.18% in our experiments. It is not good enough because of the modeling power of VAE. But the results standard adversarial training versus VAE-attacks are worse, and vice versa. The experiments support our findings.
4/255 for norm norm-based attacks and 0.1 for generative attack on the
B.3 EIGENVALUES OF COVARIANCE MATRIX OF MNIST
We plot the eigenvalues of all the classes in this section, see Figure
C FINITE SAMPLES CASE: DATA AUGMENTATION OR ADVERSARIAL TRAINING
In this section we discuss the question that can we use the generative model to generate more examples for training in our theoretical framework. Because is not closely related to our main results, we only discuss it in appendix. Let us focus the case that the number of samples are not enough.
Generative model cannot help data augmentation Given dataset {r:)i= C Rd, let f and Š = Ô AÚ be the sample mean and sample covariance matrix. The generative model learned by this dataset is x = (A, 2)
= + ja +++ e. The distribution of the data sample from the model is
Tgen (á, WW + +021)= = (p, Ú ÜAeen & NN(,
(12)
we use n samples from the original dataset and m samples from the generative model, the loss function is
C(u, E) min P.E n 17
(ar + m E Bo (,, E)}.
(13)
Theorem (Data augmentation by generative model). Given MLE fa and S, The optimal solution training with n true samples and m generated samples is pda = fa and Ede = Ü da d where
ua A for is q and de
S4 + (n + m) ②)
Σ
Ax for i> q.
k=9+1
Proof: Consider the derivative of Problem 13
VuC + m
(x. 11) +
E, E-E-'(a 11)
+ m
Then we can obtain pda j..
- n2 + m
(x, uI(2; 11)²
m n m
Ex(x 4) 11) =0.
Then have
E1s n
$ + + m
S'
m
For i S 9,
X
m7
5, 3z
n + m
n. n m
For i > q.
1
m7 02
+
m
X
+ m
11 m
n + m
(n + m) 4)
k=
The optimal solution is a little bit destoryed. this perspective, generative models give no help to data augumentation.
20

---

Under review as a conference paper at ICLR 2021
MD
die dess stack
dean cat
con diCetak
VC eluck
Areggeowacch diaty
The fre 30 agggniaae
The 751 aigans uel
fre aaageaaae
Rabatact
Chet dio
cila
Clao
The cngann
(Bie dtus
diaieid We ataa
Westtak
Hefnd dregeiclss
MD
due dea
doar cite
cono
The frst 30 morties
The 754 ctamis oel
doe
city
fre agenaaat
Figure 3: Eigenvalues of the covariance matrices of MNIST for all the 10 classes
21

---

Under review as a conference paper at ICLR 2021
Generative model can help robust training
Theorem 9 (Adversarial learned parameters), The Adversarial learned features of problem 10 under finite samples is fis f and Ets- = Ü where A is of the same form in Theorem except replacing X by di under strategy 3.
Proof: Considering strategy 3. In this case
W Ü 02)11/2
and
<'~ (p, WW + 0221).
Since
WW + 0 1= ú Ag
o²1 UT.
In words, the cigenvectors of the dsitribution U is the same the one in perturbation W As. The op- timization problem is the same the use in the proof of Theorem 5 if we replace X by d 0
If the data lie in & dimensional subspace, we do not neet to make the assumption strategy 3.
22


Reviews:
[{"review_id": "n7_0hZIr4gs", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*Summary\n\nThis paper analytically considers two flavours of adversarial training in a Gaussian mixture model.  The first uses regular adversarial examples, and the second uses examples drawn from a generative model.  The authors show that the adversarial perturbations generated in the two cases differ in a cleanly-characterisable  way: in the first case the perturbations differ from real data in a direction aligned with the smallest eigenvalues  of the data covariance. In the latter case the perturbations are in a direction aligned with the largest eigenvalues.  Experimental results on MNIST and CIFAR are presented to illustrate how the analysis transfers to real datasets.\n\n*Positives\n\nThe analytical results on the mixture-of-Gaussians setting are very interesting. It's nice how the authors are able to get algebraic results in this setting. As far as I am aware this is novel (although I am not an expert).\n\nThe direct comparison between two different forms of adversarial examples (on- and off-)manifold is a nice framing that illustrates the difference between the two types of adversarial training. \n\n*Concerns\n\nThe paper seemed quite rushed, with a lot of spelling mistakes. Some of these are detailed in the final section.\n\nThe exposition of the theory could be motivated much better. As a representative example, in section 4.2 three alternative ways of generating the latent-space perturbations are presented, but there is no motivation describing how they are qualitatively different. Furthermore, it's not spelled out very clearly if the data itself is a lower-dimensional distribution embedded in a higher dimensional space (the usual setting for the 'manifold hypothesis' and proposed to be  a very important reason for adversarial examples). It seems like this is assumed in Corollary 3 (rank \\Sigma_* = q) but then  not assumed in theorem 4, since then \\lambda_min = 0. \n\nMost crucially, I'm not sure that the experiments are very convincing that the phenomenon explored in the mixture-of-Gaussians setting is actually present in real data. Firstly, it's quite hard to make conclusions from figure 2 when the first and second eigenvalues are not shown. The authors argue that the dynamic range is too great to plot the values meaningfully, but a logarithmic y-axis would suffice. Looking at the first column, we can see that the norm-based attack has a higher value than the VAE attack, in the reverse of what the authors claim would be the expected behaviour. In the second set of experiments, the authors claim that training against the PGD attack does not transfer to defense against the VAE attack. Actually the accuracy against the VAE attack increases from 42% to 52%, which does seem to indicate transferability to me. The authors also claim that unlike adversarial training for defense to l_0, l_1, l_\\infty attacks, the regular and generative examples have no robustness trade-off (i.e. you can train a model to be robust to both at once). However, the data seems to indicate that there is a trade-off. On MNIST, the PGD-trained model gets 95.51% accuracy on a PGD attack. The VAE-trained model gets 96.66% accuracy on the VAE attack. The jointly-trained model gets 89.5% accuracy on the PGD attack and 90.28% accuracy on the VAE attack, so it deteriorates in both cases. It's true there is perhaps less trade-off than expected, but the claim their method 'exhibit no robustness trade-off' seems unsupported by the evidence.\n\n\n*Recommendation\n\nOverall I recommend to reject this paper. \nWhile the analytical results are nice, the experiments are not very convincing that the analysis carries into real data. \nFurthermore, the paper as a whole seems rushed: missing details in the experimental section and a lack of motivation in the theory section. \n\n*Questions\n\nCan the authors explain the non-shown eigenvalues in the experiments, and discuss in more detail how their claim about the robustness trade-off is supported by the evidence?\n\n*Minor points\n\nI'm not an expert, but I think the description of PDG is an incorrect characterisation for norm other than the \\ell_\\infty norm. In particular, the step taken being a multiple of the sign of the gradient is only correct for the \\ell_\\infty norm, since there it corresponds to the steepest descent step. For e.g. the \\ell_2 norm we should instead divide by the norm of the gradient, and so on.  In the experiments the 'both-adv' column is uniformly bold, even though in multiple rows it performs worse than the other models. For instance, the FGSM-adv model performs better under an FGSM-attack than the 'both-adv' model for CIFAR, so should be in bold (under the general rule that bold means the best-performing model unless otherwise specified). \nThe experimental results are quite sparse. In particular, I attempted to replicate the clean data results from figure 2 but was unable to with the details in the paper. Was there any normalization applied to the MNIST data before computing the covariance?\n\n*Typos:\n\nSection 3, para 1: '. Adversarial training is to solve' -> '. The goal of adversarial training is to solve'\nSection 4.1, after equation 3: '. Adversarial training is to solve' -> '. The goal of adversarial training is to solve'\nTheorem 6: 'The optimal solution of problem in' -> 'The optimal solution of the problem in '\nPage 7, para 1: 'class 1 and 2' is described as class 1 and 2 and class 0 and 1. \nPage 7, para 2: 'closed' -> 'close'\nPage 8, para 3: 'Figure 1' -> 'Table 1'\nPage 8, para 5: 'amplifying low variance of distribution' not grammatical sentence. \nPage 8, para 6: 'we can defense all' -> 'we can defend all the attacks'", "extra_scores": {}, "extra_text": {}}, {"review_id": "dzqFNC7_H8p", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the author mainly show that adversarial robustness can be disentangled in small variance directions and large variance directions.. Theoretically, they also investigated the excess risk and optimal saddle point of the minimax problem of latent space adversarial training. \n\nPositive:\n1.  found the regular adversarial examples attack tend to lies in small variance directions of the data.\n2. found generative adversarial examples attack towards to the large variance directions of the data.\n3. explore standard adversarial training as well as latent space adversarial training to deal with  on-manifold and off-mainfold issue.\n4. The theory analysis may be useful to use original/latent adversarial training to increase the model robustness.\n\n\nNegative:\n1. The theoretical analysis is mainly based in probabilistic principle component analysis, a linear generative model. The extension to nonlinear model is unclear, which may be more common in practice.\n2. In addition to LeNet and ResNet, it may be more convincing to test two more extra models to confirm the theoretical findings. The analysis rely on the eigenvalues, what if the original features are in high-dimensional space. computing eigenvalue decomposition may be expensive.\n3. In the Table 1, it seems that using both regular adversarial examples and generative adversarial examples sometime does not obtain test accuracy, any more discussion?\n\nIn summary, the authors provide a theoretical study on theoretical analysis of the attacking mechanisms of the two kinds of adversarial examples: regular and generative adversarial examples. They should w that adversarial robustness can be disentangled in directions of the data manifold.  Such finds may be useful in designing defense algorithms.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ceP75ed9eB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The present paper proposes an interesting study about two different kinds of adversarials, depending on the area of the variance and manifold that is attacked. Although the results are promising, there is some lack of further experimentation to provide more meaningful results, so I would recommend providing more experimentation with more different and recent attacks and datasets to enhance the contribution. Hereby I provide some suggestions and minor comments per section, as well as some general\n1.\tIntroduction Why the method is not applied experimentally to real datasets instead of just claiming that theoretically proved that it works? More details and experimentation should be provided about the application in these scenarios. Minor:\n•\t“training methods that use” (not using)\n2.\tRelated work In general, the section should be extended with more details in the concepts: white/black box approaches, optimization methods (difference between first and zeroth). Missing reference for zeroth order optimization. Also, more recent attacks in the white box setting are available now, which should be preferred rather than “first generation” 2016/2017 attacks. Same for black box approach. Consider for example SL1D, HopSkipJump etc Stating that adversarial training is the only effecting defense is a big claim. Consider supporting this kind of discussions with more references. For example: Tramer, F., Carlini, N., Brendel, W., & Madry, A. (2020). On adaptive attacks to adversarial example defenses. arXiv preprint arXiv:2002.08347.\n3.\tProblem description Why do not use Carlini and Wagner attack (for example), rather than FGSM, as the former is a more powerful attack and more widely relevant in the state of the art. In line with previous comments, more recent approaches for this kind of attacks should be preferred.\n4.\tTheoretical analysis Minor:\n•\tTo simplify (not “simply”) in 4.3 (excess risk analysis)\n5.\tExperiments The number of samples used for experimentation in each dataset should be provided in the main text, along with more details on the training process (apart from the appendix) Why do labels in Figure 2 not match clear names for data representation? (no need to keep track of the renaming from PGD to “norm-base attack”, just make an effort to keep naming consistent) Minor:\n•\tMNIST dataset should be capitalized (not “Mnist”)\n•\t“resnet” network should also be spelled as its proper acronym: ResNet (for example, as derived from Residual Networks)\n•\t“lies in a low dimension affine plane in R784. After” (capitalization after full stop)\n6.\tConclusion The final results in Table 1 do not show a clear improvement with both, were in some cases the accuracy is reduced with respect to using a single method. For example, 89.50% with both vs 95.51 % when using only PGD-adv. This happens in the majority of cases. For this reason, a more detailed study on the contribution of each method to the defence should be provided, such as statistical tests and an extensive ablation study. The conclusion paragraph itself is too brief, just a brief summary rushing to the end. More discussion should be provided, along with more insights on the potential applications (for example, how to take advantage from this knowledge to design a new defence technique). General:\n•\tMore references regarding the adversarial robustness trade-off should be added to the text and discussed, as a key point presented in the introduction and suggested as future work in the conclusion. Consider for example the following two:\n1.\tSu, D., Zhang, H., Chen, H., Yi, J., Chen, P. Y., & Gao, Y. (2018). Is Robustness the Cost of Accuracy?--A Comprehensive Study on the Robustness of 18 Deep Image Classification Models. In Proceedings of the European Conference on Computer Vision (ECCV) (pp. 631-648).\n2.\tDeniz, O., Pedraza, A., Vallez, N., Salido, J., & Bueno, G. (2020). Robustness to adversarial examples can be improved with overfitting. International Journal of Machine Learning and Cybernetics, 1-10.\n•\tEnglish spelling should be revised thoroughly, since typos are found frequently.\n\t", "extra_scores": {}, "extra_text": {}}, {"review_id": "EsluN_f6zfa", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper investigates the differences between attack and defense in adversarial machine learning, when a) the adversarial perturbation is applied directly to the data samples, and b) when perturbation is applied to the latent space of a VAE or GAN which then generates the data. The above-mentioned investigation includes both theoretical and experimental analysis which are given in Section 4 and 5 of the paper, respectively.\n\nPaper is well-motivated and is focused on some recent and interesting aspects of adversarial robustness. However, the theoretical treatment given in this paper only considers a very simple problem setting instead of providing a more comprehensive framework for future researches. Also, the paper's main claim may not be firmly established yet, and needs further mathematical work and clarification to become ICLR-ready. To be more specific, I have the following two main concerns about this paper.\n\n#1:\nAuthors, both in the abstract and also several other places in the Introduction section, have made a number of huge claims about (theoretically) analyzing the fundamental differences of on-manifold and off-manifold attacks, in general. However, the theoretical analysis in this paper is completely centered around a highly-restricted linear generative model with a carefully-chosen design. Data generation model is also assumed to be a simple Gaussian model with infinitely many observed samples, which gives easy-to-handle analytic closed forms for the solution of all the optimization problems that one may encounter in this work. Therefore, the big claims made throughout the abstract and introduction section need to be significantly relaxed.\n\nAlso, it might not be a bad idea to re-organize the overall structure of the paper: Beginning with some experimentation and then showing the theoretical results as a simple case study. Otherwise, the limitations and also the simplifying assumptions behind presented theoretical analysis should be clearly mentioned both in abstract and Introduction.\n\n#2:\nI am not completely sure about the mathematical validity of the main claim of the paper (at least, from a theoretical standpoint), which is the \"The comparison between Corollary 3 and Theorem 4\". First, in my opinion, both Corollary 3 and Theorem 4 have shown that the true dimension underlying the data (or at least, the number of dimensions that the adversary is allowed to use for its attacks) appears in the excess risk bound, i.e.\n$$\n\\mathcal{L}\\_r\\left(\\Theta_{\\star},\\mathcal{D}\\right)-\\mathcal{L}\\left(\\Theta_{\\star},\\mathcal{D}\\right)\n\\leq \n\\mathcal{O}\\left(d\\left(\\lambda_{\\min}L\\right)^{-2}\\right)\n$$\nwhich shows a linear dependence on $d$, and\n$$\n\\\\mathcal{L}\\_{\\\\mathrm{ls}}\\\\left(\\\\Theta\\_{\\\\star},\\\\mathcal{D}\\\\right)\n-\\\\mathcal{L}\\\\left(\\\\Theta\\_{\\\\star},\\\\mathcal{D}\\\\right)\n\\\\leq \n\\\\mathcal{O}\\\\left(qL^{-2}\\\\right).\n$$\nwhich shows a similar increasing behavior w.r.t. $q$. However, authors have preferred to only use the lower-bound of Theorem 4 which does not contain the factor $d$. Why? that should be properly explained. Moreover, the fact that the upper bound of Corollary 3 scales with $q$ while that of Theorem 4 scales with $d$ makes sense, since in the latent space adversarial training, adversary is restricted to use only $q\\\\leq d$ possible directions to form its attack.\n\nThe other crucial difference between the results of Theorem 4 and Corollary 3, is the appearance of $\\\\lambda_{\\\\min}$ in Theorem 4, but not Corollary 3. Authors have concluded that this difference must have something to do with the fundamental differences between on-manifold and off-manifold attacks. But, that is mainly due to the crucial difference in the mathematical meaning of Lagrange multiplier $L$  in the two formulations (Eq. (6) and Eq. (4)). In one of them, we have $\\\\Vert\\\\Delta\\\\boldsymbol{x}\\\\Vert\\\\leq\\\\epsilon$, where the perturbation is $\\\\Delta\\\\boldsymbol{x}$ itself. But in the other formulation we have $\\\\Vert\\\\Delta\\\\boldsymbol{z}\\\\Vert\\\\leq\\\\epsilon$, while the perturbation is $\\\\boldsymbol{W}\\\\Delta\\\\boldsymbol{z}$. So, for a fixed perturbation intensity $\\\\epsilon$, the magnitude of Lagrange multiplier $L$ needs to be adjusted between the two formulation (w.r.t. to spectrum of $\\\\boldsymbol{W}$). My guess is that it would justify the appearance of $\\\\lambda_{\\\\min}$ in Theorem 4 but not in Corollary 3. I'd like to know the authors' response on this issue as well.\n\nThere are several typos or grammatical errors in the paper. Some are listed below:\n\npage 2: The attacker have -> should be corrected.\npage 2: argumented -> augmented.\npage2: VAE is also be used to train robust model -> should be rephrased.\n\npage 4: Formulation of $\\\\boldsymbol{W}_{\\\\mathrm{ML}}$ may not be correct. The matrix on the r.h.s. might not have a real square root.\npage 4: Data -> data.\n\npage 5: simply -> simplify.\n\npage 6: $q$-dimensional\n\nAt this stage, I cannot recommend this paper for publication at ICLR 2021 unless the above issues are answered properly.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the authors theoretically analyzed the attacking mechanisms of the two kinds of adversarial examples in the Gaussian mixture data model case and proved that adversarial robustness can be disentangled in directions of the data manifold. The reviewers commonly felt that the idea and theoretical analysis in this paper are interesting, but experiments are not satisfactory.

At the current status, they still have a main concern regarding the correctness of comparison between the results of Theorem 4 and Corollary 3 (which is the heart of their theoretical claims, the main message of the paper and the main motivation for experiments).

As a whole, this paper has some merits but the authors still cannot clarify some concerns raised by some reviewers.


Decision: Reject